Sep 19 01:12:27.613236 ip-10-0-129-113 systemd[1]: Starting Kubernetes Kubelet...
Sep 19 01:12:31.063376 ip-10-0-129-113 kubenswrapper[2581]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 19 01:12:31.063376 ip-10-0-129-113 kubenswrapper[2581]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 19 01:12:31.063376 ip-10-0-129-113 kubenswrapper[2581]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 19 01:12:31.063376 ip-10-0-129-113 kubenswrapper[2581]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 19 01:12:31.063376 ip-10-0-129-113 kubenswrapper[2581]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 19 01:12:31.064067 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.063458    2581 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 19 01:12:31.067124 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.067107    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 19 01:12:31.067124 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.067123    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 19 01:12:31.067193 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.067126    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 19 01:12:31.067193 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.067130    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 19 01:12:31.067193 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.067133    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 19 01:12:31.067193 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.067136    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 19 01:12:31.067193 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.067139    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 19 01:12:31.067193 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.067142    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 19 01:12:31.067193 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.067145    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 19 01:12:31.067193 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.067147    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 19 01:12:31.067193 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.067150    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 19 01:12:31.067193 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.067153    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 19 01:12:31.067193 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.067156    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 19 01:12:31.067193 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.067158    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 19 01:12:31.067193 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.067161    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 19 01:12:31.067193 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.067164    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 19 01:12:31.067193 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.067166    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 19 01:12:31.067193 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.067169    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 19 01:12:31.067193 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.067172    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 19 01:12:31.067193 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.067175    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 19 01:12:31.067193 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.067178    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 19 01:12:31.067701 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.067180    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 19 01:12:31.067701 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.067183    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 19 01:12:31.067701 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.067186    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 19 01:12:31.067701 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.067188    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 19 01:12:31.067701 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.067191    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 19 01:12:31.067701 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.067193    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 19 01:12:31.067701 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.067197    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 19 01:12:31.067701 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.067199    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 19 01:12:31.067701 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.067202    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 19 01:12:31.067701 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.067205    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 19 01:12:31.067701 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.067208    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 19 01:12:31.067701 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.067210    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 19 01:12:31.067701 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.067213    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 19 01:12:31.067701 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.067215    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 19 01:12:31.067701 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.067218    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 19 01:12:31.067701 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.067220    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 19 01:12:31.067701 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.067229    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 19 01:12:31.067701 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.067232    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 19 01:12:31.067701 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.067235    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 19 01:12:31.067701 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.067237    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 19 01:12:31.068228 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.067239    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 19 01:12:31.068228 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.067242    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 19 01:12:31.068228 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.067244    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 19 01:12:31.068228 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.067247    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 19 01:12:31.068228 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.067249    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 19 01:12:31.068228 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.067252    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 19 01:12:31.068228 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.067255    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 19 01:12:31.068228 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.067257    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 19 01:12:31.068228 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.067259    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 19 01:12:31.068228 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.067262    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 19 01:12:31.068228 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.067264    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 19 01:12:31.068228 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.067268    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 19 01:12:31.068228 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.067271    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 19 01:12:31.068228 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.067276    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 19 01:12:31.068228 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.067280    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 19 01:12:31.068228 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.067284    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 19 01:12:31.068228 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.067288    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 19 01:12:31.068228 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.067291    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 19 01:12:31.068228 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.067294    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 19 01:12:31.068693 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.067299    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 19 01:12:31.068693 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.067303    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 19 01:12:31.068693 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.067306    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 19 01:12:31.068693 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.067309    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 19 01:12:31.068693 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.067315    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 19 01:12:31.068693 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.067317    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 19 01:12:31.068693 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.067320    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 19 01:12:31.068693 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.067322    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 19 01:12:31.068693 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.067325    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 19 01:12:31.068693 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.067327    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 19 01:12:31.068693 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.067338    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 19 01:12:31.068693 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.067341    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 19 01:12:31.068693 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.067343    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 19 01:12:31.068693 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.067346    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 19 01:12:31.068693 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.067354    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 19 01:12:31.068693 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.067357    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 19 01:12:31.068693 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.067359    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 19 01:12:31.068693 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.067362    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 19 01:12:31.068693 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.067365    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 19 01:12:31.068693 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.067367    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 19 01:12:31.069196 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.067370    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 19 01:12:31.069196 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.067372    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 19 01:12:31.069196 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.067375    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 19 01:12:31.069196 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.067378    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 19 01:12:31.069196 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.067380    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 19 01:12:31.069196 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.067382    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 19 01:12:31.069196 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.067385    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 19 01:12:31.069196 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.067388    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 19 01:12:31.069196 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.067391    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 19 01:12:31.069196 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.067393    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 19 01:12:31.069196 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.067395    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 19 01:12:31.069196 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.067398    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 19 01:12:31.069196 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.067407    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 19 01:12:31.069196 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.067411    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 19 01:12:31.069196 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.067414    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 19 01:12:31.069196 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.067416    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 19 01:12:31.069196 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.067419    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 19 01:12:31.069196 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.067421    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 19 01:12:31.069196 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.067424    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 19 01:12:31.069196 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.068243    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 19 01:12:31.069702 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.068250    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 19 01:12:31.069702 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.068253    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 19 01:12:31.069702 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.068256    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 19 01:12:31.069702 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.068264    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 19 01:12:31.069702 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.068267    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 19 01:12:31.069702 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.068270    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 19 01:12:31.069702 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.068273    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 19 01:12:31.069702 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.068275    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 19 01:12:31.069702 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.068278    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 19 01:12:31.069702 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.068281    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 19 01:12:31.069702 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.068284    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 19 01:12:31.069702 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.068286    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 19 01:12:31.069702 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.068289    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 19 01:12:31.069702 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.068292    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 19 01:12:31.069702 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.068294    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 19 01:12:31.069702 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.068297    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 19 01:12:31.069702 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.068299    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 19 01:12:31.069702 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.068302    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 19 01:12:31.069702 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.068306    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 19 01:12:31.070235 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.068311    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 19 01:12:31.070235 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.068315    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 19 01:12:31.070235 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.068318    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 19 01:12:31.070235 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.068320    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 19 01:12:31.070235 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.068323    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 19 01:12:31.070235 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.068326    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 19 01:12:31.070235 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.068329    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 19 01:12:31.070235 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.068332    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 19 01:12:31.070235 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.068334    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 19 01:12:31.070235 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.068337    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 19 01:12:31.070235 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.068340    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 19 01:12:31.070235 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.068343    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 19 01:12:31.070235 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.068345    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 19 01:12:31.070235 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.068348    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 19 01:12:31.070235 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.068350    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 19 01:12:31.070235 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.068353    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 19 01:12:31.070235 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.068355    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 19 01:12:31.070235 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.068363    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 19 01:12:31.070235 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.068366    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 19 01:12:31.070707 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.068368    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 19 01:12:31.070707 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.068370    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 19 01:12:31.070707 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.068373    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 19 01:12:31.070707 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.068377    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 19 01:12:31.070707 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.068379    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 19 01:12:31.070707 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.068382    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 19 01:12:31.070707 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.068386    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 19 01:12:31.070707 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.068389    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 19 01:12:31.070707 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.068393    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 19 01:12:31.070707 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.068397    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 19 01:12:31.070707 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.068400    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 19 01:12:31.070707 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.068403    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 19 01:12:31.070707 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.068406    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 19 01:12:31.070707 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.068409    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 19 01:12:31.070707 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.068411    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 19 01:12:31.070707 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.068414    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 19 01:12:31.070707 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.068417    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 19 01:12:31.070707 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.068420    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 19 01:12:31.070707 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.068423    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 19 01:12:31.070707 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.068426    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 19 01:12:31.071224 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.068428    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 19 01:12:31.071224 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.068431    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 19 01:12:31.071224 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.068433    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 19 01:12:31.071224 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.068436    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 19 01:12:31.071224 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.068439    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 19 01:12:31.071224 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.068441    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 19 01:12:31.071224 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.068444    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 19 01:12:31.071224 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.068447    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 19 01:12:31.071224 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.068449    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 19 01:12:31.071224 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.068451    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 19 01:12:31.071224 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.068454    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 19 01:12:31.071224 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.068470    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 19 01:12:31.071224 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.068473    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 19 01:12:31.071224 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.068476    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 19 01:12:31.071224 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.068478    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 19 01:12:31.071224 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.068481    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 19 01:12:31.071224 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.068484    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 19 01:12:31.071224 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.068487    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 19 01:12:31.071224 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.068489    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 19 01:12:31.071705 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.068492    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 19 01:12:31.071705 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.068522    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 19 01:12:31.071705 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.068527    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 19 01:12:31.071705 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.068530    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 19 01:12:31.071705 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.068533    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 19 01:12:31.071705 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.068536    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 19 01:12:31.071705 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.068539    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 19 01:12:31.071705 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.068542    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 19 01:12:31.071705 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.068544    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 19 01:12:31.071705 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.068547    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 19 01:12:31.071705 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.068550    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 19 01:12:31.071705 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.068552    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 19 01:12:31.071705 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.068555    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 19 01:12:31.071705 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.068557    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 19 01:12:31.071705 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.068560    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 19 01:12:31.071705 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.068562    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 19 01:12:31.071705 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.068565    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 19 01:12:31.071705 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.068567    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 19 01:12:31.071705 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.068570    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 19 01:12:31.071705 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.068572    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 19 01:12:31.072266 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.068575    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 19 01:12:31.072266 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.069027    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 19 01:12:31.072266 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.069032    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 19 01:12:31.072266 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.069036    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 19 01:12:31.072266 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.069044    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 19 01:12:31.072266 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.069047    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 19 01:12:31.072266 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.069050    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 19 01:12:31.072266 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.069058    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 19 01:12:31.072266 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.069061    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 19 01:12:31.072266 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.069063    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 19 01:12:31.072266 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.069066    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 19 01:12:31.072266 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.069068    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 19 01:12:31.072266 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.069071    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 19 01:12:31.072266 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.069074    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 19 01:12:31.072266 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.069076    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 19 01:12:31.072266 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.069079    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 19 01:12:31.072266 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.069081    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 19 01:12:31.072266 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.069084    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 19 01:12:31.072266 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.069087    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 19 01:12:31.072266 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.069089    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 19 01:12:31.072832 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.069091    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 19 01:12:31.072832 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.069094    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 19 01:12:31.072832 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.069097    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 19 01:12:31.072832 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.069099    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 19 01:12:31.072832 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.069102    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 19 01:12:31.072832 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.069104    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 19 01:12:31.072832 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.069107    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 19 01:12:31.072832 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.069109    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 19 01:12:31.072832 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.069112    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 19 01:12:31.072832 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.069114    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 19 01:12:31.072832 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.069117    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 19 01:12:31.072832 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.069119    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 19 01:12:31.072832 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.069122    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 19 01:12:31.072832 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.069124    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 19 01:12:31.072832 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.069126    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 19 01:12:31.072832 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.069129    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 19 01:12:31.072832 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.069131    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 19 01:12:31.072832 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.069139    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 19 01:12:31.072832 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.069141    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 19 01:12:31.072832 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.069144    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 19 01:12:31.073659 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.069146    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 19 01:12:31.073659 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.069149    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 19 01:12:31.073659 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.069152    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 19 01:12:31.073659 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.069154    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 19 01:12:31.073659 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.069156    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 19 01:12:31.073659 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.069159    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 19 01:12:31.073659 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.069162    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 19 01:12:31.073659 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.069164    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 19 01:12:31.073659 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.069167    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 19 01:12:31.073659 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.069170    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 19 01:12:31.073659 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.069172    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 19 01:12:31.073659 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.069176    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 19 01:12:31.073659 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.069180    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 19 01:12:31.073659 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.069183    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 19 01:12:31.073659 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.069185    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 19 01:12:31.073659 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.069188    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 19 01:12:31.073659 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.069190    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 19 01:12:31.073659 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.069193    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 19 01:12:31.073659 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.069196    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 19 01:12:31.074185 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.069199    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 19 01:12:31.074185 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.069202    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 19 01:12:31.074185 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.069204    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 19 01:12:31.074185 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.069207    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 19 01:12:31.074185 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.069210    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 19 01:12:31.074185 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.069212    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 19 01:12:31.074185 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.069214    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 19 01:12:31.074185 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.069217    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 19 01:12:31.074185 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.069219    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 19 01:12:31.074185 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.069222    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 19 01:12:31.074185 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.069224    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 19 01:12:31.074185 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.069232    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 19 01:12:31.074185 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.069235    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 19 01:12:31.074185 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.069237    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 19 01:12:31.074185 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.069241    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 19 01:12:31.074185 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.069245    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 19 01:12:31.074185 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.069252    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 19 01:12:31.074185 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.069255    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 19 01:12:31.074185 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.069258    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 19 01:12:31.074696 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.069260    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 19 01:12:31.074696 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.069263    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 19 01:12:31.074696 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.069265    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 19 01:12:31.074696 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.069268    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 19 01:12:31.074696 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.069270    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 19 01:12:31.074696 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.069273    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 19 01:12:31.074696 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.069275    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 19 01:12:31.074696 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.069278    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 19 01:12:31.074696 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.069280    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 19 01:12:31.074696 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.069283    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 19 01:12:31.074696 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.069285    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 19 01:12:31.074696 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.069287    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 19 01:12:31.074696 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.069290    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 19 01:12:31.074696 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.069294    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 19 01:12:31.074696 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.069297    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 19 01:12:31.074696 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.069300    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 19 01:12:31.074696 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.069302    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 19 01:12:31.074696 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.069304    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 19 01:12:31.074696 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.069307    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 19 01:12:31.074696 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.069309    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 19 01:12:31.075207 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.069312    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 19 01:12:31.075207 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.069314    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 19 01:12:31.075207 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069384    2581 flags.go:64] FLAG: --address="0.0.0.0"
Sep 19 01:12:31.075207 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069416    2581 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 19 01:12:31.075207 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069425    2581 flags.go:64] FLAG: --anonymous-auth="true"
Sep 19 01:12:31.075207 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069434    2581 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 19 01:12:31.075207 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069441    2581 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 19 01:12:31.075207 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069445    2581 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 19 01:12:31.075207 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069449    2581 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 19 01:12:31.075207 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069453    2581 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 19 01:12:31.075207 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069456    2581 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 19 01:12:31.075207 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069459    2581 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 19 01:12:31.075207 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069462    2581 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 19 01:12:31.075207 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069465    2581 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 19 01:12:31.075207 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069468    2581 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 19 01:12:31.075207 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069471    2581 flags.go:64] FLAG: --cgroup-root=""
Sep 19 01:12:31.075207 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069474    2581 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 19 01:12:31.075207 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069477    2581 flags.go:64] FLAG: --client-ca-file=""
Sep 19 01:12:31.075207 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069480    2581 flags.go:64] FLAG: --cloud-provider="external"
Sep 19 01:12:31.075207 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069483    2581 flags.go:64] FLAG: --cluster-dns="[]"
Sep 19 01:12:31.075207 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069488    2581 flags.go:64] FLAG: --cluster-domain=""
Sep 19 01:12:31.075207 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069491    2581 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 19 01:12:31.075207 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069494    2581 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 19 01:12:31.075207 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069497    2581 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 19 01:12:31.075787 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069500    2581 flags.go:64] FLAG: --container-log-max-files="5"
Sep 19 01:12:31.075787 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069504    2581 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 19 01:12:31.075787 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069507    2581 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 19 01:12:31.075787 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069510    2581 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 19 01:12:31.075787 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069513    2581 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 19 01:12:31.075787 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069516    2581 flags.go:64] FLAG: --contention-profiling="false"
Sep 19 01:12:31.075787 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069519    2581 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 19 01:12:31.075787 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069522    2581 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 19 01:12:31.075787 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069525    2581 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 19 01:12:31.075787 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069528    2581 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 19 01:12:31.075787 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069532    2581 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 19 01:12:31.075787 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069535    2581 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 19 01:12:31.075787 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069538    2581 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 19 01:12:31.075787 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069540    2581 flags.go:64] FLAG: --enable-load-reader="false"
Sep 19 01:12:31.075787 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069551    2581 flags.go:64] FLAG: --enable-server="true"
Sep 19 01:12:31.075787 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069555    2581 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 19 01:12:31.075787 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069562    2581 flags.go:64] FLAG: --event-burst="100"
Sep 19 01:12:31.075787 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069566    2581 flags.go:64] FLAG: --event-qps="50"
Sep 19 01:12:31.075787 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069568    2581 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 19 01:12:31.075787 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069572    2581 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 19 01:12:31.075787 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069574    2581 flags.go:64] FLAG: --eviction-hard=""
Sep 19 01:12:31.075787 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069578    2581 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 19 01:12:31.075787 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069581    2581 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 19 01:12:31.075787 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069584    2581 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 19 01:12:31.075787 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069587    2581 flags.go:64] FLAG: --eviction-soft=""
Sep 19 01:12:31.076448 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069590    2581 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 19 01:12:31.076448 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069593    2581 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 19 01:12:31.076448 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069596    2581 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 19 01:12:31.076448 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069599    2581 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 19 01:12:31.076448 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069602    2581 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 19 01:12:31.076448 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069605    2581 flags.go:64] FLAG: --fail-swap-on="true"
Sep 19 01:12:31.076448 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069607    2581 flags.go:64] FLAG: --feature-gates=""
Sep 19 01:12:31.076448 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069611    2581 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 19 01:12:31.076448 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069614    2581 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 19 01:12:31.076448 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069617    2581 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 19 01:12:31.076448 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069620    2581 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 19 01:12:31.076448 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069622    2581 flags.go:64] FLAG: --healthz-port="10248"
Sep 19 01:12:31.076448 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069625    2581 flags.go:64] FLAG: --help="false"
Sep 19 01:12:31.076448 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069628    2581 flags.go:64] FLAG: --hostname-override="ip-10-0-129-113.ec2.internal"
Sep 19 01:12:31.076448 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069631    2581 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 19 01:12:31.076448 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069634    2581 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 19 01:12:31.076448 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069637    2581 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 19 01:12:31.076448 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069640    2581 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 19 01:12:31.076448 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069644    2581 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 19 01:12:31.076448 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069646    2581 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 19 01:12:31.076448 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069650    2581 flags.go:64] FLAG: --image-service-endpoint=""
Sep 19 01:12:31.076448 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069653    2581 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 19 01:12:31.076448 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069662    2581 flags.go:64] FLAG: --kube-api-burst="100"
Sep 19 01:12:31.077111 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069665    2581 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 19 01:12:31.077111 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069668    2581 flags.go:64] FLAG: --kube-api-qps="50"
Sep 19 01:12:31.077111 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069671    2581 flags.go:64] FLAG: --kube-reserved=""
Sep 19 01:12:31.077111 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069674    2581 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 19 01:12:31.077111 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069677    2581 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 19 01:12:31.077111 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069680    2581 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 19 01:12:31.077111 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069682    2581 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 19 01:12:31.077111 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069685    2581 flags.go:64] FLAG: --lock-file=""
Sep 19 01:12:31.077111 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069688    2581 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 19 01:12:31.077111 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069691    2581 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 19 01:12:31.077111 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069694    2581 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 19 01:12:31.077111 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069699    2581 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 19 01:12:31.077111 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069702    2581 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 19 01:12:31.077111 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069704    2581 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 19 01:12:31.077111 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069707    2581 flags.go:64] FLAG: --logging-format="text"
Sep 19 01:12:31.077111 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069710    2581 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 19 01:12:31.077111 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069713    2581 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 19 01:12:31.077111 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069717    2581 flags.go:64] FLAG: --manifest-url=""
Sep 19 01:12:31.077111 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069720    2581 flags.go:64] FLAG: --manifest-url-header=""
Sep 19 01:12:31.077111 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069724    2581 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 19 01:12:31.077111 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069727    2581 flags.go:64] FLAG: --max-open-files="1000000"
Sep 19 01:12:31.077111 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069739    2581 flags.go:64] FLAG: --max-pods="110"
Sep 19 01:12:31.077111 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069742    2581 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 19 01:12:31.077111 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069745    2581 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 19 01:12:31.077111 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069748    2581 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 19 01:12:31.077768 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069751    2581 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 19 01:12:31.077768 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069753    2581 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 19 01:12:31.077768 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069756    2581 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 19 01:12:31.077768 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069759    2581 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 19 01:12:31.077768 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069766    2581 flags.go:64] FLAG: --node-status-max-images="50"
Sep 19 01:12:31.077768 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069770    2581 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 19 01:12:31.077768 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069773    2581 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 19 01:12:31.077768 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069782    2581 flags.go:64] FLAG: --pod-cidr=""
Sep 19 01:12:31.077768 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069785    2581 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 19 01:12:31.077768 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069790    2581 flags.go:64] FLAG: --pod-manifest-path=""
Sep 19 01:12:31.077768 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069793    2581 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 19 01:12:31.077768 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069796    2581 flags.go:64] FLAG: --pods-per-core="0"
Sep 19 01:12:31.077768 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069799    2581 flags.go:64] FLAG: --port="10250"
Sep 19 01:12:31.077768 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069802    2581 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 19 01:12:31.077768 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069804    2581 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-046a9ff35d057dc14"
Sep 19 01:12:31.077768 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069807    2581 flags.go:64] FLAG: --qos-reserved=""
Sep 19 01:12:31.077768 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069810    2581 flags.go:64] FLAG: --read-only-port="10255"
Sep 19 01:12:31.077768 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069813    2581 flags.go:64] FLAG: --register-node="true"
Sep 19 01:12:31.077768 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069816    2581 flags.go:64] FLAG: --register-with-taints=""
Sep 19 01:12:31.077768 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069820    2581 flags.go:64] FLAG: --registry-burst="10"
Sep 19 01:12:31.077768 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069822    2581 flags.go:64] FLAG: --registry-qps="5"
Sep 19 01:12:31.077768 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069825    2581 flags.go:64] FLAG: --reserved-cpus=""
Sep 19 01:12:31.077768 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069828    2581 flags.go:64] FLAG: --reserved-memory=""
Sep 19 01:12:31.077768 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069831    2581 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 19 01:12:31.077768 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069834    2581 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 19 01:12:31.078406 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069837    2581 flags.go:64] FLAG: --rotate-certificates="false"
Sep 19 01:12:31.078406 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069840    2581 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 19 01:12:31.078406 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069843    2581 flags.go:64] FLAG: --runonce="false"
Sep 19 01:12:31.078406 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069846    2581 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 19 01:12:31.078406 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069848    2581 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 19 01:12:31.078406 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069851    2581 flags.go:64] FLAG: --seccomp-default="false"
Sep 19 01:12:31.078406 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069854    2581 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 19 01:12:31.078406 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069871    2581 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 19 01:12:31.078406 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069875    2581 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 19 01:12:31.078406 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069878    2581 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 19 01:12:31.078406 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069880    2581 flags.go:64] FLAG: --storage-driver-password="root"
Sep 19 01:12:31.078406 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069883    2581 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 19 01:12:31.078406 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069886    2581 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 19 01:12:31.078406 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069892    2581 flags.go:64] FLAG: --storage-driver-user="root"
Sep 19 01:12:31.078406 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069895    2581 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 19 01:12:31.078406 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069903    2581 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 19 01:12:31.078406 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069906    2581 flags.go:64] FLAG: --system-cgroups=""
Sep 19 01:12:31.078406 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069909    2581 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 19 01:12:31.078406 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069914    2581 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 19 01:12:31.078406 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069917    2581 flags.go:64] FLAG: --tls-cert-file=""
Sep 19 01:12:31.078406 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069919    2581 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 19 01:12:31.078406 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069929    2581 flags.go:64] FLAG: --tls-min-version=""
Sep 19 01:12:31.078406 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069931    2581 flags.go:64] FLAG: --tls-private-key-file=""
Sep 19 01:12:31.078406 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069934    2581 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 19 01:12:31.078406 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069937    2581 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 19 01:12:31.079025 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069940    2581 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 19 01:12:31.079025 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069942    2581 flags.go:64] FLAG: --v="2"
Sep 19 01:12:31.079025 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069946    2581 flags.go:64] FLAG: --version="false"
Sep 19 01:12:31.079025 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069950    2581 flags.go:64] FLAG: --vmodule=""
Sep 19 01:12:31.079025 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069954    2581 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 19 01:12:31.079025 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.069957    2581 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 19 01:12:31.079025 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.070106    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 19 01:12:31.079025 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.070111    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 19 01:12:31.079025 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.070114    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 19 01:12:31.079025 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.070117    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 19 01:12:31.079025 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.070120    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 19 01:12:31.079025 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.070123    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 19 01:12:31.079025 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.070126    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 19 01:12:31.079025 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.070129    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 19 01:12:31.079025 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.070132    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 19 01:12:31.079025 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.070135    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 19 01:12:31.079025 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.070137    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 19 01:12:31.079025 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.070140    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 19 01:12:31.079025 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.070143    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 19 01:12:31.079025 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.070145    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 19 01:12:31.079528 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.070150    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 19 01:12:31.079528 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.070153    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 19 01:12:31.079528 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.070155    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 19 01:12:31.079528 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.070163    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 19 01:12:31.079528 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.070166    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 19 01:12:31.079528 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.070168    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 19 01:12:31.079528 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.070171    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 19 01:12:31.079528 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.070173    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 19 01:12:31.079528 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.070176    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 19 01:12:31.079528 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.070178    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 19 01:12:31.079528 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.070181    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 19 01:12:31.079528 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.070183    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 19 01:12:31.079528 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.070186    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 19 01:12:31.079528 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.070188    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 19 01:12:31.079528 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.070191    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 19 01:12:31.079528 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.070194    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 19 01:12:31.079528 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.070197    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 19 01:12:31.079528 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.070199    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 19 01:12:31.079528 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.070202    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 19 01:12:31.080020 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.070204    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 19 01:12:31.080020 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.070207    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 19 01:12:31.080020 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.070209    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 19 01:12:31.080020 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.070212    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 19 01:12:31.080020 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.070214    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 19 01:12:31.080020 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.070217    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 19 01:12:31.080020 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.070219    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 19 01:12:31.080020 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.070222    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 19 01:12:31.080020 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.070224    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 19 01:12:31.080020 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.070226    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 19 01:12:31.080020 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.070229    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 19 01:12:31.080020 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.070231    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 19 01:12:31.080020 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.070233    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 19 01:12:31.080020 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.070238    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 19 01:12:31.080020 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.070241    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 19 01:12:31.080020 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.070244    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 19 01:12:31.080020 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.070247    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 19 01:12:31.080020 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.070255    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 19 01:12:31.080020 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.070258    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 19 01:12:31.080020 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.070260    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 19 01:12:31.080515 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.070263    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 19 01:12:31.080515 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.070265    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 19 01:12:31.080515 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.070268    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 19 01:12:31.080515 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.070271    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 19 01:12:31.080515 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.070273    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 19 01:12:31.080515 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.070276    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 19 01:12:31.080515 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.070278    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 19 01:12:31.080515 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.070281    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 19 01:12:31.080515 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.070285    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 19 01:12:31.080515 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.070288    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 19 01:12:31.080515 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.070292    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 19 01:12:31.080515 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.070295    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 19 01:12:31.080515 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.070298    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 19 01:12:31.080515 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.070301    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 19 01:12:31.080515 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.070305    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 19 01:12:31.080515 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.070307    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 19 01:12:31.080515 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.070310    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 19 01:12:31.080515 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.070313    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 19 01:12:31.080515 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.070315    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 19 01:12:31.081079 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.070318    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 19 01:12:31.081079 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.070320    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 19 01:12:31.081079 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.070323    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 19 01:12:31.081079 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.070325    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 19 01:12:31.081079 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.070328    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 19 01:12:31.081079 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.070336    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 19 01:12:31.081079 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.070340    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 19 01:12:31.081079 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.070344    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 19 01:12:31.081079 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.070346    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 19 01:12:31.081079 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.070349    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 19 01:12:31.081079 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.070351    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 19 01:12:31.081079 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.070359    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 19 01:12:31.081079 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.070362    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 19 01:12:31.081079 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.070365    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 19 01:12:31.081079 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.070367    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 19 01:12:31.081079 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.070370    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 19 01:12:31.081079 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.070372    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 19 01:12:31.081079 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.070375    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 19 01:12:31.081079 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.070377    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 19 01:12:31.081079 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.070380    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 19 01:12:31.081559 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.070382    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 19 01:12:31.081559 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.070385    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 19 01:12:31.081559 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.070387    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 19 01:12:31.081559 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.070390    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 19 01:12:31.081559 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.070392    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 19 01:12:31.081559 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.070395    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 19 01:12:31.081559 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.070397    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 19 01:12:31.081559 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.070855    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 19 01:12:31.081559 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.079569    2581 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 19 01:12:31.081559 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.079584    2581 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 19 01:12:31.081559 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.079606    2581 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 19 01:12:31.081559 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.079612    2581 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 19 01:12:31.081559 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.079672    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 19 01:12:31.081559 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.079678    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 19 01:12:31.081922 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.079681    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 19 01:12:31.081922 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.079685    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 19 01:12:31.081922 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.079688    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 19 01:12:31.081922 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.079691    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 19 01:12:31.081922 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.079694    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 19 01:12:31.081922 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.079696    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 19 01:12:31.081922 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.079699    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 19 01:12:31.081922 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.079702    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 19 01:12:31.081922 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.079704    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 19 01:12:31.081922 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.079707    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 19 01:12:31.081922 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.079709    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 19 01:12:31.081922 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.079712    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 19 01:12:31.081922 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.079715    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 19 01:12:31.081922 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.079718    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 19 01:12:31.081922 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.079720    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 19 01:12:31.081922 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.079723    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 19 01:12:31.081922 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.079725    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 19 01:12:31.081922 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.079728    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 19 01:12:31.081922 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.079731    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 19 01:12:31.082376 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.079734    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 19 01:12:31.082376 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.079738    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 19 01:12:31.082376 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.079741    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 19 01:12:31.082376 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.079744    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 19 01:12:31.082376 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.079748    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 19 01:12:31.082376 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.079750    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 19 01:12:31.082376 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.079753    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 19 01:12:31.082376 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.079760    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 19 01:12:31.082376 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.079764    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 19 01:12:31.082376 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.079768    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 19 01:12:31.082376 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.079771    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 19 01:12:31.082376 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.079774    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 19 01:12:31.082376 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.079777    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 19 01:12:31.082376 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.079780    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 19 01:12:31.082376 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.079782    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 19 01:12:31.082376 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.079785    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 19 01:12:31.082376 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.079788    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 19 01:12:31.082376 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.079790    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 19 01:12:31.082376 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.079793    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 19 01:12:31.082924 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.079795    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 19 01:12:31.082924 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.079798    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 19 01:12:31.082924 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.079800    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 19 01:12:31.082924 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.079803    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 19 01:12:31.082924 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.079806    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 19 01:12:31.082924 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.079808    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 19 01:12:31.082924 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.079811    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 19 01:12:31.082924 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.079813    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 19 01:12:31.082924 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.079816    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 19 01:12:31.082924 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.079818    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 19 01:12:31.082924 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.079821    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 19 01:12:31.082924 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.079824    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 19 01:12:31.082924 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.079826    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 19 01:12:31.082924 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.079829    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 19 01:12:31.082924 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.079831    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 19 01:12:31.082924 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.079834    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 19 01:12:31.082924 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.079836    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 19 01:12:31.082924 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.079839    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 19 01:12:31.082924 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.079842    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 19 01:12:31.082924 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.079844    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 19 01:12:31.083454 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.079847    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 19 01:12:31.083454 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.079850    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 19 01:12:31.083454 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.079853    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 19 01:12:31.083454 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.079855    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 19 01:12:31.083454 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.079878    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 19 01:12:31.083454 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.079881    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 19 01:12:31.083454 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.079884    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 19 01:12:31.083454 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.079886    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 19 01:12:31.083454 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.079889    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 19 01:12:31.083454 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.079892    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 19 01:12:31.083454 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.079894    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 19 01:12:31.083454 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.079897    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 19 01:12:31.083454 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.079899    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 19 01:12:31.083454 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.079902    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 19 01:12:31.083454 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.079906    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 19 01:12:31.083454 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.079910    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 19 01:12:31.083454 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.079912    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 19 01:12:31.083454 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.079915    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 19 01:12:31.083454 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.079918    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 19 01:12:31.084230 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.079921    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 19 01:12:31.084230 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.079924    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 19 01:12:31.084230 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.079927    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 19 01:12:31.084230 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.079929    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 19 01:12:31.084230 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.079932    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 19 01:12:31.084230 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.079935    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 19 01:12:31.084230 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.079937    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 19 01:12:31.084230 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.079940    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 19 01:12:31.084230 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.079942    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 19 01:12:31.084230 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.079945    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 19 01:12:31.084230 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.079947    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 19 01:12:31.084230 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.079950    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 19 01:12:31.084230 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.079953    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 19 01:12:31.084230 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.079956    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 19 01:12:31.084230 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.079959    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 19 01:12:31.084230 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.079962    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 19 01:12:31.084230 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.079965    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 19 01:12:31.084230 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.079967    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 19 01:12:31.084230 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.079970    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 19 01:12:31.084230 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.079973    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 19 01:12:31.085007 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.079978    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 19 01:12:31.085007 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.080090    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 19 01:12:31.085007 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.080095    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 19 01:12:31.085007 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.080097    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 19 01:12:31.085007 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.080101    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 19 01:12:31.085007 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.080106    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 19 01:12:31.085007 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.080109    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 19 01:12:31.085007 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.080111    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 19 01:12:31.085007 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.080114    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 19 01:12:31.085007 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.080116    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 19 01:12:31.085007 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.080119    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 19 01:12:31.085007 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.080122    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 19 01:12:31.085007 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.080124    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 19 01:12:31.085007 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.080128    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 19 01:12:31.085007 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.080131    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 19 01:12:31.085471 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.080133    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 19 01:12:31.085471 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.080136    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 19 01:12:31.085471 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.080138    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 19 01:12:31.085471 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.080141    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 19 01:12:31.085471 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.080144    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 19 01:12:31.085471 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.080146    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 19 01:12:31.085471 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.080149    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 19 01:12:31.085471 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.080151    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 19 01:12:31.085471 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.080154    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 19 01:12:31.085471 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.080158    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 19 01:12:31.085471 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.080161    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 19 01:12:31.085471 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.080165    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 19 01:12:31.085471 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.080167    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 19 01:12:31.085471 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.080170    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 19 01:12:31.085471 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.080173    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 19 01:12:31.085471 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.080175    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 19 01:12:31.085471 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.080178    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 19 01:12:31.085471 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.080180    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 19 01:12:31.085471 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.080183    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 19 01:12:31.085995 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.080185    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 19 01:12:31.085995 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.080188    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 19 01:12:31.085995 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.080190    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 19 01:12:31.085995 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.080193    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 19 01:12:31.085995 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.080195    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 19 01:12:31.085995 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.080198    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 19 01:12:31.085995 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.080200    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 19 01:12:31.085995 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.080203    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 19 01:12:31.085995 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.080206    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 19 01:12:31.085995 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.080209    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 19 01:12:31.085995 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.080212    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 19 01:12:31.085995 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.080214    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 19 01:12:31.085995 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.080217    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 19 01:12:31.085995 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.080220    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 19 01:12:31.085995 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.080222    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 19 01:12:31.085995 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.080225    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 19 01:12:31.085995 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.080227    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 19 01:12:31.085995 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.080230    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 19 01:12:31.085995 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.080232    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 19 01:12:31.085995 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.080235    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 19 01:12:31.086489 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.080237    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 19 01:12:31.086489 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.080240    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 19 01:12:31.086489 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.080242    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 19 01:12:31.086489 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.080245    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 19 01:12:31.086489 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.080248    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 19 01:12:31.086489 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.080250    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 19 01:12:31.086489 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.080253    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 19 01:12:31.086489 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.080256    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 19 01:12:31.086489 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.080258    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 19 01:12:31.086489 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.080261    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 19 01:12:31.086489 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.080263    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 19 01:12:31.086489 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.080266    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 19 01:12:31.086489 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.080268    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 19 01:12:31.086489 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.080271    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 19 01:12:31.086489 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.080273    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 19 01:12:31.086489 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.080276    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 19 01:12:31.086489 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.080278    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 19 01:12:31.086489 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.080281    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 19 01:12:31.086489 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.080283    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 19 01:12:31.086489 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.080285    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 19 01:12:31.087053 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.080288    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 19 01:12:31.087053 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.080291    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 19 01:12:31.087053 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.080294    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 19 01:12:31.087053 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.080298    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 19 01:12:31.087053 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.080301    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 19 01:12:31.087053 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.080304    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 19 01:12:31.087053 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.080307    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 19 01:12:31.087053 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.080311    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 19 01:12:31.087053 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.080314    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 19 01:12:31.087053 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.080317    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 19 01:12:31.087053 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.080319    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 19 01:12:31.087053 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.080322    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 19 01:12:31.087053 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.080325    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 19 01:12:31.087053 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.080328    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 19 01:12:31.087053 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.080331    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 19 01:12:31.087053 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.080333    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 19 01:12:31.087053 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.080336    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 19 01:12:31.087053 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.080339    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 19 01:12:31.087053 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.080342    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 19 01:12:31.087714 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.080344    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 19 01:12:31.087714 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.080347    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 19 01:12:31.087714 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.080349    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 19 01:12:31.087714 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.080352    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 19 01:12:31.087714 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.080354    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 19 01:12:31.087714 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.080357    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 19 01:12:31.087714 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:31.080359    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 19 01:12:31.087714 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.080364    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 19 01:12:31.087714 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.080474    2581 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 19 01:12:31.087714 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.082425    2581 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 19 01:12:31.087714 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.083432    2581 server.go:1019] "Starting client certificate rotation"
Sep 19 01:12:31.087714 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.083580    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 19 01:12:31.087714 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.083631    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 19 01:12:31.088097 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.085655    2581 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 19 01:12:31.088097 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.086894    2581 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 19 01:12:31.094642 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.094627    2581 log.go:25] "Validated CRI v1 runtime API"
Sep 19 01:12:31.096371 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.096356    2581 log.go:25] "Validated CRI v1 image API"
Sep 19 01:12:31.098586 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.098569    2581 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 19 01:12:31.100734 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.100709    2581 fs.go:135] Filesystem UUIDs: map[6295aa9e-98e7-44fb-8034-a2cb71664189:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 80a9707c-bd46-47b4-86a4-2394ac74b4c8:/dev/nvme0n1p3]
Sep 19 01:12:31.100809 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.100733    2581 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 19 01:12:31.107065 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.106927    2581 manager.go:216] Machine: {Timestamp:2026-09-19 01:12:31.106395597 +0000 UTC m=+1.994064665 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102348 MemoryCapacity:32812150784 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2dadc3282b5c25946014fef8eaa485 SystemUUID:ec2dadc3-282b-5c25-9460-14fef8eaa485 BootID:12c2813e-ffb0-4993-8c67-901a6f03f81a Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406073344 Type:vfs Inodes:4005389 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:de:42:50:dd Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:de:42:50:dd Speed:0 Mtu:9001} {Name:ovs-system MacAddress:56:8b:67:c4:8c:96 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812150784 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 19 01:12:31.107175 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.107053    2581 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 19 01:12:31.107175 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.107147    2581 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 19 01:12:31.108694 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.108668    2581 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 19 01:12:31.108819 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.108697    2581 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-113.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 19 01:12:31.109714 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.109704    2581 topology_manager.go:138] "Creating topology manager with none policy"
Sep 19 01:12:31.109747 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.109716    2581 container_manager_linux.go:309] "Creating device plugin manager"
Sep 19 01:12:31.109747 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.109724    2581 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 19 01:12:31.109747 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.109734    2581 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 19 01:12:31.109829 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.109798    2581 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 19 01:12:31.113782 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.113770    2581 state_mem.go:36] "Initialized new in-memory state store"
Sep 19 01:12:31.113917 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.113907    2581 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 19 01:12:31.113963 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.113942    2581 kubelet.go:486] "Attempting to sync node with API server"
Sep 19 01:12:31.113963 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.113952    2581 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 19 01:12:31.114031 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.113966    2581 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 19 01:12:31.114031 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.113974    2581 kubelet.go:398] "Adding apiserver pod source"
Sep 19 01:12:31.114031 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.113983    2581 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 19 01:12:31.114412 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.114399    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 19 01:12:31.114465 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.114427    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 19 01:12:31.115939 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.115920    2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 19 01:12:31.116530 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.116513    2581 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 19 01:12:31.119689 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.118786    2581 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 19 01:12:31.119689 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.118817    2581 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 19 01:12:31.119835 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.119823    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 19 01:12:31.119894 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.119841    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 19 01:12:31.119894 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.119847    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 19 01:12:31.119894 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.119852    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 19 01:12:31.119894 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.119879    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 19 01:12:31.119894 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.119885    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 19 01:12:31.120030 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.119899    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 19 01:12:31.120030 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.119904    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 19 01:12:31.120030 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.119910    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 19 01:12:31.120030 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.119916    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 19 01:12:31.120030 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.119927    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 19 01:12:31.120030 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.119933    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 19 01:12:31.120030 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.119961    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 19 01:12:31.120030 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.119966    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 19 01:12:31.124955 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.124929    2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-113.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 19 01:12:31.125071 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:31.124970    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-113.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 19 01:12:31.125071 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:31.125042    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 19 01:12:31.126560 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.126547    2581 server.go:1268] "Started kubelet"
Sep 19 01:12:31.126633 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.126612    2581 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 19 01:12:31.126755 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.126705    2581 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 19 01:12:31.126836 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.126791    2581 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 19 01:12:31.127816 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.127799    2581 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 19 01:12:31.127920 ip-10-0-129-113 systemd[1]: Started Kubernetes Kubelet.
Sep 19 01:12:31.128980 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.128799    2581 server.go:310] "Adding debug handlers to kubelet server"
Sep 19 01:12:31.130895 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:31.130848    2581 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 19 01:12:31.135232 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.135218    2581 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 19 01:12:31.135285 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.135234    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 19 01:12:31.135390 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:31.135304    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-113.ec2.internal\" not found"
Sep 19 01:12:31.135897 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.135843    2581 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 19 01:12:31.135897 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.135845    2581 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 19 01:12:31.136028 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.135902    2581 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 19 01:12:31.136078 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.136060    2581 reconstruct.go:97] "Volume reconstruction finished"
Sep 19 01:12:31.136078 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.136069    2581 reconciler.go:29] "Reconciler: start to sync state"
Sep 19 01:12:31.136526 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.136489    2581 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 19 01:12:31.136526 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.136527    2581 factory.go:55] Registering systemd factory
Sep 19 01:12:31.136647 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.136535    2581 factory.go:223] Registration of the systemd container factory successfully
Sep 19 01:12:31.136757 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.136741    2581 factory.go:153] Registering CRI-O factory
Sep 19 01:12:31.136815 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.136759    2581 factory.go:223] Registration of the crio container factory successfully
Sep 19 01:12:31.136815 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.136775    2581 factory.go:103] Registering Raw factory
Sep 19 01:12:31.136815 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.136787    2581 manager.go:1196] Started watching for new ooms in manager
Sep 19 01:12:31.137551 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.137534    2581 manager.go:349] Starting recovery of all containers
Sep 19 01:12:31.137648 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:31.137630    2581 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-113.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 19 01:12:31.154545 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.154518    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-rvpnk"
Sep 19 01:12:31.154626 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.154548    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-rvpnk"
Sep 19 01:12:31.161042 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.161001    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 19 01:12:31.164750 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.164733    2581 manager.go:354] Recovery completed
Sep 19 01:12:31.166425 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.166406    2581 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 19 01:12:31.169149 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.169133    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 19 01:12:31.170877 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.170843    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-113.ec2.internal" event="NodeHasSufficientMemory"
Sep 19 01:12:31.170950 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.170891    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-113.ec2.internal" event="NodeHasNoDiskPressure"
Sep 19 01:12:31.170950 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.170904    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-113.ec2.internal" event="NodeHasSufficientPID"
Sep 19 01:12:31.171370 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.171357    2581 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 19 01:12:31.171370 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.171370    2581 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 19 01:12:31.171495 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.171387    2581 state_mem.go:36] "Initialized new in-memory state store"
Sep 19 01:12:31.171495 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.171359    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 19 01:12:31.172097 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.172083    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-113.ec2.internal" event="NodeHasSufficientMemory"
Sep 19 01:12:31.172166 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.172111    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-113.ec2.internal" event="NodeHasNoDiskPressure"
Sep 19 01:12:31.172166 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.172124    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-113.ec2.internal" event="NodeHasSufficientPID"
Sep 19 01:12:31.173433 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.173421    2581 policy_none.go:49] "None policy: Start"
Sep 19 01:12:31.173490 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.173439    2581 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 19 01:12:31.173490 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.173452    2581 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 19 01:12:31.175331 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.175319    2581 policy_none.go:47] "Start"
Sep 19 01:12:31.212450 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.212436    2581 manager.go:337] "Starting Device Plugin manager"
Sep 19 01:12:31.212526 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:31.212467    2581 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 19 01:12:31.212526 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.212476    2581 server.go:87] "Starting device plugin registration server"
Sep 19 01:12:31.212728 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.212700    2581 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 19 01:12:31.212813 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.212731    2581 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 19 01:12:31.212984 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.212960    2581 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 19 01:12:31.213055 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.213048    2581 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 19 01:12:31.213114 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.213057    2581 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 19 01:12:31.213459 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:31.213440    2581 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 19 01:12:31.213535 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:31.213504    2581 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-113.ec2.internal\" not found"
Sep 19 01:12:31.265706 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.265684    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 19 01:12:31.267715 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.265729    2581 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 19 01:12:31.267715 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.265748    2581 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 19 01:12:31.267715 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:31.265781    2581 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 19 01:12:31.268818 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.268800    2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 19 01:12:31.313371 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.313248    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 19 01:12:31.314192 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.314153    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-113.ec2.internal" event="NodeHasSufficientMemory"
Sep 19 01:12:31.314192 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.314185    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-113.ec2.internal" event="NodeHasNoDiskPressure"
Sep 19 01:12:31.314281 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.314194    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-113.ec2.internal" event="NodeHasSufficientPID"
Sep 19 01:12:31.314281 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.314213    2581 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-113.ec2.internal"
Sep 19 01:12:31.323039 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.323019    2581 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-113.ec2.internal"
Sep 19 01:12:31.323115 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:31.323040    2581 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-113.ec2.internal\": node \"ip-10-0-129-113.ec2.internal\" not found"
Sep 19 01:12:31.343523 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:31.343501    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-113.ec2.internal\" not found"
Sep 19 01:12:31.366769 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.366751    2581 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-113.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-113.ec2.internal"]
Sep 19 01:12:31.366825 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.366812    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 19 01:12:31.368072 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.368051    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-113.ec2.internal" event="NodeHasSufficientMemory"
Sep 19 01:12:31.368148 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.368076    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-113.ec2.internal" event="NodeHasNoDiskPressure"
Sep 19 01:12:31.368148 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.368095    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-113.ec2.internal" event="NodeHasSufficientPID"
Sep 19 01:12:31.370309 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.370298    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 19 01:12:31.370489 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.370475    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-113.ec2.internal"
Sep 19 01:12:31.370544 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.370502    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 19 01:12:31.371001 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.370985    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-113.ec2.internal" event="NodeHasSufficientMemory"
Sep 19 01:12:31.371070 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.371010    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-113.ec2.internal" event="NodeHasSufficientMemory"
Sep 19 01:12:31.371070 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.371029    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-113.ec2.internal" event="NodeHasNoDiskPressure"
Sep 19 01:12:31.371070 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.371033    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-113.ec2.internal" event="NodeHasNoDiskPressure"
Sep 19 01:12:31.371070 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.371040    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-113.ec2.internal" event="NodeHasSufficientPID"
Sep 19 01:12:31.371070 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.371044    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-113.ec2.internal" event="NodeHasSufficientPID"
Sep 19 01:12:31.373261 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.373240    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-113.ec2.internal"
Sep 19 01:12:31.373339 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.373267    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 19 01:12:31.373930 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.373915    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-113.ec2.internal" event="NodeHasSufficientMemory"
Sep 19 01:12:31.374002 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.373936    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-113.ec2.internal" event="NodeHasNoDiskPressure"
Sep 19 01:12:31.374002 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.373947    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-113.ec2.internal" event="NodeHasSufficientPID"
Sep 19 01:12:31.385441 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:31.385416    2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-113.ec2.internal\" not found" node="ip-10-0-129-113.ec2.internal"
Sep 19 01:12:31.389200 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:31.389183    2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-113.ec2.internal\" not found" node="ip-10-0-129-113.ec2.internal"
Sep 19 01:12:31.437294 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.437277    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/76eada0400f575bdcabf278316c26ab9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-113.ec2.internal\" (UID: \"76eada0400f575bdcabf278316c26ab9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-113.ec2.internal"
Sep 19 01:12:31.437351 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.437304    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/76eada0400f575bdcabf278316c26ab9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-113.ec2.internal\" (UID: \"76eada0400f575bdcabf278316c26ab9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-113.ec2.internal"
Sep 19 01:12:31.437351 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.437329    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bf17d6182a2ab91b4c163d92cc1c844e-config\") pod \"kube-apiserver-proxy-ip-10-0-129-113.ec2.internal\" (UID: \"bf17d6182a2ab91b4c163d92cc1c844e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-113.ec2.internal"
Sep 19 01:12:31.444373 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:31.444348    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-113.ec2.internal\" not found"
Sep 19 01:12:31.537741 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.537718    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/76eada0400f575bdcabf278316c26ab9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-113.ec2.internal\" (UID: \"76eada0400f575bdcabf278316c26ab9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-113.ec2.internal"
Sep 19 01:12:31.537836 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.537747    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/76eada0400f575bdcabf278316c26ab9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-113.ec2.internal\" (UID: \"76eada0400f575bdcabf278316c26ab9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-113.ec2.internal"
Sep 19 01:12:31.537836 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.537794    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bf17d6182a2ab91b4c163d92cc1c844e-config\") pod \"kube-apiserver-proxy-ip-10-0-129-113.ec2.internal\" (UID: \"bf17d6182a2ab91b4c163d92cc1c844e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-113.ec2.internal"
Sep 19 01:12:31.537836 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.537822    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/76eada0400f575bdcabf278316c26ab9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-113.ec2.internal\" (UID: \"76eada0400f575bdcabf278316c26ab9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-113.ec2.internal"
Sep 19 01:12:31.537973 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.537822    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/76eada0400f575bdcabf278316c26ab9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-113.ec2.internal\" (UID: \"76eada0400f575bdcabf278316c26ab9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-113.ec2.internal"
Sep 19 01:12:31.537973 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.537827    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bf17d6182a2ab91b4c163d92cc1c844e-config\") pod \"kube-apiserver-proxy-ip-10-0-129-113.ec2.internal\" (UID: \"bf17d6182a2ab91b4c163d92cc1c844e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-113.ec2.internal"
Sep 19 01:12:31.544845 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:31.544831    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-113.ec2.internal\" not found"
Sep 19 01:12:31.645923 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:31.645873    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-113.ec2.internal\" not found"
Sep 19 01:12:31.689038 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.689015    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-113.ec2.internal"
Sep 19 01:12:31.692176 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:31.692161    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-113.ec2.internal"
Sep 19 01:12:31.746103 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:31.746084    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-113.ec2.internal\" not found"
Sep 19 01:12:31.846582 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:31.846560    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-113.ec2.internal\" not found"
Sep 19 01:12:31.947176 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:31.947106    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-113.ec2.internal\" not found"
Sep 19 01:12:32.047619 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:32.047586    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-113.ec2.internal\" not found"
Sep 19 01:12:32.083764 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:32.083736    2581 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 19 01:12:32.084207 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:32.083903    2581 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 19 01:12:32.084207 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:32.083927    2581 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 19 01:12:32.135363 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:32.135334    2581 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 19 01:12:32.148404 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:32.148383    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-113.ec2.internal\" not found"
Sep 19 01:12:32.159628 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:32.159595    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-18 01:07:31 +0000 UTC" deadline="2028-05-09 04:26:52.562700249 +0000 UTC"
Sep 19 01:12:32.159628 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:32.159622    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14355h14m20.403080836s"
Sep 19 01:12:32.249327 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:32.249207    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-113.ec2.internal\" not found"
Sep 19 01:12:32.281712 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:32.281679    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf17d6182a2ab91b4c163d92cc1c844e.slice/crio-946e4acbbd3d5f9bd3726d1bbfbcddb78c562c2b186421b4fc3d113b36b72b62 WatchSource:0}: Error finding container 946e4acbbd3d5f9bd3726d1bbfbcddb78c562c2b186421b4fc3d113b36b72b62: Status 404 returned error can't find the container with id 946e4acbbd3d5f9bd3726d1bbfbcddb78c562c2b186421b4fc3d113b36b72b62
Sep 19 01:12:32.282227 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:32.282205    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76eada0400f575bdcabf278316c26ab9.slice/crio-a8a8aa43c60497f474854ba7e8c26624ad7484f84b890fadaba975bec477f212 WatchSource:0}: Error finding container a8a8aa43c60497f474854ba7e8c26624ad7484f84b890fadaba975bec477f212: Status 404 returned error can't find the container with id a8a8aa43c60497f474854ba7e8c26624ad7484f84b890fadaba975bec477f212
Sep 19 01:12:32.287238 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:32.287224    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 19 01:12:32.349560 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:32.349535    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-113.ec2.internal\" not found"
Sep 19 01:12:32.450011 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:32.449986    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-113.ec2.internal\" not found"
Sep 19 01:12:32.550479 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:32.550424    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-113.ec2.internal\" not found"
Sep 19 01:12:32.618059 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:32.618039    2581 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 19 01:12:32.635632 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:32.635602    2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-113.ec2.internal"
Sep 19 01:12:32.661734 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:32.661714    2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 19 01:12:32.662553 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:32.662540    2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-113.ec2.internal"
Sep 19 01:12:32.681165 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:32.681144    2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 19 01:12:33.115499 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.115469    2581 apiserver.go:52] "Watching apiserver"
Sep 19 01:12:33.122357 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.122333    2581 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 19 01:12:33.122901 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.122845    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-113.ec2.internal","openshift-multus/multus-kwlqw","openshift-multus/network-metrics-daemon-dkpb2","openshift-network-operator/iptables-alerter-xmldk","openshift-ovn-kubernetes/ovnkube-node-x785d","kube-system/global-pull-secret-syncer-hrgqz","kube-system/kube-apiserver-proxy-ip-10-0-129-113.ec2.internal","openshift-cluster-node-tuning-operator/tuned-h94dv","openshift-image-registry/node-ca-vrlkw","openshift-multus/multus-additional-cni-plugins-42mjj","openshift-network-diagnostics/network-check-target-lmcnn","kube-system/konnectivity-agent-8ndtv","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lsdhk","openshift-dns/node-resolver-nx2sn"]
Sep 19 01:12:33.125597 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.125577    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-42mjj"
Sep 19 01:12:33.127831 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.127802    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 19 01:12:33.127946 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.127929    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lsdhk"
Sep 19 01:12:33.128224 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.128205    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 19 01:12:33.128566 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.128551    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-9pxzt\""
Sep 19 01:12:33.128742 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.128719    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 19 01:12:33.128793 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.128722    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 19 01:12:33.129034 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.129012    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 19 01:12:33.130135 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.130119    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dkpb2"
Sep 19 01:12:33.130229 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:33.130181    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dkpb2" podUID="3884e026-09ad-4bc8-bcb8-ae622d873aea"
Sep 19 01:12:33.132557 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.132537    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xmldk"
Sep 19 01:12:33.135900 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.135398    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 19 01:12:33.135900 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.135485    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 19 01:12:33.135900 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.135494    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-x785d"
Sep 19 01:12:33.135900 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.135582    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-vxt45\""
Sep 19 01:12:33.135900 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.135612    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 19 01:12:33.135900 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.135797    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 19 01:12:33.137272 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.136405    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 19 01:12:33.137272 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.136452    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-gsx45\""
Sep 19 01:12:33.137272 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.136692    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 19 01:12:33.137272 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.137062    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 19 01:12:33.138932 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.138752    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 19 01:12:33.140270 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.140230    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 19 01:12:33.143530 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.140466    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 19 01:12:33.143530 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.142047    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 19 01:12:33.143530 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.142629    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 19 01:12:33.143530 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.142896    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 19 01:12:33.146070 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.144782    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrgqz"
Sep 19 01:12:33.146070 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:33.144877    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrgqz" podUID="2ff7ef56-5c24-40a5-84d6-7259cf83a29c"
Sep 19 01:12:33.146070 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.145274    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vrlkw"
Sep 19 01:12:33.147219 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.147200    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d16ba40c-ab77-4204-b80e-d58f3a57e49a-cni-binary-copy\") pod \"multus-additional-cni-plugins-42mjj\" (UID: \"d16ba40c-ab77-4204-b80e-d58f3a57e49a\") " pod="openshift-multus/multus-additional-cni-plugins-42mjj"
Sep 19 01:12:33.147311 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.147239    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d16ba40c-ab77-4204-b80e-d58f3a57e49a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-42mjj\" (UID: \"d16ba40c-ab77-4204-b80e-d58f3a57e49a\") " pod="openshift-multus/multus-additional-cni-plugins-42mjj"
Sep 19 01:12:33.147311 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.147269    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a190ceb2-3036-4697-89d0-e103dca9cf5a-registration-dir\") pod \"aws-ebs-csi-driver-node-lsdhk\" (UID: \"a190ceb2-3036-4697-89d0-e103dca9cf5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lsdhk"
Sep 19 01:12:33.147427 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.147337    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmmbq\" (UniqueName: \"kubernetes.io/projected/3884e026-09ad-4bc8-bcb8-ae622d873aea-kube-api-access-mmmbq\") pod \"network-metrics-daemon-dkpb2\" (UID: \"3884e026-09ad-4bc8-bcb8-ae622d873aea\") " pod="openshift-multus/network-metrics-daemon-dkpb2"
Sep 19 01:12:33.147478 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.147419    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f759a12f-5b6f-4924-96f2-dd5e00fc1348-log-socket\") pod \"ovnkube-node-x785d\" (UID: \"f759a12f-5b6f-4924-96f2-dd5e00fc1348\") " pod="openshift-ovn-kubernetes/ovnkube-node-x785d"
Sep 19 01:12:33.148347 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.147852    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f759a12f-5b6f-4924-96f2-dd5e00fc1348-ovnkube-script-lib\") pod \"ovnkube-node-x785d\" (UID: \"f759a12f-5b6f-4924-96f2-dd5e00fc1348\") " pod="openshift-ovn-kubernetes/ovnkube-node-x785d"
Sep 19 01:12:33.148347 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.147944    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2ff7ef56-5c24-40a5-84d6-7259cf83a29c-original-pull-secret\") pod \"global-pull-secret-syncer-hrgqz\" (UID: \"2ff7ef56-5c24-40a5-84d6-7259cf83a29c\") " pod="kube-system/global-pull-secret-syncer-hrgqz"
Sep 19 01:12:33.148347 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.147978    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a190ceb2-3036-4697-89d0-e103dca9cf5a-socket-dir\") pod \"aws-ebs-csi-driver-node-lsdhk\" (UID: \"a190ceb2-3036-4697-89d0-e103dca9cf5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lsdhk"
Sep 19 01:12:33.148347 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.148010    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a190ceb2-3036-4697-89d0-e103dca9cf5a-sys-fs\") pod \"aws-ebs-csi-driver-node-lsdhk\" (UID: \"a190ceb2-3036-4697-89d0-e103dca9cf5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lsdhk"
Sep 19 01:12:33.148347 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.148037    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3884e026-09ad-4bc8-bcb8-ae622d873aea-metrics-certs\") pod \"network-metrics-daemon-dkpb2\" (UID: \"3884e026-09ad-4bc8-bcb8-ae622d873aea\") " pod="openshift-multus/network-metrics-daemon-dkpb2"
Sep 19 01:12:33.148347 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.148061    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-smspl\""
Sep 19 01:12:33.148347 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.148105    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f759a12f-5b6f-4924-96f2-dd5e00fc1348-host-run-netns\") pod \"ovnkube-node-x785d\" (UID: \"f759a12f-5b6f-4924-96f2-dd5e00fc1348\") " pod="openshift-ovn-kubernetes/ovnkube-node-x785d"
Sep 19 01:12:33.148347 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.148142    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d16ba40c-ab77-4204-b80e-d58f3a57e49a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-42mjj\" (UID: \"d16ba40c-ab77-4204-b80e-d58f3a57e49a\") " pod="openshift-multus/multus-additional-cni-plugins-42mjj"
Sep 19 01:12:33.148347 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.148172    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wt9cj\" (UniqueName: \"kubernetes.io/projected/caf84f55-01ed-4c93-81d0-f8e545bf90bf-kube-api-access-wt9cj\") pod \"iptables-alerter-xmldk\" (UID: \"caf84f55-01ed-4c93-81d0-f8e545bf90bf\") " pod="openshift-network-operator/iptables-alerter-xmldk"
Sep 19 01:12:33.148347 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.148202    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f759a12f-5b6f-4924-96f2-dd5e00fc1348-run-openvswitch\") pod \"ovnkube-node-x785d\" (UID: \"f759a12f-5b6f-4924-96f2-dd5e00fc1348\") " pod="openshift-ovn-kubernetes/ovnkube-node-x785d"
Sep 19 01:12:33.148347 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.148227    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f759a12f-5b6f-4924-96f2-dd5e00fc1348-host-run-ovn-kubernetes\") pod \"ovnkube-node-x785d\" (UID: \"f759a12f-5b6f-4924-96f2-dd5e00fc1348\") " pod="openshift-ovn-kubernetes/ovnkube-node-x785d"
Sep 19 01:12:33.148347 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.148256    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f759a12f-5b6f-4924-96f2-dd5e00fc1348-ovnkube-config\") pod \"ovnkube-node-x785d\" (UID: \"f759a12f-5b6f-4924-96f2-dd5e00fc1348\") " pod="openshift-ovn-kubernetes/ovnkube-node-x785d"
Sep 19 01:12:33.148347 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.148284    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/caf84f55-01ed-4c93-81d0-f8e545bf90bf-host-slash\") pod \"iptables-alerter-xmldk\" (UID: \"caf84f55-01ed-4c93-81d0-f8e545bf90bf\") " pod="openshift-network-operator/iptables-alerter-xmldk"
Sep 19 01:12:33.148347 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.148348    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f759a12f-5b6f-4924-96f2-dd5e00fc1348-host-kubelet\") pod \"ovnkube-node-x785d\" (UID: \"f759a12f-5b6f-4924-96f2-dd5e00fc1348\") " pod="openshift-ovn-kubernetes/ovnkube-node-x785d"
Sep 19 01:12:33.148980 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.148387    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d16ba40c-ab77-4204-b80e-d58f3a57e49a-system-cni-dir\") pod \"multus-additional-cni-plugins-42mjj\" (UID: \"d16ba40c-ab77-4204-b80e-d58f3a57e49a\") " pod="openshift-multus/multus-additional-cni-plugins-42mjj"
Sep 19 01:12:33.148980 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.148421    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/caf84f55-01ed-4c93-81d0-f8e545bf90bf-iptables-alerter-script\") pod \"iptables-alerter-xmldk\" (UID: \"caf84f55-01ed-4c93-81d0-f8e545bf90bf\") " pod="openshift-network-operator/iptables-alerter-xmldk"
Sep 19 01:12:33.148980 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.148448    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f759a12f-5b6f-4924-96f2-dd5e00fc1348-systemd-units\") pod \"ovnkube-node-x785d\" (UID: \"f759a12f-5b6f-4924-96f2-dd5e00fc1348\") " pod="openshift-ovn-kubernetes/ovnkube-node-x785d"
Sep 19 01:12:33.148980 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.148489    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f759a12f-5b6f-4924-96f2-dd5e00fc1348-host-slash\") pod \"ovnkube-node-x785d\" (UID: \"f759a12f-5b6f-4924-96f2-dd5e00fc1348\") " pod="openshift-ovn-kubernetes/ovnkube-node-x785d"
Sep 19 01:12:33.148980 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.148585    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-nx2sn"
Sep 19 01:12:33.148980 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.148559    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f759a12f-5b6f-4924-96f2-dd5e00fc1348-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x785d\" (UID: \"f759a12f-5b6f-4924-96f2-dd5e00fc1348\") " pod="openshift-ovn-kubernetes/ovnkube-node-x785d"
Sep 19 01:12:33.148980 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.148645    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d16ba40c-ab77-4204-b80e-d58f3a57e49a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-42mjj\" (UID: \"d16ba40c-ab77-4204-b80e-d58f3a57e49a\") " pod="openshift-multus/multus-additional-cni-plugins-42mjj"
Sep 19 01:12:33.148980 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.148692    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85ljl\" (UniqueName: \"kubernetes.io/projected/d16ba40c-ab77-4204-b80e-d58f3a57e49a-kube-api-access-85ljl\") pod \"multus-additional-cni-plugins-42mjj\" (UID: \"d16ba40c-ab77-4204-b80e-d58f3a57e49a\") " pod="openshift-multus/multus-additional-cni-plugins-42mjj"
Sep 19 01:12:33.148980 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.148756    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f759a12f-5b6f-4924-96f2-dd5e00fc1348-env-overrides\") pod \"ovnkube-node-x785d\" (UID: \"f759a12f-5b6f-4924-96f2-dd5e00fc1348\") " pod="openshift-ovn-kubernetes/ovnkube-node-x785d"
Sep 19 01:12:33.148980 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.148839    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w75qs\" (UniqueName: \"kubernetes.io/projected/f759a12f-5b6f-4924-96f2-dd5e00fc1348-kube-api-access-w75qs\") pod \"ovnkube-node-x785d\" (UID: \"f759a12f-5b6f-4924-96f2-dd5e00fc1348\") " pod="openshift-ovn-kubernetes/ovnkube-node-x785d"
Sep 19 01:12:33.148980 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.148906    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d16ba40c-ab77-4204-b80e-d58f3a57e49a-cnibin\") pod \"multus-additional-cni-plugins-42mjj\" (UID: \"d16ba40c-ab77-4204-b80e-d58f3a57e49a\") " pod="openshift-multus/multus-additional-cni-plugins-42mjj"
Sep 19 01:12:33.148980 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.148955    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a190ceb2-3036-4697-89d0-e103dca9cf5a-device-dir\") pod \"aws-ebs-csi-driver-node-lsdhk\" (UID: \"a190ceb2-3036-4697-89d0-e103dca9cf5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lsdhk"
Sep 19 01:12:33.149448 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.149011    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fp5q\" (UniqueName: \"kubernetes.io/projected/a190ceb2-3036-4697-89d0-e103dca9cf5a-kube-api-access-6fp5q\") pod \"aws-ebs-csi-driver-node-lsdhk\" (UID: \"a190ceb2-3036-4697-89d0-e103dca9cf5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lsdhk"
Sep 19 01:12:33.149448 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.149091    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2ff7ef56-5c24-40a5-84d6-7259cf83a29c-dbus\") pod \"global-pull-secret-syncer-hrgqz\" (UID: \"2ff7ef56-5c24-40a5-84d6-7259cf83a29c\") " pod="kube-system/global-pull-secret-syncer-hrgqz"
Sep 19 01:12:33.149448 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.149122    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f759a12f-5b6f-4924-96f2-dd5e00fc1348-run-systemd\") pod \"ovnkube-node-x785d\" (UID: \"f759a12f-5b6f-4924-96f2-dd5e00fc1348\") " pod="openshift-ovn-kubernetes/ovnkube-node-x785d"
Sep 19 01:12:33.149448 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.149155    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f759a12f-5b6f-4924-96f2-dd5e00fc1348-var-lib-openvswitch\") pod \"ovnkube-node-x785d\" (UID: \"f759a12f-5b6f-4924-96f2-dd5e00fc1348\") " pod="openshift-ovn-kubernetes/ovnkube-node-x785d"
Sep 19 01:12:33.149448 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.149209    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f759a12f-5b6f-4924-96f2-dd5e00fc1348-run-ovn\") pod \"ovnkube-node-x785d\" (UID: \"f759a12f-5b6f-4924-96f2-dd5e00fc1348\") " pod="openshift-ovn-kubernetes/ovnkube-node-x785d"
Sep 19 01:12:33.149448 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.149241    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f759a12f-5b6f-4924-96f2-dd5e00fc1348-node-log\") pod \"ovnkube-node-x785d\" (UID: \"f759a12f-5b6f-4924-96f2-dd5e00fc1348\") " pod="openshift-ovn-kubernetes/ovnkube-node-x785d"
Sep 19 01:12:33.149448 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.149270    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2ff7ef56-5c24-40a5-84d6-7259cf83a29c-kubelet-config\") pod \"global-pull-secret-syncer-hrgqz\" (UID: \"2ff7ef56-5c24-40a5-84d6-7259cf83a29c\") " pod="kube-system/global-pull-secret-syncer-hrgqz"
Sep 19 01:12:33.149448 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.149295    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a190ceb2-3036-4697-89d0-e103dca9cf5a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lsdhk\" (UID: \"a190ceb2-3036-4697-89d0-e103dca9cf5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lsdhk"
Sep 19 01:12:33.149448 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.149323    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a190ceb2-3036-4697-89d0-e103dca9cf5a-etc-selinux\") pod \"aws-ebs-csi-driver-node-lsdhk\" (UID: \"a190ceb2-3036-4697-89d0-e103dca9cf5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lsdhk"
Sep 19 01:12:33.149448 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.149424    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f759a12f-5b6f-4924-96f2-dd5e00fc1348-etc-openvswitch\") pod \"ovnkube-node-x785d\" (UID: \"f759a12f-5b6f-4924-96f2-dd5e00fc1348\") " pod="openshift-ovn-kubernetes/ovnkube-node-x785d"
Sep 19 01:12:33.150084 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.149455    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f759a12f-5b6f-4924-96f2-dd5e00fc1348-host-cni-bin\") pod \"ovnkube-node-x785d\" (UID: \"f759a12f-5b6f-4924-96f2-dd5e00fc1348\") " pod="openshift-ovn-kubernetes/ovnkube-node-x785d"
Sep 19 01:12:33.150084 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.149481    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f759a12f-5b6f-4924-96f2-dd5e00fc1348-host-cni-netd\") pod \"ovnkube-node-x785d\" (UID: \"f759a12f-5b6f-4924-96f2-dd5e00fc1348\") " pod="openshift-ovn-kubernetes/ovnkube-node-x785d"
Sep 19 01:12:33.150084 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.149550    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f759a12f-5b6f-4924-96f2-dd5e00fc1348-ovn-node-metrics-cert\") pod \"ovnkube-node-x785d\" (UID: \"f759a12f-5b6f-4924-96f2-dd5e00fc1348\") " pod="openshift-ovn-kubernetes/ovnkube-node-x785d"
Sep 19 01:12:33.150499 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.150477    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-5zjgs\""
Sep 19 01:12:33.150626 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.150614    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 19 01:12:33.151382 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.151362    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-kwlqw"
Sep 19 01:12:33.151797 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.151771    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 19 01:12:33.151933 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.151779    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 19 01:12:33.152419 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.152257    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 19 01:12:33.152419 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.152257    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 19 01:12:33.152419 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.152344    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-l27rc\""
Sep 19 01:12:33.153580 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.153535    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-h94dv"
Sep 19 01:12:33.154203 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.153950    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 19 01:12:33.154203 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.154062    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-lwkns\""
Sep 19 01:12:33.155996 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.155975    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lmcnn"
Sep 19 01:12:33.156094 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:33.156061    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lmcnn" podUID="b0be445f-6160-4532-8a71-55bfdf782241"
Sep 19 01:12:33.158192 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.158170    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8ndtv"
Sep 19 01:12:33.159474 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.158919    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 19 01:12:33.159474 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.158993    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7l5hk\""
Sep 19 01:12:33.159474 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.159185    2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 19 01:12:33.159474 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.159391    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 19 01:12:33.160304 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.160197    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 19 01:12:33.160410 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.160339    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-f5z2m\""
Sep 19 01:12:33.160667 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.160648    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 19 01:12:33.183099 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.183078    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-65grd"
Sep 19 01:12:33.195343 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.195015    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-65grd"
Sep 19 01:12:33.236959 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.236943    2581 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 19 01:12:33.249836 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.249809    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f759a12f-5b6f-4924-96f2-dd5e00fc1348-host-kubelet\") pod \"ovnkube-node-x785d\" (UID: \"f759a12f-5b6f-4924-96f2-dd5e00fc1348\") " pod="openshift-ovn-kubernetes/ovnkube-node-x785d"
Sep 19 01:12:33.249951 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.249847    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d16ba40c-ab77-4204-b80e-d58f3a57e49a-system-cni-dir\") pod \"multus-additional-cni-plugins-42mjj\" (UID: \"d16ba40c-ab77-4204-b80e-d58f3a57e49a\") " pod="openshift-multus/multus-additional-cni-plugins-42mjj"
Sep 19 01:12:33.249951 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.249892    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/62df040a-200c-4054-806f-252a833df006-multus-conf-dir\") pod \"multus-kwlqw\" (UID: \"62df040a-200c-4054-806f-252a833df006\") " pod="openshift-multus/multus-kwlqw"
Sep 19 01:12:33.249951 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.249921    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f759a12f-5b6f-4924-96f2-dd5e00fc1348-systemd-units\") pod \"ovnkube-node-x785d\" (UID: \"f759a12f-5b6f-4924-96f2-dd5e00fc1348\") " pod="openshift-ovn-kubernetes/ovnkube-node-x785d"
Sep 19 01:12:33.250103 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.249954    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f759a12f-5b6f-4924-96f2-dd5e00fc1348-host-slash\") pod \"ovnkube-node-x785d\" (UID: \"f759a12f-5b6f-4924-96f2-dd5e00fc1348\") " pod="openshift-ovn-kubernetes/ovnkube-node-x785d"
Sep 19 01:12:33.250103 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.249979    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f759a12f-5b6f-4924-96f2-dd5e00fc1348-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x785d\" (UID: \"f759a12f-5b6f-4924-96f2-dd5e00fc1348\") " pod="openshift-ovn-kubernetes/ovnkube-node-x785d"
Sep 19 01:12:33.250103 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.250004    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ff802e32-5c32-466d-b76f-252275c6fb82-host\") pod \"node-ca-vrlkw\" (UID: \"ff802e32-5c32-466d-b76f-252275c6fb82\") " pod="openshift-image-registry/node-ca-vrlkw"
Sep 19 01:12:33.250103 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.250026    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f759a12f-5b6f-4924-96f2-dd5e00fc1348-env-overrides\") pod \"ovnkube-node-x785d\" (UID: \"f759a12f-5b6f-4924-96f2-dd5e00fc1348\") " pod="openshift-ovn-kubernetes/ovnkube-node-x785d"
Sep 19 01:12:33.250103 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.250049    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w75qs\" (UniqueName: \"kubernetes.io/projected/f759a12f-5b6f-4924-96f2-dd5e00fc1348-kube-api-access-w75qs\") pod \"ovnkube-node-x785d\" (UID: \"f759a12f-5b6f-4924-96f2-dd5e00fc1348\") " pod="openshift-ovn-kubernetes/ovnkube-node-x785d"
Sep 19 01:12:33.250103 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.250071    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d16ba40c-ab77-4204-b80e-d58f3a57e49a-cnibin\") pod \"multus-additional-cni-plugins-42mjj\" (UID: \"d16ba40c-ab77-4204-b80e-d58f3a57e49a\") " pod="openshift-multus/multus-additional-cni-plugins-42mjj"
Sep 19 01:12:33.250103 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.250096    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/62df040a-200c-4054-806f-252a833df006-cni-binary-copy\") pod \"multus-kwlqw\" (UID: \"62df040a-200c-4054-806f-252a833df006\") " pod="openshift-multus/multus-kwlqw"
Sep 19 01:12:33.250410 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.250123    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrtwg\" (UniqueName: \"kubernetes.io/projected/62df040a-200c-4054-806f-252a833df006-kube-api-access-lrtwg\") pod \"multus-kwlqw\" (UID: \"62df040a-200c-4054-806f-252a833df006\") " pod="openshift-multus/multus-kwlqw"
Sep 19 01:12:33.250410 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.250148    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f759a12f-5b6f-4924-96f2-dd5e00fc1348-run-systemd\") pod \"ovnkube-node-x785d\" (UID: \"f759a12f-5b6f-4924-96f2-dd5e00fc1348\") " pod="openshift-ovn-kubernetes/ovnkube-node-x785d"
Sep 19 01:12:33.250410 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.250171    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f759a12f-5b6f-4924-96f2-dd5e00fc1348-var-lib-openvswitch\") pod \"ovnkube-node-x785d\" (UID: \"f759a12f-5b6f-4924-96f2-dd5e00fc1348\") " pod="openshift-ovn-kubernetes/ovnkube-node-x785d"
Sep 19 01:12:33.250410 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.250193    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f759a12f-5b6f-4924-96f2-dd5e00fc1348-node-log\") pod \"ovnkube-node-x785d\" (UID: \"f759a12f-5b6f-4924-96f2-dd5e00fc1348\") " pod="openshift-ovn-kubernetes/ovnkube-node-x785d"
Sep 19 01:12:33.250410 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.250216    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2ff7ef56-5c24-40a5-84d6-7259cf83a29c-kubelet-config\") pod \"global-pull-secret-syncer-hrgqz\" (UID: \"2ff7ef56-5c24-40a5-84d6-7259cf83a29c\") " pod="kube-system/global-pull-secret-syncer-hrgqz"
Sep 19 01:12:33.250410 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.250238    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/62df040a-200c-4054-806f-252a833df006-host-var-lib-kubelet\") pod \"multus-kwlqw\" (UID: \"62df040a-200c-4054-806f-252a833df006\") " pod="openshift-multus/multus-kwlqw"
Sep 19 01:12:33.250410 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.250261    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/627fcf92-40ec-4025-a7dd-83243fcf057c-sys\") pod \"tuned-h94dv\" (UID: \"627fcf92-40ec-4025-a7dd-83243fcf057c\") " pod="openshift-cluster-node-tuning-operator/tuned-h94dv"
Sep 19 01:12:33.250410 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.250282    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/627fcf92-40ec-4025-a7dd-83243fcf057c-lib-modules\") pod \"tuned-h94dv\" (UID: \"627fcf92-40ec-4025-a7dd-83243fcf057c\") " pod="openshift-cluster-node-tuning-operator/tuned-h94dv"
Sep 19 01:12:33.250410 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.250304    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f759a12f-5b6f-4924-96f2-dd5e00fc1348-etc-openvswitch\") pod \"ovnkube-node-x785d\" (UID: \"f759a12f-5b6f-4924-96f2-dd5e00fc1348\") " pod="openshift-ovn-kubernetes/ovnkube-node-x785d"
Sep 19 01:12:33.250410 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.250329    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f759a12f-5b6f-4924-96f2-dd5e00fc1348-host-cni-bin\") pod \"ovnkube-node-x785d\" (UID: \"f759a12f-5b6f-4924-96f2-dd5e00fc1348\") " pod="openshift-ovn-kubernetes/ovnkube-node-x785d"
Sep 19 01:12:33.250410 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.250359    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f759a12f-5b6f-4924-96f2-dd5e00fc1348-host-cni-netd\") pod \"ovnkube-node-x785d\" (UID: \"f759a12f-5b6f-4924-96f2-dd5e00fc1348\") " pod="openshift-ovn-kubernetes/ovnkube-node-x785d"
Sep 19 01:12:33.250410 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.250381    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f759a12f-5b6f-4924-96f2-dd5e00fc1348-ovn-node-metrics-cert\") pod \"ovnkube-node-x785d\" (UID: \"f759a12f-5b6f-4924-96f2-dd5e00fc1348\") " pod="openshift-ovn-kubernetes/ovnkube-node-x785d"
Sep 19 01:12:33.250410 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.250405    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d16ba40c-ab77-4204-b80e-d58f3a57e49a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-42mjj\" (UID: \"d16ba40c-ab77-4204-b80e-d58f3a57e49a\") " pod="openshift-multus/multus-additional-cni-plugins-42mjj"
Sep 19 01:12:33.250992 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.250430    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a190ceb2-3036-4697-89d0-e103dca9cf5a-registration-dir\") pod \"aws-ebs-csi-driver-node-lsdhk\" (UID: \"a190ceb2-3036-4697-89d0-e103dca9cf5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lsdhk"
Sep 19 01:12:33.250992 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.250453    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x27vk\" (UniqueName: \"kubernetes.io/projected/ff802e32-5c32-466d-b76f-252275c6fb82-kube-api-access-x27vk\") pod \"node-ca-vrlkw\" (UID: \"ff802e32-5c32-466d-b76f-252275c6fb82\") " pod="openshift-image-registry/node-ca-vrlkw"
Sep 19 01:12:33.250992 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.250476    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/be4767d8-9458-4c5a-bc2b-d033a86f7017-hosts-file\") pod \"node-resolver-nx2sn\" (UID: \"be4767d8-9458-4c5a-bc2b-d033a86f7017\") " pod="openshift-dns/node-resolver-nx2sn"
Sep 19 01:12:33.250992 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.250499    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f759a12f-5b6f-4924-96f2-dd5e00fc1348-log-socket\") pod \"ovnkube-node-x785d\" (UID: \"f759a12f-5b6f-4924-96f2-dd5e00fc1348\") " pod="openshift-ovn-kubernetes/ovnkube-node-x785d"
Sep 19 01:12:33.250992 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.250527    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f759a12f-5b6f-4924-96f2-dd5e00fc1348-ovnkube-script-lib\") pod \"ovnkube-node-x785d\" (UID: \"f759a12f-5b6f-4924-96f2-dd5e00fc1348\") " pod="openshift-ovn-kubernetes/ovnkube-node-x785d"
Sep 19 01:12:33.250992 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.250560    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2ff7ef56-5c24-40a5-84d6-7259cf83a29c-original-pull-secret\") pod \"global-pull-secret-syncer-hrgqz\" (UID: \"2ff7ef56-5c24-40a5-84d6-7259cf83a29c\") " pod="kube-system/global-pull-secret-syncer-hrgqz"
Sep 19 01:12:33.250992 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.250588    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a190ceb2-3036-4697-89d0-e103dca9cf5a-sys-fs\") pod \"aws-ebs-csi-driver-node-lsdhk\" (UID: \"a190ceb2-3036-4697-89d0-e103dca9cf5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lsdhk"
Sep 19 01:12:33.250992 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.250630    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/627fcf92-40ec-4025-a7dd-83243fcf057c-etc-sysconfig\") pod \"tuned-h94dv\" (UID: \"627fcf92-40ec-4025-a7dd-83243fcf057c\") " pod="openshift-cluster-node-tuning-operator/tuned-h94dv"
Sep 19 01:12:33.250992 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.250653    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/627fcf92-40ec-4025-a7dd-83243fcf057c-etc-sysctl-conf\") pod \"tuned-h94dv\" (UID: \"627fcf92-40ec-4025-a7dd-83243fcf057c\") " pod="openshift-cluster-node-tuning-operator/tuned-h94dv"
Sep 19 01:12:33.250992 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.250677    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f759a12f-5b6f-4924-96f2-dd5e00fc1348-host-run-netns\") pod \"ovnkube-node-x785d\" (UID: \"f759a12f-5b6f-4924-96f2-dd5e00fc1348\") " pod="openshift-ovn-kubernetes/ovnkube-node-x785d"
Sep 19 01:12:33.250992 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.250708    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d16ba40c-ab77-4204-b80e-d58f3a57e49a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-42mjj\" (UID: \"d16ba40c-ab77-4204-b80e-d58f3a57e49a\") " pod="openshift-multus/multus-additional-cni-plugins-42mjj"
Sep 19 01:12:33.250992 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.250731    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/62df040a-200c-4054-806f-252a833df006-system-cni-dir\") pod \"multus-kwlqw\" (UID: \"62df040a-200c-4054-806f-252a833df006\") " pod="openshift-multus/multus-kwlqw"
Sep 19 01:12:33.250992 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.250756    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/62df040a-200c-4054-806f-252a833df006-multus-socket-dir-parent\") pod \"multus-kwlqw\" (UID: \"62df040a-200c-4054-806f-252a833df006\") " pod="openshift-multus/multus-kwlqw"
Sep 19 01:12:33.250992 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.250778    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/51c7d851-c313-4247-a65c-45f9403ddb9c-konnectivity-ca\") pod \"konnectivity-agent-8ndtv\" (UID: \"51c7d851-c313-4247-a65c-45f9403ddb9c\") " pod="kube-system/konnectivity-agent-8ndtv"
Sep 19 01:12:33.250992 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.250809    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f759a12f-5b6f-4924-96f2-dd5e00fc1348-ovnkube-config\") pod \"ovnkube-node-x785d\" (UID: \"f759a12f-5b6f-4924-96f2-dd5e00fc1348\") " pod="openshift-ovn-kubernetes/ovnkube-node-x785d"
Sep 19 01:12:33.250992 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.250836    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/caf84f55-01ed-4c93-81d0-f8e545bf90bf-host-slash\") pod \"iptables-alerter-xmldk\" (UID: \"caf84f55-01ed-4c93-81d0-f8e545bf90bf\") " pod="openshift-network-operator/iptables-alerter-xmldk"
Sep 19 01:12:33.250992 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.250882    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/caf84f55-01ed-4c93-81d0-f8e545bf90bf-iptables-alerter-script\") pod \"iptables-alerter-xmldk\" (UID: \"caf84f55-01ed-4c93-81d0-f8e545bf90bf\") " pod="openshift-network-operator/iptables-alerter-xmldk"
Sep 19 01:12:33.251838 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.250906    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ff802e32-5c32-466d-b76f-252275c6fb82-serviceca\") pod \"node-ca-vrlkw\" (UID: \"ff802e32-5c32-466d-b76f-252275c6fb82\") " pod="openshift-image-registry/node-ca-vrlkw"
Sep 19 01:12:33.251838 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.250932    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d16ba40c-ab77-4204-b80e-d58f3a57e49a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-42mjj\" (UID: \"d16ba40c-ab77-4204-b80e-d58f3a57e49a\") " pod="openshift-multus/multus-additional-cni-plugins-42mjj"
Sep 19 01:12:33.251838 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.250957    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85ljl\" (UniqueName: \"kubernetes.io/projected/d16ba40c-ab77-4204-b80e-d58f3a57e49a-kube-api-access-85ljl\") pod \"multus-additional-cni-plugins-42mjj\" (UID: \"d16ba40c-ab77-4204-b80e-d58f3a57e49a\") " pod="openshift-multus/multus-additional-cni-plugins-42mjj"
Sep 19 01:12:33.251838 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.250995    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/62df040a-200c-4054-806f-252a833df006-cnibin\") pod \"multus-kwlqw\" (UID: \"62df040a-200c-4054-806f-252a833df006\") " pod="openshift-multus/multus-kwlqw"
Sep 19 01:12:33.251838 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.251018    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/62df040a-200c-4054-806f-252a833df006-hostroot\") pod \"multus-kwlqw\" (UID: \"62df040a-200c-4054-806f-252a833df006\") " pod="openshift-multus/multus-kwlqw"
Sep 19 01:12:33.251838 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.251040    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/627fcf92-40ec-4025-a7dd-83243fcf057c-host\") pod \"tuned-h94dv\" (UID: \"627fcf92-40ec-4025-a7dd-83243fcf057c\") " pod="openshift-cluster-node-tuning-operator/tuned-h94dv"
Sep 19 01:12:33.251838 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.251063    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/627fcf92-40ec-4025-a7dd-83243fcf057c-etc-tuned\") pod \"tuned-h94dv\" (UID: \"627fcf92-40ec-4025-a7dd-83243fcf057c\") " pod="openshift-cluster-node-tuning-operator/tuned-h94dv"
Sep 19 01:12:33.251838 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.251087    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a190ceb2-3036-4697-89d0-e103dca9cf5a-device-dir\") pod \"aws-ebs-csi-driver-node-lsdhk\" (UID: \"a190ceb2-3036-4697-89d0-e103dca9cf5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lsdhk"
Sep 19 01:12:33.251838 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.251114    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fp5q\" (UniqueName: \"kubernetes.io/projected/a190ceb2-3036-4697-89d0-e103dca9cf5a-kube-api-access-6fp5q\") pod \"aws-ebs-csi-driver-node-lsdhk\" (UID: \"a190ceb2-3036-4697-89d0-e103dca9cf5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lsdhk"
Sep 19 01:12:33.251838 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.251138    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/62df040a-200c-4054-806f-252a833df006-host-var-lib-cni-bin\") pod \"multus-kwlqw\" (UID: \"62df040a-200c-4054-806f-252a833df006\") " pod="openshift-multus/multus-kwlqw"
Sep 19 01:12:33.251838 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.251162    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wt9cj\" (UniqueName: \"kubernetes.io/projected/caf84f55-01ed-4c93-81d0-f8e545bf90bf-kube-api-access-wt9cj\") pod \"iptables-alerter-xmldk\" (UID: \"caf84f55-01ed-4c93-81d0-f8e545bf90bf\") " pod="openshift-network-operator/iptables-alerter-xmldk"
Sep 19 01:12:33.251838 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.251188    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/62df040a-200c-4054-806f-252a833df006-host-run-k8s-cni-cncf-io\") pod \"multus-kwlqw\" (UID: \"62df040a-200c-4054-806f-252a833df006\") " pod="openshift-multus/multus-kwlqw"
Sep 19 01:12:33.251838 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.251213    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2ff7ef56-5c24-40a5-84d6-7259cf83a29c-dbus\") pod \"global-pull-secret-syncer-hrgqz\" (UID: \"2ff7ef56-5c24-40a5-84d6-7259cf83a29c\") " pod="kube-system/global-pull-secret-syncer-hrgqz"
Sep 19 01:12:33.251838 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.251238    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f759a12f-5b6f-4924-96f2-dd5e00fc1348-run-ovn\") pod \"ovnkube-node-x785d\" (UID: \"f759a12f-5b6f-4924-96f2-dd5e00fc1348\") " pod="openshift-ovn-kubernetes/ovnkube-node-x785d"
Sep 19 01:12:33.251838 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.251263    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a190ceb2-3036-4697-89d0-e103dca9cf5a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lsdhk\" (UID: \"a190ceb2-3036-4697-89d0-e103dca9cf5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lsdhk"
Sep 19 01:12:33.251838 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.251287    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a190ceb2-3036-4697-89d0-e103dca9cf5a-etc-selinux\") pod \"aws-ebs-csi-driver-node-lsdhk\" (UID: \"a190ceb2-3036-4697-89d0-e103dca9cf5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lsdhk"
Sep 19 01:12:33.251838 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.251312    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmmbq\" (UniqueName: \"kubernetes.io/projected/3884e026-09ad-4bc8-bcb8-ae622d873aea-kube-api-access-mmmbq\") pod \"network-metrics-daemon-dkpb2\" (UID: \"3884e026-09ad-4bc8-bcb8-ae622d873aea\") " pod="openshift-multus/network-metrics-daemon-dkpb2"
Sep 19 01:12:33.252577 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.251336    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/627fcf92-40ec-4025-a7dd-83243fcf057c-tmp\") pod \"tuned-h94dv\" (UID: \"627fcf92-40ec-4025-a7dd-83243fcf057c\") " pod="openshift-cluster-node-tuning-operator/tuned-h94dv"
Sep 19 01:12:33.252577 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.251362    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d16ba40c-ab77-4204-b80e-d58f3a57e49a-cni-binary-copy\") pod \"multus-additional-cni-plugins-42mjj\" (UID: \"d16ba40c-ab77-4204-b80e-d58f3a57e49a\") " pod="openshift-multus/multus-additional-cni-plugins-42mjj"
Sep 19 01:12:33.252577 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.251387    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/62df040a-200c-4054-806f-252a833df006-host-var-lib-cni-multus\") pod \"multus-kwlqw\" (UID: \"62df040a-200c-4054-806f-252a833df006\") " pod="openshift-multus/multus-kwlqw"
Sep 19 01:12:33.252577 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.251414    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/62df040a-200c-4054-806f-252a833df006-multus-daemon-config\") pod \"multus-kwlqw\" (UID: \"62df040a-200c-4054-806f-252a833df006\") " pod="openshift-multus/multus-kwlqw"
Sep 19 01:12:33.252577 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.251440    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/51c7d851-c313-4247-a65c-45f9403ddb9c-agent-certs\") pod \"konnectivity-agent-8ndtv\" (UID: \"51c7d851-c313-4247-a65c-45f9403ddb9c\") " pod="kube-system/konnectivity-agent-8ndtv"
Sep 19 01:12:33.252577 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.251462    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a190ceb2-3036-4697-89d0-e103dca9cf5a-socket-dir\") pod \"aws-ebs-csi-driver-node-lsdhk\" (UID: \"a190ceb2-3036-4697-89d0-e103dca9cf5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lsdhk"
Sep 19 01:12:33.252577 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.251478    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3884e026-09ad-4bc8-bcb8-ae622d873aea-metrics-certs\") pod \"network-metrics-daemon-dkpb2\" (UID: \"3884e026-09ad-4bc8-bcb8-ae622d873aea\") " pod="openshift-multus/network-metrics-daemon-dkpb2"
Sep 19 01:12:33.252577 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.251492    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/62df040a-200c-4054-806f-252a833df006-host-run-multus-certs\") pod \"multus-kwlqw\" (UID: \"62df040a-200c-4054-806f-252a833df006\") " pod="openshift-multus/multus-kwlqw"
Sep 19 01:12:33.252577 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.251516    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/627fcf92-40ec-4025-a7dd-83243fcf057c-etc-kubernetes\") pod \"tuned-h94dv\" (UID: \"627fcf92-40ec-4025-a7dd-83243fcf057c\") " pod="openshift-cluster-node-tuning-operator/tuned-h94dv"
Sep 19 01:12:33.252577 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.251544    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/627fcf92-40ec-4025-a7dd-83243fcf057c-etc-systemd\") pod \"tuned-h94dv\" (UID: \"627fcf92-40ec-4025-a7dd-83243fcf057c\") " pod="openshift-cluster-node-tuning-operator/tuned-h94dv"
Sep 19 01:12:33.252577 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.251573    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/627fcf92-40ec-4025-a7dd-83243fcf057c-run\") pod \"tuned-h94dv\" (UID: \"627fcf92-40ec-4025-a7dd-83243fcf057c\") " pod="openshift-cluster-node-tuning-operator/tuned-h94dv"
Sep 19 01:12:33.252577 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.251596    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/627fcf92-40ec-4025-a7dd-83243fcf057c-var-lib-kubelet\") pod \"tuned-h94dv\" (UID: \"627fcf92-40ec-4025-a7dd-83243fcf057c\") " pod="openshift-cluster-node-tuning-operator/tuned-h94dv"
Sep 19 01:12:33.252577 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.251616    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/62df040a-200c-4054-806f-252a833df006-multus-cni-dir\") pod \"multus-kwlqw\" (UID: \"62df040a-200c-4054-806f-252a833df006\") " pod="openshift-multus/multus-kwlqw"
Sep 19 01:12:33.252577 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.251631    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be4767d8-9458-4c5a-bc2b-d033a86f7017-tmp-dir\") pod \"node-resolver-nx2sn\" (UID: \"be4767d8-9458-4c5a-bc2b-d033a86f7017\") " pod="openshift-dns/node-resolver-nx2sn"
Sep 19 01:12:33.252577 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.251648    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2f7r\" (UniqueName: \"kubernetes.io/projected/be4767d8-9458-4c5a-bc2b-d033a86f7017-kube-api-access-v2f7r\") pod \"node-resolver-nx2sn\" (UID: \"be4767d8-9458-4c5a-bc2b-d033a86f7017\") " pod="openshift-dns/node-resolver-nx2sn"
Sep 19 01:12:33.252577 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.251663    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgtk8\" (UniqueName: \"kubernetes.io/projected/627fcf92-40ec-4025-a7dd-83243fcf057c-kube-api-access-lgtk8\") pod \"tuned-h94dv\" (UID: \"627fcf92-40ec-4025-a7dd-83243fcf057c\") " pod="openshift-cluster-node-tuning-operator/tuned-h94dv"
Sep 19 01:12:33.252577 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.251679    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f759a12f-5b6f-4924-96f2-dd5e00fc1348-run-openvswitch\") pod \"ovnkube-node-x785d\" (UID: \"f759a12f-5b6f-4924-96f2-dd5e00fc1348\") " pod="openshift-ovn-kubernetes/ovnkube-node-x785d"
Sep 19 01:12:33.253390 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.251695    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f759a12f-5b6f-4924-96f2-dd5e00fc1348-host-run-ovn-kubernetes\") pod \"ovnkube-node-x785d\" (UID: \"f759a12f-5b6f-4924-96f2-dd5e00fc1348\") " pod="openshift-ovn-kubernetes/ovnkube-node-x785d"
Sep 19 01:12:33.253390 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.251710    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/62df040a-200c-4054-806f-252a833df006-host-run-netns\") pod \"multus-kwlqw\" (UID: \"62df040a-200c-4054-806f-252a833df006\") " pod="openshift-multus/multus-kwlqw"
Sep 19 01:12:33.253390 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.251726    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/62df040a-200c-4054-806f-252a833df006-etc-kubernetes\") pod \"multus-kwlqw\" (UID: \"62df040a-200c-4054-806f-252a833df006\") " pod="openshift-multus/multus-kwlqw"
Sep 19 01:12:33.253390 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.251741    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8c9j\" (UniqueName: \"kubernetes.io/projected/b0be445f-6160-4532-8a71-55bfdf782241-kube-api-access-g8c9j\") pod \"network-check-target-lmcnn\" (UID: \"b0be445f-6160-4532-8a71-55bfdf782241\") " pod="openshift-network-diagnostics/network-check-target-lmcnn"
Sep 19 01:12:33.253390 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.251758    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/627fcf92-40ec-4025-a7dd-83243fcf057c-etc-modprobe-d\") pod \"tuned-h94dv\" (UID: \"627fcf92-40ec-4025-a7dd-83243fcf057c\") " pod="openshift-cluster-node-tuning-operator/tuned-h94dv"
Sep 19 01:12:33.253390 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.251772    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/627fcf92-40ec-4025-a7dd-83243fcf057c-etc-sysctl-d\") pod \"tuned-h94dv\" (UID: \"627fcf92-40ec-4025-a7dd-83243fcf057c\") " pod="openshift-cluster-node-tuning-operator/tuned-h94dv"
Sep 19 01:12:33.253390 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.251847    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f759a12f-5b6f-4924-96f2-dd5e00fc1348-host-kubelet\") pod \"ovnkube-node-x785d\" (UID: \"f759a12f-5b6f-4924-96f2-dd5e00fc1348\") " pod="openshift-ovn-kubernetes/ovnkube-node-x785d"
Sep 19 01:12:33.253390 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.251896    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d16ba40c-ab77-4204-b80e-d58f3a57e49a-system-cni-dir\") pod \"multus-additional-cni-plugins-42mjj\" (UID: \"d16ba40c-ab77-4204-b80e-d58f3a57e49a\") " pod="openshift-multus/multus-additional-cni-plugins-42mjj"
Sep 19 01:12:33.253390 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.251922    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f759a12f-5b6f-4924-96f2-dd5e00fc1348-systemd-units\") pod \"ovnkube-node-x785d\" (UID: \"f759a12f-5b6f-4924-96f2-dd5e00fc1348\") " pod="openshift-ovn-kubernetes/ovnkube-node-x785d"
Sep 19 01:12:33.253390 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.251944    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f759a12f-5b6f-4924-96f2-dd5e00fc1348-host-slash\") pod \"ovnkube-node-x785d\" (UID: \"f759a12f-5b6f-4924-96f2-dd5e00fc1348\") " pod="openshift-ovn-kubernetes/ovnkube-node-x785d"
Sep 19 01:12:33.253390 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.251966    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f759a12f-5b6f-4924-96f2-dd5e00fc1348-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x785d\" (UID: \"f759a12f-5b6f-4924-96f2-dd5e00fc1348\") " pod="openshift-ovn-kubernetes/ovnkube-node-x785d"
Sep 19 01:12:33.253390 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.252325    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f759a12f-5b6f-4924-96f2-dd5e00fc1348-env-overrides\") pod \"ovnkube-node-x785d\" (UID: \"f759a12f-5b6f-4924-96f2-dd5e00fc1348\") " pod="openshift-ovn-kubernetes/ovnkube-node-x785d"
Sep 19 01:12:33.253390 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.252554    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d16ba40c-ab77-4204-b80e-d58f3a57e49a-cnibin\") pod \"multus-additional-cni-plugins-42mjj\" (UID: \"d16ba40c-ab77-4204-b80e-d58f3a57e49a\") " pod="openshift-multus/multus-additional-cni-plugins-42mjj"
Sep 19 01:12:33.253390 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.252602    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f759a12f-5b6f-4924-96f2-dd5e00fc1348-run-systemd\") pod \"ovnkube-node-x785d\" (UID: \"f759a12f-5b6f-4924-96f2-dd5e00fc1348\") " pod="openshift-ovn-kubernetes/ovnkube-node-x785d"
Sep 19 01:12:33.253390 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.252627    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f759a12f-5b6f-4924-96f2-dd5e00fc1348-var-lib-openvswitch\") pod \"ovnkube-node-x785d\" (UID: \"f759a12f-5b6f-4924-96f2-dd5e00fc1348\") " pod="openshift-ovn-kubernetes/ovnkube-node-x785d"
Sep 19 01:12:33.253390 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.252994    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2ff7ef56-5c24-40a5-84d6-7259cf83a29c-kubelet-config\") pod \"global-pull-secret-syncer-hrgqz\" (UID: \"2ff7ef56-5c24-40a5-84d6-7259cf83a29c\") " pod="kube-system/global-pull-secret-syncer-hrgqz"
Sep 19 01:12:33.253390 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.252652    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f759a12f-5b6f-4924-96f2-dd5e00fc1348-node-log\") pod \"ovnkube-node-x785d\" (UID: \"f759a12f-5b6f-4924-96f2-dd5e00fc1348\") " pod="openshift-ovn-kubernetes/ovnkube-node-x785d"
Sep 19 01:12:33.253390 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:33.253143    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 01:12:33.254222 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:33.253217    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3884e026-09ad-4bc8-bcb8-ae622d873aea-metrics-certs podName:3884e026-09ad-4bc8-bcb8-ae622d873aea nodeName:}" failed. No retries permitted until 2026-09-19 01:12:33.753180539 +0000 UTC m=+4.640849595 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3884e026-09ad-4bc8-bcb8-ae622d873aea-metrics-certs") pod "network-metrics-daemon-dkpb2" (UID: "3884e026-09ad-4bc8-bcb8-ae622d873aea") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 01:12:33.254222 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.253252    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f759a12f-5b6f-4924-96f2-dd5e00fc1348-etc-openvswitch\") pod \"ovnkube-node-x785d\" (UID: \"f759a12f-5b6f-4924-96f2-dd5e00fc1348\") " pod="openshift-ovn-kubernetes/ovnkube-node-x785d"
Sep 19 01:12:33.254222 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.253292    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2ff7ef56-5c24-40a5-84d6-7259cf83a29c-dbus\") pod \"global-pull-secret-syncer-hrgqz\" (UID: \"2ff7ef56-5c24-40a5-84d6-7259cf83a29c\") " pod="kube-system/global-pull-secret-syncer-hrgqz"
Sep 19 01:12:33.254222 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.253315    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f759a12f-5b6f-4924-96f2-dd5e00fc1348-run-openvswitch\") pod \"ovnkube-node-x785d\" (UID: \"f759a12f-5b6f-4924-96f2-dd5e00fc1348\") " pod="openshift-ovn-kubernetes/ovnkube-node-x785d"
Sep 19 01:12:33.254222 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.253311    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f759a12f-5b6f-4924-96f2-dd5e00fc1348-run-ovn\") pod \"ovnkube-node-x785d\" (UID: \"f759a12f-5b6f-4924-96f2-dd5e00fc1348\") " pod="openshift-ovn-kubernetes/ovnkube-node-x785d"
Sep 19 01:12:33.254222 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.253330    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f759a12f-5b6f-4924-96f2-dd5e00fc1348-host-cni-bin\") pod \"ovnkube-node-x785d\" (UID: \"f759a12f-5b6f-4924-96f2-dd5e00fc1348\") " pod="openshift-ovn-kubernetes/ovnkube-node-x785d"
Sep 19 01:12:33.254222 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.253448    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a190ceb2-3036-4697-89d0-e103dca9cf5a-etc-selinux\") pod \"aws-ebs-csi-driver-node-lsdhk\" (UID: \"a190ceb2-3036-4697-89d0-e103dca9cf5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lsdhk"
Sep 19 01:12:33.254222 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.253495    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a190ceb2-3036-4697-89d0-e103dca9cf5a-registration-dir\") pod \"aws-ebs-csi-driver-node-lsdhk\" (UID: \"a190ceb2-3036-4697-89d0-e103dca9cf5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lsdhk"
Sep 19 01:12:33.254222 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.253504    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/caf84f55-01ed-4c93-81d0-f8e545bf90bf-host-slash\") pod \"iptables-alerter-xmldk\" (UID: \"caf84f55-01ed-4c93-81d0-f8e545bf90bf\") " pod="openshift-network-operator/iptables-alerter-xmldk"
Sep 19 01:12:33.254222 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.253598    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a190ceb2-3036-4697-89d0-e103dca9cf5a-socket-dir\") pod \"aws-ebs-csi-driver-node-lsdhk\" (UID: \"a190ceb2-3036-4697-89d0-e103dca9cf5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lsdhk"
Sep 19 01:12:33.254222 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.253644    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f759a12f-5b6f-4924-96f2-dd5e00fc1348-host-cni-netd\") pod \"ovnkube-node-x785d\" (UID: \"f759a12f-5b6f-4924-96f2-dd5e00fc1348\") " pod="openshift-ovn-kubernetes/ovnkube-node-x785d"
Sep 19 01:12:33.254222 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.253667    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a190ceb2-3036-4697-89d0-e103dca9cf5a-device-dir\") pod \"aws-ebs-csi-driver-node-lsdhk\" (UID: \"a190ceb2-3036-4697-89d0-e103dca9cf5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lsdhk"
Sep 19 01:12:33.254222 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.253721    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a190ceb2-3036-4697-89d0-e103dca9cf5a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lsdhk\" (UID: \"a190ceb2-3036-4697-89d0-e103dca9cf5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lsdhk"
Sep 19 01:12:33.254222 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.253898    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f759a12f-5b6f-4924-96f2-dd5e00fc1348-ovnkube-config\") pod \"ovnkube-node-x785d\" (UID: \"f759a12f-5b6f-4924-96f2-dd5e00fc1348\") " pod="openshift-ovn-kubernetes/ovnkube-node-x785d"
Sep 19 01:12:33.254222 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.253911    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d16ba40c-ab77-4204-b80e-d58f3a57e49a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-42mjj\" (UID: \"d16ba40c-ab77-4204-b80e-d58f3a57e49a\") " pod="openshift-multus/multus-additional-cni-plugins-42mjj"
Sep 19 01:12:33.254222 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.253949    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f759a12f-5b6f-4924-96f2-dd5e00fc1348-log-socket\") pod \"ovnkube-node-x785d\" (UID: \"f759a12f-5b6f-4924-96f2-dd5e00fc1348\") " pod="openshift-ovn-kubernetes/ovnkube-node-x785d"
Sep 19 01:12:33.254222 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.253959    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f759a12f-5b6f-4924-96f2-dd5e00fc1348-ovnkube-script-lib\") pod \"ovnkube-node-x785d\" (UID: \"f759a12f-5b6f-4924-96f2-dd5e00fc1348\") " pod="openshift-ovn-kubernetes/ovnkube-node-x785d"
Sep 19 01:12:33.254972 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.253983    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a190ceb2-3036-4697-89d0-e103dca9cf5a-sys-fs\") pod \"aws-ebs-csi-driver-node-lsdhk\" (UID: \"a190ceb2-3036-4697-89d0-e103dca9cf5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lsdhk"
Sep 19 01:12:33.254972 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.253998    2581 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 19 01:12:33.254972 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:33.254050    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 19 01:12:33.254972 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:33.254096    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ff7ef56-5c24-40a5-84d6-7259cf83a29c-original-pull-secret podName:2ff7ef56-5c24-40a5-84d6-7259cf83a29c nodeName:}" failed. No retries permitted until 2026-09-19 01:12:33.754081366 +0000 UTC m=+4.641750426 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2ff7ef56-5c24-40a5-84d6-7259cf83a29c-original-pull-secret") pod "global-pull-secret-syncer-hrgqz" (UID: "2ff7ef56-5c24-40a5-84d6-7259cf83a29c") : object "kube-system"/"original-pull-secret" not registered
Sep 19 01:12:33.254972 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.254138    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f759a12f-5b6f-4924-96f2-dd5e00fc1348-host-run-ovn-kubernetes\") pod \"ovnkube-node-x785d\" (UID: \"f759a12f-5b6f-4924-96f2-dd5e00fc1348\") " pod="openshift-ovn-kubernetes/ovnkube-node-x785d"
Sep 19 01:12:33.254972 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.254144    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/caf84f55-01ed-4c93-81d0-f8e545bf90bf-iptables-alerter-script\") pod \"iptables-alerter-xmldk\" (UID: \"caf84f55-01ed-4c93-81d0-f8e545bf90bf\") " pod="openshift-network-operator/iptables-alerter-xmldk"
Sep 19 01:12:33.254972 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.254197    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f759a12f-5b6f-4924-96f2-dd5e00fc1348-host-run-netns\") pod \"ovnkube-node-x785d\" (UID: \"f759a12f-5b6f-4924-96f2-dd5e00fc1348\") " pod="openshift-ovn-kubernetes/ovnkube-node-x785d"
Sep 19 01:12:33.254972 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.254463    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d16ba40c-ab77-4204-b80e-d58f3a57e49a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-42mjj\" (UID: \"d16ba40c-ab77-4204-b80e-d58f3a57e49a\") " pod="openshift-multus/multus-additional-cni-plugins-42mjj"
Sep 19 01:12:33.254972 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.254529    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d16ba40c-ab77-4204-b80e-d58f3a57e49a-cni-binary-copy\") pod \"multus-additional-cni-plugins-42mjj\" (UID: \"d16ba40c-ab77-4204-b80e-d58f3a57e49a\") " pod="openshift-multus/multus-additional-cni-plugins-42mjj"
Sep 19 01:12:33.254972 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.254564    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d16ba40c-ab77-4204-b80e-d58f3a57e49a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-42mjj\" (UID: \"d16ba40c-ab77-4204-b80e-d58f3a57e49a\") " pod="openshift-multus/multus-additional-cni-plugins-42mjj"
Sep 19 01:12:33.257882 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.257836    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f759a12f-5b6f-4924-96f2-dd5e00fc1348-ovn-node-metrics-cert\") pod \"ovnkube-node-x785d\" (UID: \"f759a12f-5b6f-4924-96f2-dd5e00fc1348\") " pod="openshift-ovn-kubernetes/ovnkube-node-x785d"
Sep 19 01:12:33.265058 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.264981    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wt9cj\" (UniqueName: \"kubernetes.io/projected/caf84f55-01ed-4c93-81d0-f8e545bf90bf-kube-api-access-wt9cj\") pod \"iptables-alerter-xmldk\" (UID: \"caf84f55-01ed-4c93-81d0-f8e545bf90bf\") " pod="openshift-network-operator/iptables-alerter-xmldk"
Sep 19 01:12:33.265386 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.265368    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fp5q\" (UniqueName: \"kubernetes.io/projected/a190ceb2-3036-4697-89d0-e103dca9cf5a-kube-api-access-6fp5q\") pod \"aws-ebs-csi-driver-node-lsdhk\" (UID: \"a190ceb2-3036-4697-89d0-e103dca9cf5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lsdhk"
Sep 19 01:12:33.265782 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.265761    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmmbq\" (UniqueName: \"kubernetes.io/projected/3884e026-09ad-4bc8-bcb8-ae622d873aea-kube-api-access-mmmbq\") pod \"network-metrics-daemon-dkpb2\" (UID: \"3884e026-09ad-4bc8-bcb8-ae622d873aea\") " pod="openshift-multus/network-metrics-daemon-dkpb2"
Sep 19 01:12:33.270656 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.270602    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-85ljl\" (UniqueName: \"kubernetes.io/projected/d16ba40c-ab77-4204-b80e-d58f3a57e49a-kube-api-access-85ljl\") pod \"multus-additional-cni-plugins-42mjj\" (UID: \"d16ba40c-ab77-4204-b80e-d58f3a57e49a\") " pod="openshift-multus/multus-additional-cni-plugins-42mjj"
Sep 19 01:12:33.275410 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.275361    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-113.ec2.internal" event={"ID":"bf17d6182a2ab91b4c163d92cc1c844e","Type":"ContainerStarted","Data":"946e4acbbd3d5f9bd3726d1bbfbcddb78c562c2b186421b4fc3d113b36b72b62"}
Sep 19 01:12:33.276274 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.276234    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w75qs\" (UniqueName: \"kubernetes.io/projected/f759a12f-5b6f-4924-96f2-dd5e00fc1348-kube-api-access-w75qs\") pod \"ovnkube-node-x785d\" (UID: \"f759a12f-5b6f-4924-96f2-dd5e00fc1348\") " pod="openshift-ovn-kubernetes/ovnkube-node-x785d"
Sep 19 01:12:33.276647 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.276620    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-113.ec2.internal" event={"ID":"76eada0400f575bdcabf278316c26ab9","Type":"ContainerStarted","Data":"a8a8aa43c60497f474854ba7e8c26624ad7484f84b890fadaba975bec477f212"}
Sep 19 01:12:33.352440 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.352415    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/62df040a-200c-4054-806f-252a833df006-cni-binary-copy\") pod \"multus-kwlqw\" (UID: \"62df040a-200c-4054-806f-252a833df006\") " pod="openshift-multus/multus-kwlqw"
Sep 19 01:12:33.352524 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.352444    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrtwg\" (UniqueName: \"kubernetes.io/projected/62df040a-200c-4054-806f-252a833df006-kube-api-access-lrtwg\") pod \"multus-kwlqw\" (UID: \"62df040a-200c-4054-806f-252a833df006\") " pod="openshift-multus/multus-kwlqw"
Sep 19 01:12:33.352524 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.352462    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/62df040a-200c-4054-806f-252a833df006-host-var-lib-kubelet\") pod \"multus-kwlqw\" (UID: \"62df040a-200c-4054-806f-252a833df006\") " pod="openshift-multus/multus-kwlqw"
Sep 19 01:12:33.352524 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.352484    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/627fcf92-40ec-4025-a7dd-83243fcf057c-sys\") pod \"tuned-h94dv\" (UID: \"627fcf92-40ec-4025-a7dd-83243fcf057c\") " pod="openshift-cluster-node-tuning-operator/tuned-h94dv"
Sep 19 01:12:33.352524 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.352503    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/627fcf92-40ec-4025-a7dd-83243fcf057c-lib-modules\") pod \"tuned-h94dv\" (UID: \"627fcf92-40ec-4025-a7dd-83243fcf057c\") " pod="openshift-cluster-node-tuning-operator/tuned-h94dv"
Sep 19 01:12:33.352728 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.352524    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x27vk\" (UniqueName: \"kubernetes.io/projected/ff802e32-5c32-466d-b76f-252275c6fb82-kube-api-access-x27vk\") pod \"node-ca-vrlkw\" (UID: \"ff802e32-5c32-466d-b76f-252275c6fb82\") " pod="openshift-image-registry/node-ca-vrlkw"
Sep 19 01:12:33.352728 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.352555    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/62df040a-200c-4054-806f-252a833df006-host-var-lib-kubelet\") pod \"multus-kwlqw\" (UID: \"62df040a-200c-4054-806f-252a833df006\") " pod="openshift-multus/multus-kwlqw"
Sep 19 01:12:33.352728 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.352579    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/627fcf92-40ec-4025-a7dd-83243fcf057c-sys\") pod \"tuned-h94dv\" (UID: \"627fcf92-40ec-4025-a7dd-83243fcf057c\") " pod="openshift-cluster-node-tuning-operator/tuned-h94dv"
Sep 19 01:12:33.352728 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.352667    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/627fcf92-40ec-4025-a7dd-83243fcf057c-lib-modules\") pod \"tuned-h94dv\" (UID: \"627fcf92-40ec-4025-a7dd-83243fcf057c\") " pod="openshift-cluster-node-tuning-operator/tuned-h94dv"
Sep 19 01:12:33.352728 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.352681    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/be4767d8-9458-4c5a-bc2b-d033a86f7017-hosts-file\") pod \"node-resolver-nx2sn\" (UID: \"be4767d8-9458-4c5a-bc2b-d033a86f7017\") " pod="openshift-dns/node-resolver-nx2sn"
Sep 19 01:12:33.352728 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.352721    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/627fcf92-40ec-4025-a7dd-83243fcf057c-etc-sysconfig\") pod \"tuned-h94dv\" (UID: \"627fcf92-40ec-4025-a7dd-83243fcf057c\") " pod="openshift-cluster-node-tuning-operator/tuned-h94dv"
Sep 19 01:12:33.353013 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.352732    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/be4767d8-9458-4c5a-bc2b-d033a86f7017-hosts-file\") pod \"node-resolver-nx2sn\" (UID: \"be4767d8-9458-4c5a-bc2b-d033a86f7017\") " pod="openshift-dns/node-resolver-nx2sn"
Sep 19 01:12:33.353013 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.352736    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/627fcf92-40ec-4025-a7dd-83243fcf057c-etc-sysctl-conf\") pod \"tuned-h94dv\" (UID: \"627fcf92-40ec-4025-a7dd-83243fcf057c\") " pod="openshift-cluster-node-tuning-operator/tuned-h94dv"
Sep 19 01:12:33.353013 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.352764    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/627fcf92-40ec-4025-a7dd-83243fcf057c-etc-sysconfig\") pod \"tuned-h94dv\" (UID: \"627fcf92-40ec-4025-a7dd-83243fcf057c\") " pod="openshift-cluster-node-tuning-operator/tuned-h94dv"
Sep 19 01:12:33.353013 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.352771    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/62df040a-200c-4054-806f-252a833df006-system-cni-dir\") pod \"multus-kwlqw\" (UID: \"62df040a-200c-4054-806f-252a833df006\") " pod="openshift-multus/multus-kwlqw"
Sep 19 01:12:33.353013 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.352808    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/62df040a-200c-4054-806f-252a833df006-multus-socket-dir-parent\") pod \"multus-kwlqw\" (UID: \"62df040a-200c-4054-806f-252a833df006\") " pod="openshift-multus/multus-kwlqw"
Sep 19 01:12:33.353013 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.352821    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/62df040a-200c-4054-806f-252a833df006-system-cni-dir\") pod \"multus-kwlqw\" (UID: \"62df040a-200c-4054-806f-252a833df006\") " pod="openshift-multus/multus-kwlqw"
Sep 19 01:12:33.353013 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.352833    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/51c7d851-c313-4247-a65c-45f9403ddb9c-konnectivity-ca\") pod \"konnectivity-agent-8ndtv\" (UID: \"51c7d851-c313-4247-a65c-45f9403ddb9c\") " pod="kube-system/konnectivity-agent-8ndtv"
Sep 19 01:12:33.353013 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.352840    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/627fcf92-40ec-4025-a7dd-83243fcf057c-etc-sysctl-conf\") pod \"tuned-h94dv\" (UID: \"627fcf92-40ec-4025-a7dd-83243fcf057c\") " pod="openshift-cluster-node-tuning-operator/tuned-h94dv"
Sep 19 01:12:33.353013 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.352880    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ff802e32-5c32-466d-b76f-252275c6fb82-serviceca\") pod \"node-ca-vrlkw\" (UID: \"ff802e32-5c32-466d-b76f-252275c6fb82\") " pod="openshift-image-registry/node-ca-vrlkw"
Sep 19 01:12:33.353013 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.352889    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/62df040a-200c-4054-806f-252a833df006-multus-socket-dir-parent\") pod \"multus-kwlqw\" (UID: \"62df040a-200c-4054-806f-252a833df006\") " pod="openshift-multus/multus-kwlqw"
Sep 19 01:12:33.353013 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.352909    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/62df040a-200c-4054-806f-252a833df006-cnibin\") pod \"multus-kwlqw\" (UID: \"62df040a-200c-4054-806f-252a833df006\") " pod="openshift-multus/multus-kwlqw"
Sep 19 01:12:33.353013 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.352934    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/62df040a-200c-4054-806f-252a833df006-hostroot\") pod \"multus-kwlqw\" (UID: \"62df040a-200c-4054-806f-252a833df006\") " pod="openshift-multus/multus-kwlqw"
Sep 19 01:12:33.353013 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.352957    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/627fcf92-40ec-4025-a7dd-83243fcf057c-host\") pod \"tuned-h94dv\" (UID: \"627fcf92-40ec-4025-a7dd-83243fcf057c\") " pod="openshift-cluster-node-tuning-operator/tuned-h94dv"
Sep 19 01:12:33.353013 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.352979    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/627fcf92-40ec-4025-a7dd-83243fcf057c-etc-tuned\") pod \"tuned-h94dv\" (UID: \"627fcf92-40ec-4025-a7dd-83243fcf057c\") " pod="openshift-cluster-node-tuning-operator/tuned-h94dv"
Sep 19 01:12:33.353013 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.353005    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/62df040a-200c-4054-806f-252a833df006-host-var-lib-cni-bin\") pod \"multus-kwlqw\" (UID: \"62df040a-200c-4054-806f-252a833df006\") " pod="openshift-multus/multus-kwlqw"
Sep 19 01:12:33.353013 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.353008    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/62df040a-200c-4054-806f-252a833df006-cnibin\") pod \"multus-kwlqw\" (UID: \"62df040a-200c-4054-806f-252a833df006\") " pod="openshift-multus/multus-kwlqw"
Sep 19 01:12:33.353720 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.353019    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/62df040a-200c-4054-806f-252a833df006-hostroot\") pod \"multus-kwlqw\" (UID: \"62df040a-200c-4054-806f-252a833df006\") " pod="openshift-multus/multus-kwlqw"
Sep 19 01:12:33.353720 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.353049    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/62df040a-200c-4054-806f-252a833df006-host-var-lib-cni-bin\") pod \"multus-kwlqw\" (UID: \"62df040a-200c-4054-806f-252a833df006\") " pod="openshift-multus/multus-kwlqw"
Sep 19 01:12:33.353720 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.353048    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/62df040a-200c-4054-806f-252a833df006-host-run-k8s-cni-cncf-io\") pod \"multus-kwlqw\" (UID: \"62df040a-200c-4054-806f-252a833df006\") " pod="openshift-multus/multus-kwlqw"
Sep 19 01:12:33.353720 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.353072    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/62df040a-200c-4054-806f-252a833df006-cni-binary-copy\") pod \"multus-kwlqw\" (UID: \"62df040a-200c-4054-806f-252a833df006\") " pod="openshift-multus/multus-kwlqw"
Sep 19 01:12:33.353720 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.353088    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/627fcf92-40ec-4025-a7dd-83243fcf057c-tmp\") pod \"tuned-h94dv\" (UID: \"627fcf92-40ec-4025-a7dd-83243fcf057c\") " pod="openshift-cluster-node-tuning-operator/tuned-h94dv"
Sep 19 01:12:33.353720 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.353088    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/62df040a-200c-4054-806f-252a833df006-host-run-k8s-cni-cncf-io\") pod \"multus-kwlqw\" (UID: \"62df040a-200c-4054-806f-252a833df006\") " pod="openshift-multus/multus-kwlqw"
Sep 19 01:12:33.353720 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.353114    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/62df040a-200c-4054-806f-252a833df006-host-var-lib-cni-multus\") pod \"multus-kwlqw\" (UID: \"62df040a-200c-4054-806f-252a833df006\") " pod="openshift-multus/multus-kwlqw"
Sep 19 01:12:33.353720 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.353144    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/62df040a-200c-4054-806f-252a833df006-host-var-lib-cni-multus\") pod \"multus-kwlqw\" (UID: \"62df040a-200c-4054-806f-252a833df006\") " pod="openshift-multus/multus-kwlqw"
Sep 19 01:12:33.353720 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.353156    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/62df040a-200c-4054-806f-252a833df006-multus-daemon-config\") pod \"multus-kwlqw\" (UID: \"62df040a-200c-4054-806f-252a833df006\") " pod="openshift-multus/multus-kwlqw"
Sep 19 01:12:33.353720 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.353192    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/51c7d851-c313-4247-a65c-45f9403ddb9c-agent-certs\") pod \"konnectivity-agent-8ndtv\" (UID: \"51c7d851-c313-4247-a65c-45f9403ddb9c\") " pod="kube-system/konnectivity-agent-8ndtv"
Sep 19 01:12:33.353720 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.353229    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/62df040a-200c-4054-806f-252a833df006-host-run-multus-certs\") pod \"multus-kwlqw\" (UID: \"62df040a-200c-4054-806f-252a833df006\") " pod="openshift-multus/multus-kwlqw"
Sep 19 01:12:33.353720 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.353257    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/627fcf92-40ec-4025-a7dd-83243fcf057c-etc-kubernetes\") pod \"tuned-h94dv\" (UID: \"627fcf92-40ec-4025-a7dd-83243fcf057c\") " pod="openshift-cluster-node-tuning-operator/tuned-h94dv"
Sep 19 01:12:33.353720 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.353280    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/627fcf92-40ec-4025-a7dd-83243fcf057c-etc-systemd\") pod \"tuned-h94dv\" (UID: \"627fcf92-40ec-4025-a7dd-83243fcf057c\") " pod="openshift-cluster-node-tuning-operator/tuned-h94dv"
Sep 19 01:12:33.353720 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.353300    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/627fcf92-40ec-4025-a7dd-83243fcf057c-run\") pod \"tuned-h94dv\" (UID: \"627fcf92-40ec-4025-a7dd-83243fcf057c\") " pod="openshift-cluster-node-tuning-operator/tuned-h94dv"
Sep 19 01:12:33.353720 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.353303    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ff802e32-5c32-466d-b76f-252275c6fb82-serviceca\") pod \"node-ca-vrlkw\" (UID: \"ff802e32-5c32-466d-b76f-252275c6fb82\") " pod="openshift-image-registry/node-ca-vrlkw"
Sep 19 01:12:33.353720 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.353350    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/627fcf92-40ec-4025-a7dd-83243fcf057c-host\") pod \"tuned-h94dv\" (UID: \"627fcf92-40ec-4025-a7dd-83243fcf057c\") " pod="openshift-cluster-node-tuning-operator/tuned-h94dv"
Sep 19 01:12:33.353720 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.353372    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/627fcf92-40ec-4025-a7dd-83243fcf057c-etc-kubernetes\") pod \"tuned-h94dv\" (UID: \"627fcf92-40ec-4025-a7dd-83243fcf057c\") " pod="openshift-cluster-node-tuning-operator/tuned-h94dv"
Sep 19 01:12:33.353720 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.353380    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/627fcf92-40ec-4025-a7dd-83243fcf057c-var-lib-kubelet\") pod \"tuned-h94dv\" (UID: \"627fcf92-40ec-4025-a7dd-83243fcf057c\") " pod="openshift-cluster-node-tuning-operator/tuned-h94dv"
Sep 19 01:12:33.354376 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.353397    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/627fcf92-40ec-4025-a7dd-83243fcf057c-etc-systemd\") pod \"tuned-h94dv\" (UID: \"627fcf92-40ec-4025-a7dd-83243fcf057c\") " pod="openshift-cluster-node-tuning-operator/tuned-h94dv"
Sep 19 01:12:33.354376 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.353405    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/62df040a-200c-4054-806f-252a833df006-multus-cni-dir\") pod \"multus-kwlqw\" (UID: \"62df040a-200c-4054-806f-252a833df006\") " pod="openshift-multus/multus-kwlqw"
Sep 19 01:12:33.354376 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.353412    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/62df040a-200c-4054-806f-252a833df006-host-run-multus-certs\") pod \"multus-kwlqw\" (UID: \"62df040a-200c-4054-806f-252a833df006\") " pod="openshift-multus/multus-kwlqw"
Sep 19 01:12:33.354376 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.353419    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/51c7d851-c313-4247-a65c-45f9403ddb9c-konnectivity-ca\") pod \"konnectivity-agent-8ndtv\" (UID: \"51c7d851-c313-4247-a65c-45f9403ddb9c\") " pod="kube-system/konnectivity-agent-8ndtv"
Sep 19 01:12:33.354376 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.353429    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be4767d8-9458-4c5a-bc2b-d033a86f7017-tmp-dir\") pod \"node-resolver-nx2sn\" (UID: \"be4767d8-9458-4c5a-bc2b-d033a86f7017\") " pod="openshift-dns/node-resolver-nx2sn"
Sep 19 01:12:33.354376 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.353353    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/627fcf92-40ec-4025-a7dd-83243fcf057c-run\") pod \"tuned-h94dv\" (UID: \"627fcf92-40ec-4025-a7dd-83243fcf057c\") " pod="openshift-cluster-node-tuning-operator/tuned-h94dv"
Sep 19 01:12:33.354376 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.353455    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2f7r\" (UniqueName: \"kubernetes.io/projected/be4767d8-9458-4c5a-bc2b-d033a86f7017-kube-api-access-v2f7r\") pod \"node-resolver-nx2sn\" (UID: \"be4767d8-9458-4c5a-bc2b-d033a86f7017\") " pod="openshift-dns/node-resolver-nx2sn"
Sep 19 01:12:33.354376 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.353450    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/627fcf92-40ec-4025-a7dd-83243fcf057c-var-lib-kubelet\") pod \"tuned-h94dv\" (UID: \"627fcf92-40ec-4025-a7dd-83243fcf057c\") " pod="openshift-cluster-node-tuning-operator/tuned-h94dv"
Sep 19 01:12:33.354376 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.353474    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/62df040a-200c-4054-806f-252a833df006-multus-cni-dir\") pod \"multus-kwlqw\" (UID: \"62df040a-200c-4054-806f-252a833df006\") " pod="openshift-multus/multus-kwlqw"
Sep 19 01:12:33.354376 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.353517    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgtk8\" (UniqueName: \"kubernetes.io/projected/627fcf92-40ec-4025-a7dd-83243fcf057c-kube-api-access-lgtk8\") pod \"tuned-h94dv\" (UID: \"627fcf92-40ec-4025-a7dd-83243fcf057c\") " pod="openshift-cluster-node-tuning-operator/tuned-h94dv"
Sep 19 01:12:33.354376 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.353553    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/62df040a-200c-4054-806f-252a833df006-host-run-netns\") pod \"multus-kwlqw\" (UID: \"62df040a-200c-4054-806f-252a833df006\") " pod="openshift-multus/multus-kwlqw"
Sep 19 01:12:33.354376 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.353550    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/62df040a-200c-4054-806f-252a833df006-multus-daemon-config\") pod \"multus-kwlqw\" (UID: \"62df040a-200c-4054-806f-252a833df006\") " pod="openshift-multus/multus-kwlqw"
Sep 19 01:12:33.354376 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.353591    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/62df040a-200c-4054-806f-252a833df006-host-run-netns\") pod \"multus-kwlqw\" (UID: \"62df040a-200c-4054-806f-252a833df006\") " pod="openshift-multus/multus-kwlqw"
Sep 19 01:12:33.354376 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.353620    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/62df040a-200c-4054-806f-252a833df006-etc-kubernetes\") pod \"multus-kwlqw\" (UID: \"62df040a-200c-4054-806f-252a833df006\") " pod="openshift-multus/multus-kwlqw"
Sep 19 01:12:33.354376 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.353649    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8c9j\" (UniqueName: \"kubernetes.io/projected/b0be445f-6160-4532-8a71-55bfdf782241-kube-api-access-g8c9j\") pod \"network-check-target-lmcnn\" (UID: \"b0be445f-6160-4532-8a71-55bfdf782241\") " pod="openshift-network-diagnostics/network-check-target-lmcnn"
Sep 19 01:12:33.354376 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.353674    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/627fcf92-40ec-4025-a7dd-83243fcf057c-etc-modprobe-d\") pod \"tuned-h94dv\" (UID: \"627fcf92-40ec-4025-a7dd-83243fcf057c\") " pod="openshift-cluster-node-tuning-operator/tuned-h94dv"
Sep 19 01:12:33.354376 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.353688    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/62df040a-200c-4054-806f-252a833df006-etc-kubernetes\") pod \"multus-kwlqw\" (UID: \"62df040a-200c-4054-806f-252a833df006\") " pod="openshift-multus/multus-kwlqw"
Sep 19 01:12:33.354376 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.353695    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/627fcf92-40ec-4025-a7dd-83243fcf057c-etc-sysctl-d\") pod \"tuned-h94dv\" (UID: \"627fcf92-40ec-4025-a7dd-83243fcf057c\") " pod="openshift-cluster-node-tuning-operator/tuned-h94dv"
Sep 19 01:12:33.354942 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.353714    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be4767d8-9458-4c5a-bc2b-d033a86f7017-tmp-dir\") pod \"node-resolver-nx2sn\" (UID: \"be4767d8-9458-4c5a-bc2b-d033a86f7017\") " pod="openshift-dns/node-resolver-nx2sn"
Sep 19 01:12:33.354942 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.353738    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/62df040a-200c-4054-806f-252a833df006-multus-conf-dir\") pod \"multus-kwlqw\" (UID: \"62df040a-200c-4054-806f-252a833df006\") " pod="openshift-multus/multus-kwlqw"
Sep 19 01:12:33.354942 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.353764    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/627fcf92-40ec-4025-a7dd-83243fcf057c-etc-modprobe-d\") pod \"tuned-h94dv\" (UID: \"627fcf92-40ec-4025-a7dd-83243fcf057c\") " pod="openshift-cluster-node-tuning-operator/tuned-h94dv"
Sep 19 01:12:33.354942 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.353771    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ff802e32-5c32-466d-b76f-252275c6fb82-host\") pod \"node-ca-vrlkw\" (UID: \"ff802e32-5c32-466d-b76f-252275c6fb82\") " pod="openshift-image-registry/node-ca-vrlkw"
Sep 19 01:12:33.354942 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.353775    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/62df040a-200c-4054-806f-252a833df006-multus-conf-dir\") pod \"multus-kwlqw\" (UID: \"62df040a-200c-4054-806f-252a833df006\") " pod="openshift-multus/multus-kwlqw"
Sep 19 01:12:33.354942 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.353806    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ff802e32-5c32-466d-b76f-252275c6fb82-host\") pod \"node-ca-vrlkw\" (UID: \"ff802e32-5c32-466d-b76f-252275c6fb82\") " pod="openshift-image-registry/node-ca-vrlkw"
Sep 19 01:12:33.354942 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.353913    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/627fcf92-40ec-4025-a7dd-83243fcf057c-etc-sysctl-d\") pod \"tuned-h94dv\" (UID: \"627fcf92-40ec-4025-a7dd-83243fcf057c\") " pod="openshift-cluster-node-tuning-operator/tuned-h94dv"
Sep 19 01:12:33.355767 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.355720    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/627fcf92-40ec-4025-a7dd-83243fcf057c-etc-tuned\") pod \"tuned-h94dv\" (UID: \"627fcf92-40ec-4025-a7dd-83243fcf057c\") " pod="openshift-cluster-node-tuning-operator/tuned-h94dv"
Sep 19 01:12:33.355889 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.355843    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/627fcf92-40ec-4025-a7dd-83243fcf057c-tmp\") pod \"tuned-h94dv\" (UID: \"627fcf92-40ec-4025-a7dd-83243fcf057c\") " pod="openshift-cluster-node-tuning-operator/tuned-h94dv"
Sep 19 01:12:33.356063 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.356045    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/51c7d851-c313-4247-a65c-45f9403ddb9c-agent-certs\") pod \"konnectivity-agent-8ndtv\" (UID: \"51c7d851-c313-4247-a65c-45f9403ddb9c\") " pod="kube-system/konnectivity-agent-8ndtv"
Sep 19 01:12:33.361899 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.361882    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrtwg\" (UniqueName: \"kubernetes.io/projected/62df040a-200c-4054-806f-252a833df006-kube-api-access-lrtwg\") pod \"multus-kwlqw\" (UID: \"62df040a-200c-4054-806f-252a833df006\") " pod="openshift-multus/multus-kwlqw"
Sep 19 01:12:33.362381 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:33.362359    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 19 01:12:33.362472 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:33.362386    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 19 01:12:33.362472 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:33.362400    2581 projected.go:196] Error preparing data for projected volume kube-api-access-g8c9j for pod openshift-network-diagnostics/network-check-target-lmcnn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 01:12:33.362472 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:33.362465    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b0be445f-6160-4532-8a71-55bfdf782241-kube-api-access-g8c9j podName:b0be445f-6160-4532-8a71-55bfdf782241 nodeName:}" failed. No retries permitted until 2026-09-19 01:12:33.862445581 +0000 UTC m=+4.750114650 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-g8c9j" (UniqueName: "kubernetes.io/projected/b0be445f-6160-4532-8a71-55bfdf782241-kube-api-access-g8c9j") pod "network-check-target-lmcnn" (UID: "b0be445f-6160-4532-8a71-55bfdf782241") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 01:12:33.364217 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.364195    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2f7r\" (UniqueName: \"kubernetes.io/projected/be4767d8-9458-4c5a-bc2b-d033a86f7017-kube-api-access-v2f7r\") pod \"node-resolver-nx2sn\" (UID: \"be4767d8-9458-4c5a-bc2b-d033a86f7017\") " pod="openshift-dns/node-resolver-nx2sn"
Sep 19 01:12:33.364559 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.364538    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x27vk\" (UniqueName: \"kubernetes.io/projected/ff802e32-5c32-466d-b76f-252275c6fb82-kube-api-access-x27vk\") pod \"node-ca-vrlkw\" (UID: \"ff802e32-5c32-466d-b76f-252275c6fb82\") " pod="openshift-image-registry/node-ca-vrlkw"
Sep 19 01:12:33.364804 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.364784    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgtk8\" (UniqueName: \"kubernetes.io/projected/627fcf92-40ec-4025-a7dd-83243fcf057c-kube-api-access-lgtk8\") pod \"tuned-h94dv\" (UID: \"627fcf92-40ec-4025-a7dd-83243fcf057c\") " pod="openshift-cluster-node-tuning-operator/tuned-h94dv"
Sep 19 01:12:33.376308 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.376249    2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 19 01:12:33.412461 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.412433    2581 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 19 01:12:33.441937 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.441909    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-42mjj"
Sep 19 01:12:33.453270 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.453252    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lsdhk"
Sep 19 01:12:33.465095 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.465068    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xmldk"
Sep 19 01:12:33.472250 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.472228    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-x785d"
Sep 19 01:12:33.480483 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.480464    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vrlkw"
Sep 19 01:12:33.487087 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.487062    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-nx2sn"
Sep 19 01:12:33.494662 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.494646    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-kwlqw"
Sep 19 01:12:33.503209 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.503187    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-h94dv"
Sep 19 01:12:33.507929 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.507895    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8ndtv"
Sep 19 01:12:33.756706 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.756625    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2ff7ef56-5c24-40a5-84d6-7259cf83a29c-original-pull-secret\") pod \"global-pull-secret-syncer-hrgqz\" (UID: \"2ff7ef56-5c24-40a5-84d6-7259cf83a29c\") " pod="kube-system/global-pull-secret-syncer-hrgqz"
Sep 19 01:12:33.756706 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.756683    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3884e026-09ad-4bc8-bcb8-ae622d873aea-metrics-certs\") pod \"network-metrics-daemon-dkpb2\" (UID: \"3884e026-09ad-4bc8-bcb8-ae622d873aea\") " pod="openshift-multus/network-metrics-daemon-dkpb2"
Sep 19 01:12:33.756948 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:33.756797    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 19 01:12:33.756948 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:33.756827    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 01:12:33.756948 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:33.756888    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ff7ef56-5c24-40a5-84d6-7259cf83a29c-original-pull-secret podName:2ff7ef56-5c24-40a5-84d6-7259cf83a29c nodeName:}" failed. No retries permitted until 2026-09-19 01:12:34.756843104 +0000 UTC m=+5.644512179 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2ff7ef56-5c24-40a5-84d6-7259cf83a29c-original-pull-secret") pod "global-pull-secret-syncer-hrgqz" (UID: "2ff7ef56-5c24-40a5-84d6-7259cf83a29c") : object "kube-system"/"original-pull-secret" not registered
Sep 19 01:12:33.756948 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:33.756907    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3884e026-09ad-4bc8-bcb8-ae622d873aea-metrics-certs podName:3884e026-09ad-4bc8-bcb8-ae622d873aea nodeName:}" failed. No retries permitted until 2026-09-19 01:12:34.756899334 +0000 UTC m=+5.644568388 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3884e026-09ad-4bc8-bcb8-ae622d873aea-metrics-certs") pod "network-metrics-daemon-dkpb2" (UID: "3884e026-09ad-4bc8-bcb8-ae622d873aea") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 01:12:33.957792 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:33.957758    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8c9j\" (UniqueName: \"kubernetes.io/projected/b0be445f-6160-4532-8a71-55bfdf782241-kube-api-access-g8c9j\") pod \"network-check-target-lmcnn\" (UID: \"b0be445f-6160-4532-8a71-55bfdf782241\") " pod="openshift-network-diagnostics/network-check-target-lmcnn"
Sep 19 01:12:33.957952 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:33.957938    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 19 01:12:33.958018 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:33.957957    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 19 01:12:33.958018 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:33.957967    2581 projected.go:196] Error preparing data for projected volume kube-api-access-g8c9j for pod openshift-network-diagnostics/network-check-target-lmcnn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 01:12:33.958096 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:33.958020    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b0be445f-6160-4532-8a71-55bfdf782241-kube-api-access-g8c9j podName:b0be445f-6160-4532-8a71-55bfdf782241 nodeName:}" failed. No retries permitted until 2026-09-19 01:12:34.958004365 +0000 UTC m=+5.845673433 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-g8c9j" (UniqueName: "kubernetes.io/projected/b0be445f-6160-4532-8a71-55bfdf782241-kube-api-access-g8c9j") pod "network-check-target-lmcnn" (UID: "b0be445f-6160-4532-8a71-55bfdf782241") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 01:12:34.197665 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:34.197618    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaf84f55_01ed_4c93_81d0_f8e545bf90bf.slice/crio-e64506dc390c9f62cb4a7ed000d4448bac86bfb72f069de769f32312f02c2b6d WatchSource:0}: Error finding container e64506dc390c9f62cb4a7ed000d4448bac86bfb72f069de769f32312f02c2b6d: Status 404 returned error can't find the container with id e64506dc390c9f62cb4a7ed000d4448bac86bfb72f069de769f32312f02c2b6d
Sep 19 01:12:34.198425 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:34.198357    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-18 01:07:33 +0000 UTC" deadline="2028-06-06 06:47:53.63311117 +0000 UTC"
Sep 19 01:12:34.198425 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:34.198384    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15029h35m19.434731575s"
Sep 19 01:12:34.201939 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:34.201915    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62df040a_200c_4054_806f_252a833df006.slice/crio-11263c7108673620c8047d4978e6111919ec433f27d7c5b03173de15c3001805 WatchSource:0}: Error finding container 11263c7108673620c8047d4978e6111919ec433f27d7c5b03173de15c3001805: Status 404 returned error can't find the container with id 11263c7108673620c8047d4978e6111919ec433f27d7c5b03173de15c3001805
Sep 19 01:12:34.204316 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:34.204293    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe4767d8_9458_4c5a_bc2b_d033a86f7017.slice/crio-f822c8e549a87bc7f4135c2ad9d7ea3b8f52262e72f4dd8becf1b98897e19c72 WatchSource:0}: Error finding container f822c8e549a87bc7f4135c2ad9d7ea3b8f52262e72f4dd8becf1b98897e19c72: Status 404 returned error can't find the container with id f822c8e549a87bc7f4135c2ad9d7ea3b8f52262e72f4dd8becf1b98897e19c72
Sep 19 01:12:34.205464 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:34.205412    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff802e32_5c32_466d_b76f_252275c6fb82.slice/crio-32ece94223a07d4676596e939bb23dd1bbccc2c678f010fe0f990d720c39d06b WatchSource:0}: Error finding container 32ece94223a07d4676596e939bb23dd1bbccc2c678f010fe0f990d720c39d06b: Status 404 returned error can't find the container with id 32ece94223a07d4676596e939bb23dd1bbccc2c678f010fe0f990d720c39d06b
Sep 19 01:12:34.206444 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:34.206402    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd16ba40c_ab77_4204_b80e_d58f3a57e49a.slice/crio-c19fcdb6e199a26bbcdc8094ae86a415a87f929948a09e2a4254872574bb38ea WatchSource:0}: Error finding container c19fcdb6e199a26bbcdc8094ae86a415a87f929948a09e2a4254872574bb38ea: Status 404 returned error can't find the container with id c19fcdb6e199a26bbcdc8094ae86a415a87f929948a09e2a4254872574bb38ea
Sep 19 01:12:34.206921 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:34.206895    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod627fcf92_40ec_4025_a7dd_83243fcf057c.slice/crio-6b4e389e4c1d135a7ac32795757eba573252dcada618cc81ef1143dd3439f653 WatchSource:0}: Error finding container 6b4e389e4c1d135a7ac32795757eba573252dcada618cc81ef1143dd3439f653: Status 404 returned error can't find the container with id 6b4e389e4c1d135a7ac32795757eba573252dcada618cc81ef1143dd3439f653
Sep 19 01:12:34.208212 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:34.208182    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda190ceb2_3036_4697_89d0_e103dca9cf5a.slice/crio-1381a0fe3b2c8c28e75e4f264c7793bfc269d70131ae17089b7a1bafe11e000d WatchSource:0}: Error finding container 1381a0fe3b2c8c28e75e4f264c7793bfc269d70131ae17089b7a1bafe11e000d: Status 404 returned error can't find the container with id 1381a0fe3b2c8c28e75e4f264c7793bfc269d70131ae17089b7a1bafe11e000d
Sep 19 01:12:34.210842 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:34.210815    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51c7d851_c313_4247_a65c_45f9403ddb9c.slice/crio-c743b972ed5c7266de75ced132e2ef9a94f6185db6c35c8b98f2d68dae7b28e7 WatchSource:0}: Error finding container c743b972ed5c7266de75ced132e2ef9a94f6185db6c35c8b98f2d68dae7b28e7: Status 404 returned error can't find the container with id c743b972ed5c7266de75ced132e2ef9a94f6185db6c35c8b98f2d68dae7b28e7
Sep 19 01:12:34.211963 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:12:34.211625    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf759a12f_5b6f_4924_96f2_dd5e00fc1348.slice/crio-3a0d4a8356f64b64dfd8dfdaef2c33d30718450a35d167aaf0d998b7f322cd23 WatchSource:0}: Error finding container 3a0d4a8356f64b64dfd8dfdaef2c33d30718450a35d167aaf0d998b7f322cd23: Status 404 returned error can't find the container with id 3a0d4a8356f64b64dfd8dfdaef2c33d30718450a35d167aaf0d998b7f322cd23
Sep 19 01:12:34.266242 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:34.266213    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dkpb2"
Sep 19 01:12:34.266357 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:34.266336    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dkpb2" podUID="3884e026-09ad-4bc8-bcb8-ae622d873aea"
Sep 19 01:12:34.279197 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:34.279170    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x785d" event={"ID":"f759a12f-5b6f-4924-96f2-dd5e00fc1348","Type":"ContainerStarted","Data":"3a0d4a8356f64b64dfd8dfdaef2c33d30718450a35d167aaf0d998b7f322cd23"}
Sep 19 01:12:34.280485 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:34.280463    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8ndtv" event={"ID":"51c7d851-c313-4247-a65c-45f9403ddb9c","Type":"ContainerStarted","Data":"c743b972ed5c7266de75ced132e2ef9a94f6185db6c35c8b98f2d68dae7b28e7"}
Sep 19 01:12:34.281798 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:34.281776    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lsdhk" event={"ID":"a190ceb2-3036-4697-89d0-e103dca9cf5a","Type":"ContainerStarted","Data":"1381a0fe3b2c8c28e75e4f264c7793bfc269d70131ae17089b7a1bafe11e000d"}
Sep 19 01:12:34.285701 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:34.285680    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-kwlqw" event={"ID":"62df040a-200c-4054-806f-252a833df006","Type":"ContainerStarted","Data":"11263c7108673620c8047d4978e6111919ec433f27d7c5b03173de15c3001805"}
Sep 19 01:12:34.287506 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:34.287478    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-h94dv" event={"ID":"627fcf92-40ec-4025-a7dd-83243fcf057c","Type":"ContainerStarted","Data":"6b4e389e4c1d135a7ac32795757eba573252dcada618cc81ef1143dd3439f653"}
Sep 19 01:12:34.288749 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:34.288729    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-42mjj" event={"ID":"d16ba40c-ab77-4204-b80e-d58f3a57e49a","Type":"ContainerStarted","Data":"c19fcdb6e199a26bbcdc8094ae86a415a87f929948a09e2a4254872574bb38ea"}
Sep 19 01:12:34.292226 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:34.292186    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vrlkw" event={"ID":"ff802e32-5c32-466d-b76f-252275c6fb82","Type":"ContainerStarted","Data":"32ece94223a07d4676596e939bb23dd1bbccc2c678f010fe0f990d720c39d06b"}
Sep 19 01:12:34.296123 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:34.296100    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-nx2sn" event={"ID":"be4767d8-9458-4c5a-bc2b-d033a86f7017","Type":"ContainerStarted","Data":"f822c8e549a87bc7f4135c2ad9d7ea3b8f52262e72f4dd8becf1b98897e19c72"}
Sep 19 01:12:34.302400 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:34.302381    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xmldk" event={"ID":"caf84f55-01ed-4c93-81d0-f8e545bf90bf","Type":"ContainerStarted","Data":"e64506dc390c9f62cb4a7ed000d4448bac86bfb72f069de769f32312f02c2b6d"}
Sep 19 01:12:34.764236 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:34.763938    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3884e026-09ad-4bc8-bcb8-ae622d873aea-metrics-certs\") pod \"network-metrics-daemon-dkpb2\" (UID: \"3884e026-09ad-4bc8-bcb8-ae622d873aea\") " pod="openshift-multus/network-metrics-daemon-dkpb2"
Sep 19 01:12:34.764236 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:34.764221    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2ff7ef56-5c24-40a5-84d6-7259cf83a29c-original-pull-secret\") pod \"global-pull-secret-syncer-hrgqz\" (UID: \"2ff7ef56-5c24-40a5-84d6-7259cf83a29c\") " pod="kube-system/global-pull-secret-syncer-hrgqz"
Sep 19 01:12:34.764466 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:34.764232    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 01:12:34.764466 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:34.764306    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3884e026-09ad-4bc8-bcb8-ae622d873aea-metrics-certs podName:3884e026-09ad-4bc8-bcb8-ae622d873aea nodeName:}" failed. No retries permitted until 2026-09-19 01:12:36.764285622 +0000 UTC m=+7.651954694 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3884e026-09ad-4bc8-bcb8-ae622d873aea-metrics-certs") pod "network-metrics-daemon-dkpb2" (UID: "3884e026-09ad-4bc8-bcb8-ae622d873aea") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 01:12:34.764466 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:34.764322    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 19 01:12:34.764466 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:34.764370    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ff7ef56-5c24-40a5-84d6-7259cf83a29c-original-pull-secret podName:2ff7ef56-5c24-40a5-84d6-7259cf83a29c nodeName:}" failed. No retries permitted until 2026-09-19 01:12:36.764355602 +0000 UTC m=+7.652024671 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2ff7ef56-5c24-40a5-84d6-7259cf83a29c-original-pull-secret") pod "global-pull-secret-syncer-hrgqz" (UID: "2ff7ef56-5c24-40a5-84d6-7259cf83a29c") : object "kube-system"/"original-pull-secret" not registered
Sep 19 01:12:34.966181 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:34.966141    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8c9j\" (UniqueName: \"kubernetes.io/projected/b0be445f-6160-4532-8a71-55bfdf782241-kube-api-access-g8c9j\") pod \"network-check-target-lmcnn\" (UID: \"b0be445f-6160-4532-8a71-55bfdf782241\") " pod="openshift-network-diagnostics/network-check-target-lmcnn"
Sep 19 01:12:34.966357 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:34.966295    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 19 01:12:34.966357 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:34.966318    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 19 01:12:34.966357 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:34.966331    2581 projected.go:196] Error preparing data for projected volume kube-api-access-g8c9j for pod openshift-network-diagnostics/network-check-target-lmcnn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 01:12:34.966507 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:34.966387    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b0be445f-6160-4532-8a71-55bfdf782241-kube-api-access-g8c9j podName:b0be445f-6160-4532-8a71-55bfdf782241 nodeName:}" failed. No retries permitted until 2026-09-19 01:12:36.966370048 +0000 UTC m=+7.854039120 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-g8c9j" (UniqueName: "kubernetes.io/projected/b0be445f-6160-4532-8a71-55bfdf782241-kube-api-access-g8c9j") pod "network-check-target-lmcnn" (UID: "b0be445f-6160-4532-8a71-55bfdf782241") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 01:12:35.199046 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:35.198934    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-18 01:07:33 +0000 UTC" deadline="2028-02-29 05:14:44.783147131 +0000 UTC"
Sep 19 01:12:35.199046 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:35.198987    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12676h2m9.584163155s"
Sep 19 01:12:35.266197 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:35.266169    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lmcnn"
Sep 19 01:12:35.266348 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:35.266220    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrgqz"
Sep 19 01:12:35.266348 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:35.266306    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lmcnn" podUID="b0be445f-6160-4532-8a71-55bfdf782241"
Sep 19 01:12:35.266513 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:35.266450    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrgqz" podUID="2ff7ef56-5c24-40a5-84d6-7259cf83a29c"
Sep 19 01:12:35.313146 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:35.313111    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-113.ec2.internal" event={"ID":"bf17d6182a2ab91b4c163d92cc1c844e","Type":"ContainerStarted","Data":"1962bc86675ecc41a90cfa996566aa6995e344492dfa2a2be4d3503b5d2ffede"}
Sep 19 01:12:35.337438 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:35.337406    2581 generic.go:356] "Generic (PLEG): container finished" podID="76eada0400f575bdcabf278316c26ab9" containerID="04f53f12b976d9316ba95a325c88abc80071485dcf42c4531dccfac35bcbc4ce" exitCode=0
Sep 19 01:12:35.337583 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:35.337461    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-113.ec2.internal" event={"ID":"76eada0400f575bdcabf278316c26ab9","Type":"ContainerDied","Data":"04f53f12b976d9316ba95a325c88abc80071485dcf42c4531dccfac35bcbc4ce"}
Sep 19 01:12:35.354498 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:35.354446    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-113.ec2.internal" podStartSLOduration=3.35442708 podStartE2EDuration="3.35442708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 01:12:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 01:12:35.353783583 +0000 UTC m=+6.241452662" watchObservedRunningTime="2026-09-19 01:12:35.35442708 +0000 UTC m=+6.242096158"
Sep 19 01:12:36.266524 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:36.266494    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dkpb2"
Sep 19 01:12:36.267003 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:36.266631    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dkpb2" podUID="3884e026-09ad-4bc8-bcb8-ae622d873aea"
Sep 19 01:12:36.354976 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:36.354907    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-113.ec2.internal" event={"ID":"76eada0400f575bdcabf278316c26ab9","Type":"ContainerStarted","Data":"1bb4e11b93d1bf2cab9b15a4231c1c15187d1bac2f1048c8d0c77bccc3bd0ce9"}
Sep 19 01:12:36.371375 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:36.371318    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-113.ec2.internal" podStartSLOduration=4.371297315 podStartE2EDuration="4.371297315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 01:12:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 01:12:36.370469005 +0000 UTC m=+7.258138083" watchObservedRunningTime="2026-09-19 01:12:36.371297315 +0000 UTC m=+7.258966395"
Sep 19 01:12:36.779085 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:36.778324    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3884e026-09ad-4bc8-bcb8-ae622d873aea-metrics-certs\") pod \"network-metrics-daemon-dkpb2\" (UID: \"3884e026-09ad-4bc8-bcb8-ae622d873aea\") " pod="openshift-multus/network-metrics-daemon-dkpb2"
Sep 19 01:12:36.779085 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:36.778412    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2ff7ef56-5c24-40a5-84d6-7259cf83a29c-original-pull-secret\") pod \"global-pull-secret-syncer-hrgqz\" (UID: \"2ff7ef56-5c24-40a5-84d6-7259cf83a29c\") " pod="kube-system/global-pull-secret-syncer-hrgqz"
Sep 19 01:12:36.779085 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:36.778550    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 19 01:12:36.779085 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:36.778612    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ff7ef56-5c24-40a5-84d6-7259cf83a29c-original-pull-secret podName:2ff7ef56-5c24-40a5-84d6-7259cf83a29c nodeName:}" failed. No retries permitted until 2026-09-19 01:12:40.778594725 +0000 UTC m=+11.666263794 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2ff7ef56-5c24-40a5-84d6-7259cf83a29c-original-pull-secret") pod "global-pull-secret-syncer-hrgqz" (UID: "2ff7ef56-5c24-40a5-84d6-7259cf83a29c") : object "kube-system"/"original-pull-secret" not registered
Sep 19 01:12:36.779085 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:36.778681    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 01:12:36.779085 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:36.778716    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3884e026-09ad-4bc8-bcb8-ae622d873aea-metrics-certs podName:3884e026-09ad-4bc8-bcb8-ae622d873aea nodeName:}" failed. No retries permitted until 2026-09-19 01:12:40.77870494 +0000 UTC m=+11.666374010 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3884e026-09ad-4bc8-bcb8-ae622d873aea-metrics-certs") pod "network-metrics-daemon-dkpb2" (UID: "3884e026-09ad-4bc8-bcb8-ae622d873aea") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 01:12:36.980790 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:36.980752    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8c9j\" (UniqueName: \"kubernetes.io/projected/b0be445f-6160-4532-8a71-55bfdf782241-kube-api-access-g8c9j\") pod \"network-check-target-lmcnn\" (UID: \"b0be445f-6160-4532-8a71-55bfdf782241\") " pod="openshift-network-diagnostics/network-check-target-lmcnn"
Sep 19 01:12:36.980998 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:36.980963    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 19 01:12:36.980998 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:36.980988    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 19 01:12:36.981116 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:36.981001    2581 projected.go:196] Error preparing data for projected volume kube-api-access-g8c9j for pod openshift-network-diagnostics/network-check-target-lmcnn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 01:12:36.981116 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:36.981062    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b0be445f-6160-4532-8a71-55bfdf782241-kube-api-access-g8c9j podName:b0be445f-6160-4532-8a71-55bfdf782241 nodeName:}" failed. No retries permitted until 2026-09-19 01:12:40.981041561 +0000 UTC m=+11.868710640 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-g8c9j" (UniqueName: "kubernetes.io/projected/b0be445f-6160-4532-8a71-55bfdf782241-kube-api-access-g8c9j") pod "network-check-target-lmcnn" (UID: "b0be445f-6160-4532-8a71-55bfdf782241") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 01:12:37.266576 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:37.266493    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lmcnn"
Sep 19 01:12:37.267076 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:37.266625    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lmcnn" podUID="b0be445f-6160-4532-8a71-55bfdf782241"
Sep 19 01:12:37.267076 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:37.266689    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrgqz"
Sep 19 01:12:37.267076 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:37.266763    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrgqz" podUID="2ff7ef56-5c24-40a5-84d6-7259cf83a29c"
Sep 19 01:12:38.266587 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:38.266552    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dkpb2"
Sep 19 01:12:38.267035 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:38.266668    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dkpb2" podUID="3884e026-09ad-4bc8-bcb8-ae622d873aea"
Sep 19 01:12:39.268256 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:39.266969    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lmcnn"
Sep 19 01:12:39.268256 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:39.267110    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lmcnn" podUID="b0be445f-6160-4532-8a71-55bfdf782241"
Sep 19 01:12:39.268256 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:39.267164    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrgqz"
Sep 19 01:12:39.268256 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:39.267277    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrgqz" podUID="2ff7ef56-5c24-40a5-84d6-7259cf83a29c"
Sep 19 01:12:40.267371 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:40.266746    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dkpb2"
Sep 19 01:12:40.267371 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:40.266911    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dkpb2" podUID="3884e026-09ad-4bc8-bcb8-ae622d873aea"
Sep 19 01:12:40.815689 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:40.815649    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2ff7ef56-5c24-40a5-84d6-7259cf83a29c-original-pull-secret\") pod \"global-pull-secret-syncer-hrgqz\" (UID: \"2ff7ef56-5c24-40a5-84d6-7259cf83a29c\") " pod="kube-system/global-pull-secret-syncer-hrgqz"
Sep 19 01:12:40.816148 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:40.815722    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3884e026-09ad-4bc8-bcb8-ae622d873aea-metrics-certs\") pod \"network-metrics-daemon-dkpb2\" (UID: \"3884e026-09ad-4bc8-bcb8-ae622d873aea\") " pod="openshift-multus/network-metrics-daemon-dkpb2"
Sep 19 01:12:40.816148 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:40.815854    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 01:12:40.816148 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:40.816078    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3884e026-09ad-4bc8-bcb8-ae622d873aea-metrics-certs podName:3884e026-09ad-4bc8-bcb8-ae622d873aea nodeName:}" failed. No retries permitted until 2026-09-19 01:12:48.816048014 +0000 UTC m=+19.703717091 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3884e026-09ad-4bc8-bcb8-ae622d873aea-metrics-certs") pod "network-metrics-daemon-dkpb2" (UID: "3884e026-09ad-4bc8-bcb8-ae622d873aea") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 01:12:40.816406 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:40.815952    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 19 01:12:40.816465 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:40.816451    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ff7ef56-5c24-40a5-84d6-7259cf83a29c-original-pull-secret podName:2ff7ef56-5c24-40a5-84d6-7259cf83a29c nodeName:}" failed. No retries permitted until 2026-09-19 01:12:48.81643504 +0000 UTC m=+19.704104095 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2ff7ef56-5c24-40a5-84d6-7259cf83a29c-original-pull-secret") pod "global-pull-secret-syncer-hrgqz" (UID: "2ff7ef56-5c24-40a5-84d6-7259cf83a29c") : object "kube-system"/"original-pull-secret" not registered
Sep 19 01:12:41.018444 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:41.018407    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8c9j\" (UniqueName: \"kubernetes.io/projected/b0be445f-6160-4532-8a71-55bfdf782241-kube-api-access-g8c9j\") pod \"network-check-target-lmcnn\" (UID: \"b0be445f-6160-4532-8a71-55bfdf782241\") " pod="openshift-network-diagnostics/network-check-target-lmcnn"
Sep 19 01:12:41.018621 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:41.018592    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 19 01:12:41.018621 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:41.018617    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 19 01:12:41.018734 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:41.018629    2581 projected.go:196] Error preparing data for projected volume kube-api-access-g8c9j for pod openshift-network-diagnostics/network-check-target-lmcnn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 01:12:41.018734 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:41.018692    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b0be445f-6160-4532-8a71-55bfdf782241-kube-api-access-g8c9j podName:b0be445f-6160-4532-8a71-55bfdf782241 nodeName:}" failed. No retries permitted until 2026-09-19 01:12:49.018673611 +0000 UTC m=+19.906342683 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-g8c9j" (UniqueName: "kubernetes.io/projected/b0be445f-6160-4532-8a71-55bfdf782241-kube-api-access-g8c9j") pod "network-check-target-lmcnn" (UID: "b0be445f-6160-4532-8a71-55bfdf782241") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 01:12:41.269358 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:41.269329    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lmcnn"
Sep 19 01:12:41.269510 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:41.269452    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lmcnn" podUID="b0be445f-6160-4532-8a71-55bfdf782241"
Sep 19 01:12:41.269850 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:41.269831    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrgqz"
Sep 19 01:12:41.269963 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:41.269936    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrgqz" podUID="2ff7ef56-5c24-40a5-84d6-7259cf83a29c"
Sep 19 01:12:42.266372 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:42.266333    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dkpb2"
Sep 19 01:12:42.266845 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:42.266468    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dkpb2" podUID="3884e026-09ad-4bc8-bcb8-ae622d873aea"
Sep 19 01:12:43.270399 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:43.270369    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrgqz"
Sep 19 01:12:43.270759 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:43.270484    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrgqz" podUID="2ff7ef56-5c24-40a5-84d6-7259cf83a29c"
Sep 19 01:12:43.270827 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:43.270786    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lmcnn"
Sep 19 01:12:43.270884 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:43.270850    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lmcnn" podUID="b0be445f-6160-4532-8a71-55bfdf782241"
Sep 19 01:12:44.265949 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:44.265922    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dkpb2"
Sep 19 01:12:44.266125 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:44.266032    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dkpb2" podUID="3884e026-09ad-4bc8-bcb8-ae622d873aea"
Sep 19 01:12:45.266550 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:45.266518    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrgqz"
Sep 19 01:12:45.266983 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:45.266523    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lmcnn"
Sep 19 01:12:45.266983 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:45.266648    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrgqz" podUID="2ff7ef56-5c24-40a5-84d6-7259cf83a29c"
Sep 19 01:12:45.266983 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:45.266730    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lmcnn" podUID="b0be445f-6160-4532-8a71-55bfdf782241"
Sep 19 01:12:46.266984 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:46.266950    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dkpb2"
Sep 19 01:12:46.267379 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:46.267101    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dkpb2" podUID="3884e026-09ad-4bc8-bcb8-ae622d873aea"
Sep 19 01:12:47.266100 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:47.266062    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrgqz"
Sep 19 01:12:47.266100 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:47.266087    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lmcnn"
Sep 19 01:12:47.266312 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:47.266179    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lmcnn" podUID="b0be445f-6160-4532-8a71-55bfdf782241"
Sep 19 01:12:47.266584 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:47.266559    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrgqz" podUID="2ff7ef56-5c24-40a5-84d6-7259cf83a29c"
Sep 19 01:12:48.266532 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:48.266501    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dkpb2"
Sep 19 01:12:48.266939 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:48.266613    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dkpb2" podUID="3884e026-09ad-4bc8-bcb8-ae622d873aea"
Sep 19 01:12:48.877833 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:48.877799    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2ff7ef56-5c24-40a5-84d6-7259cf83a29c-original-pull-secret\") pod \"global-pull-secret-syncer-hrgqz\" (UID: \"2ff7ef56-5c24-40a5-84d6-7259cf83a29c\") " pod="kube-system/global-pull-secret-syncer-hrgqz"
Sep 19 01:12:48.878017 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:48.877849    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3884e026-09ad-4bc8-bcb8-ae622d873aea-metrics-certs\") pod \"network-metrics-daemon-dkpb2\" (UID: \"3884e026-09ad-4bc8-bcb8-ae622d873aea\") " pod="openshift-multus/network-metrics-daemon-dkpb2"
Sep 19 01:12:48.878017 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:48.877977    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 01:12:48.878017 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:48.877979    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 19 01:12:48.878178 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:48.878030    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3884e026-09ad-4bc8-bcb8-ae622d873aea-metrics-certs podName:3884e026-09ad-4bc8-bcb8-ae622d873aea nodeName:}" failed. No retries permitted until 2026-09-19 01:13:04.878012135 +0000 UTC m=+35.765681190 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3884e026-09ad-4bc8-bcb8-ae622d873aea-metrics-certs") pod "network-metrics-daemon-dkpb2" (UID: "3884e026-09ad-4bc8-bcb8-ae622d873aea") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 01:12:48.878178 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:48.878046    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ff7ef56-5c24-40a5-84d6-7259cf83a29c-original-pull-secret podName:2ff7ef56-5c24-40a5-84d6-7259cf83a29c nodeName:}" failed. No retries permitted until 2026-09-19 01:13:04.878039952 +0000 UTC m=+35.765709007 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2ff7ef56-5c24-40a5-84d6-7259cf83a29c-original-pull-secret") pod "global-pull-secret-syncer-hrgqz" (UID: "2ff7ef56-5c24-40a5-84d6-7259cf83a29c") : object "kube-system"/"original-pull-secret" not registered
Sep 19 01:12:49.079377 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:49.079347    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8c9j\" (UniqueName: \"kubernetes.io/projected/b0be445f-6160-4532-8a71-55bfdf782241-kube-api-access-g8c9j\") pod \"network-check-target-lmcnn\" (UID: \"b0be445f-6160-4532-8a71-55bfdf782241\") " pod="openshift-network-diagnostics/network-check-target-lmcnn"
Sep 19 01:12:49.079539 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:49.079486    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 19 01:12:49.079539 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:49.079509    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 19 01:12:49.079539 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:49.079523    2581 projected.go:196] Error preparing data for projected volume kube-api-access-g8c9j for pod openshift-network-diagnostics/network-check-target-lmcnn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 01:12:49.079661 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:49.079588    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b0be445f-6160-4532-8a71-55bfdf782241-kube-api-access-g8c9j podName:b0be445f-6160-4532-8a71-55bfdf782241 nodeName:}" failed. No retries permitted until 2026-09-19 01:13:05.079569455 +0000 UTC m=+35.967238516 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-g8c9j" (UniqueName: "kubernetes.io/projected/b0be445f-6160-4532-8a71-55bfdf782241-kube-api-access-g8c9j") pod "network-check-target-lmcnn" (UID: "b0be445f-6160-4532-8a71-55bfdf782241") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 01:12:49.267062 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:49.266730    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lmcnn"
Sep 19 01:12:49.267062 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:49.266885    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lmcnn" podUID="b0be445f-6160-4532-8a71-55bfdf782241"
Sep 19 01:12:49.267062 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:49.266894    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrgqz"
Sep 19 01:12:49.267062 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:49.266982    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrgqz" podUID="2ff7ef56-5c24-40a5-84d6-7259cf83a29c"
Sep 19 01:12:50.266579 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:50.266542    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dkpb2"
Sep 19 01:12:50.266732 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:50.266664    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dkpb2" podUID="3884e026-09ad-4bc8-bcb8-ae622d873aea"
Sep 19 01:12:51.267678 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:51.267644    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrgqz"
Sep 19 01:12:51.267678 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:51.267681    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lmcnn"
Sep 19 01:12:51.268173 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:51.267744    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrgqz" podUID="2ff7ef56-5c24-40a5-84d6-7259cf83a29c"
Sep 19 01:12:51.268173 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:51.267884    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lmcnn" podUID="b0be445f-6160-4532-8a71-55bfdf782241"
Sep 19 01:12:52.266605 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:52.266584    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dkpb2"
Sep 19 01:12:52.266742 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:52.266715    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dkpb2" podUID="3884e026-09ad-4bc8-bcb8-ae622d873aea"
Sep 19 01:12:52.388613 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:52.388233    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8ndtv" event={"ID":"51c7d851-c313-4247-a65c-45f9403ddb9c","Type":"ContainerStarted","Data":"acc8234d2109a73c965d5b87a2cf66c42cc59e06c55c548d4eee2110b746622b"}
Sep 19 01:12:52.391966 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:52.391937    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-h94dv" event={"ID":"627fcf92-40ec-4025-a7dd-83243fcf057c","Type":"ContainerStarted","Data":"2057b70e2d0cc41ba81a132caf49808cedd56f4709ee995e592fe10ee72bb983"}
Sep 19 01:12:52.393513 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:52.393493    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-42mjj" event={"ID":"d16ba40c-ab77-4204-b80e-d58f3a57e49a","Type":"ContainerStarted","Data":"4c7689e6415c45507103ab418772ea80f6651364af34860fdf77933f7a5a49aa"}
Sep 19 01:12:52.437390 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:52.437233    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-h94dv" podStartSLOduration=3.527503822 podStartE2EDuration="21.437217807s" totalImagesPullingTime="17.909713985s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 01:12:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 01:12:52.437006077 +0000 UTC m=+23.324675154" watchObservedRunningTime="2026-09-19 01:12:52.437217807 +0000 UTC m=+23.324886885"
Sep 19 01:12:52.454618 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:52.454580    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-8ndtv" podStartSLOduration=3.5894979620000003 podStartE2EDuration="21.454566303s" totalImagesPullingTime="17.865068341s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 01:12:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 01:12:52.453656271 +0000 UTC m=+23.341325348" watchObservedRunningTime="2026-09-19 01:12:52.454566303 +0000 UTC m=+23.342235380"
Sep 19 01:12:52.488725 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:52.488695    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-8ndtv"
Sep 19 01:12:52.489223 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:52.489205    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-8ndtv"
Sep 19 01:12:53.266745 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:53.266574    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrgqz"
Sep 19 01:12:53.266980 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:53.266621    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lmcnn"
Sep 19 01:12:53.266980 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:53.266832    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrgqz" podUID="2ff7ef56-5c24-40a5-84d6-7259cf83a29c"
Sep 19 01:12:53.266980 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:53.266901    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lmcnn" podUID="b0be445f-6160-4532-8a71-55bfdf782241"
Sep 19 01:12:53.397567 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:53.397528    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-nx2sn" event={"ID":"be4767d8-9458-4c5a-bc2b-d033a86f7017","Type":"ContainerStarted","Data":"a1f8fdb0d3705298d37804f5a18bdb843b38bc8d7d62a44caa9b53ad8abca8bd"}
Sep 19 01:12:53.401115 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:53.401094    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x785d_f759a12f-5b6f-4924-96f2-dd5e00fc1348/ovn-acl-logging/0.log"
Sep 19 01:12:53.401529 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:53.401508    2581 generic.go:356] "Generic (PLEG): container finished" podID="f759a12f-5b6f-4924-96f2-dd5e00fc1348" containerID="8fb7d14bd01fab4f14feca00d93ebd58220099e73fbdb4c76c91b688d96fe4f6" exitCode=1
Sep 19 01:12:53.401651 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:53.401590    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x785d" event={"ID":"f759a12f-5b6f-4924-96f2-dd5e00fc1348","Type":"ContainerStarted","Data":"f6e6d6c1a4405a4f65ebc017c043d5706d0608dd9d3daaf7ee2ea3601f3013de"}
Sep 19 01:12:53.401713 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:53.401670    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x785d" event={"ID":"f759a12f-5b6f-4924-96f2-dd5e00fc1348","Type":"ContainerStarted","Data":"dd0d0aa3be3945956faf43a6d73cd0eb1f91f05b0ddccbde1d4bd9aadd9d0ef5"}
Sep 19 01:12:53.401766 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:53.401719    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x785d" event={"ID":"f759a12f-5b6f-4924-96f2-dd5e00fc1348","Type":"ContainerStarted","Data":"7ea0774ba041ebe16cb56962d19003a824db339d5efc1852ec272f8714f3ccdf"}
Sep 19 01:12:53.401819 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:53.401768    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x785d" event={"ID":"f759a12f-5b6f-4924-96f2-dd5e00fc1348","Type":"ContainerStarted","Data":"8571622c382dab90c2e7bc4ba6ca44bb3c2c05982da56678c93c868acbe83845"}
Sep 19 01:12:53.401819 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:53.401813    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x785d" event={"ID":"f759a12f-5b6f-4924-96f2-dd5e00fc1348","Type":"ContainerDied","Data":"8fb7d14bd01fab4f14feca00d93ebd58220099e73fbdb4c76c91b688d96fe4f6"}
Sep 19 01:12:53.401940 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:53.401854    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x785d" event={"ID":"f759a12f-5b6f-4924-96f2-dd5e00fc1348","Type":"ContainerStarted","Data":"577bf92a87a1e701b5bf7f6dd59621db3b1067ffc245d93d3f6962bb34880841"}
Sep 19 01:12:53.403420 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:53.403400    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lsdhk" event={"ID":"a190ceb2-3036-4697-89d0-e103dca9cf5a","Type":"ContainerStarted","Data":"dc42ea2f2f66c2a3a17268c897b4136d2da0c8ea04c9da90d6b474431c842149"}
Sep 19 01:12:53.405147 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:53.405122    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-kwlqw" event={"ID":"62df040a-200c-4054-806f-252a833df006","Type":"ContainerStarted","Data":"f807782d98fba0d046f264899352940b0f50d689e0803fd0b45df69d552436ee"}
Sep 19 01:12:53.406963 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:53.406942    2581 generic.go:356] "Generic (PLEG): container finished" podID="d16ba40c-ab77-4204-b80e-d58f3a57e49a" containerID="4c7689e6415c45507103ab418772ea80f6651364af34860fdf77933f7a5a49aa" exitCode=0
Sep 19 01:12:53.407065 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:53.407022    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-42mjj" event={"ID":"d16ba40c-ab77-4204-b80e-d58f3a57e49a","Type":"ContainerDied","Data":"4c7689e6415c45507103ab418772ea80f6651364af34860fdf77933f7a5a49aa"}
Sep 19 01:12:53.408728 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:53.408650    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vrlkw" event={"ID":"ff802e32-5c32-466d-b76f-252275c6fb82","Type":"ContainerStarted","Data":"793868a68e989b66097b9dbf5d1dd83556b82a31717219317f464b9d33186aaf"}
Sep 19 01:12:53.409066 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:53.409048    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-8ndtv"
Sep 19 01:12:53.409548 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:53.409530    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-8ndtv"
Sep 19 01:12:53.414498 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:53.414460    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-nx2sn" podStartSLOduration=4.479469617 podStartE2EDuration="22.414448651s" totalImagesPullingTime="17.934979034s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 01:12:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 01:12:53.413938522 +0000 UTC m=+24.301607622" watchObservedRunningTime="2026-09-19 01:12:53.414448651 +0000 UTC m=+24.302117728"
Sep 19 01:12:53.430495 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:53.430422    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-vrlkw" podStartSLOduration=4.50245913 podStartE2EDuration="22.430412083s" totalImagesPullingTime="17.927952953s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 01:12:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 01:12:53.430026902 +0000 UTC m=+24.317695995" watchObservedRunningTime="2026-09-19 01:12:53.430412083 +0000 UTC m=+24.318081159"
Sep 19 01:12:53.494300 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:53.494266    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-kwlqw" podStartSLOduration=4.328981705 podStartE2EDuration="22.494254434s" totalImagesPullingTime="18.165272729s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 01:12:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 01:12:53.494175843 +0000 UTC m=+24.381844937" watchObservedRunningTime="2026-09-19 01:12:53.494254434 +0000 UTC m=+24.381923511"
Sep 19 01:12:53.901801 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:53.901780    2581 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 19 01:12:54.229791 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:54.229692    2581 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-19T01:12:53.901796822Z","UUID":"87cefad5-91d5-4e21-b3ad-952acad05f53","Handler":null,"Name":"","Endpoint":""}
Sep 19 01:12:54.231294 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:54.231263    2581 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 19 01:12:54.231448 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:54.231310    2581 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 19 01:12:54.266023 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:54.265998    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dkpb2"
Sep 19 01:12:54.266163 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:54.266114    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dkpb2" podUID="3884e026-09ad-4bc8-bcb8-ae622d873aea"
Sep 19 01:12:54.414015 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:54.413980    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lsdhk" event={"ID":"a190ceb2-3036-4697-89d0-e103dca9cf5a","Type":"ContainerStarted","Data":"452d31d59de0747c3f10a5614fc697a28adaef602b6058cf3735d72881f8baf7"}
Sep 19 01:12:54.416026 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:54.415994    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xmldk" event={"ID":"caf84f55-01ed-4c93-81d0-f8e545bf90bf","Type":"ContainerStarted","Data":"a6ff08dde2f372be184e850a8cb0f7c67013014443b7591b2a603e7591dee460"}
Sep 19 01:12:54.436448 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:54.436400    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-xmldk" podStartSLOduration=5.500751037 podStartE2EDuration="23.436385427s" totalImagesPullingTime="17.93563439s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 01:12:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 01:12:54.434314332 +0000 UTC m=+25.321983426" watchObservedRunningTime="2026-09-19 01:12:54.436385427 +0000 UTC m=+25.324054517"
Sep 19 01:12:55.266804 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:55.266709    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrgqz"
Sep 19 01:12:55.266804 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:55.266756    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lmcnn"
Sep 19 01:12:55.267011 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:55.266841    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrgqz" podUID="2ff7ef56-5c24-40a5-84d6-7259cf83a29c"
Sep 19 01:12:55.267011 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:55.266981    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lmcnn" podUID="b0be445f-6160-4532-8a71-55bfdf782241"
Sep 19 01:12:55.421659 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:55.421635    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x785d_f759a12f-5b6f-4924-96f2-dd5e00fc1348/ovn-acl-logging/0.log"
Sep 19 01:12:55.422132 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:55.422076    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x785d" event={"ID":"f759a12f-5b6f-4924-96f2-dd5e00fc1348","Type":"ContainerStarted","Data":"69e6b532d07634b8d660b96fa5b37beba3de0a60ec1a5ae02bcacd31808e4eca"}
Sep 19 01:12:55.424413 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:55.424385    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lsdhk" event={"ID":"a190ceb2-3036-4697-89d0-e103dca9cf5a","Type":"ContainerStarted","Data":"b598ae89ad2460655f242a3e5e75a57327ae7efd5bfccb54556166c6d6cbb106"}
Sep 19 01:12:55.445776 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:55.445719    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lsdhk" podStartSLOduration=4.21645181 podStartE2EDuration="24.445706763s" totalImagesPullingTime="20.229254953s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 01:12:31 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 01:12:55.445438279 +0000 UTC m=+26.333107384" watchObservedRunningTime="2026-09-19 01:12:55.445706763 +0000 UTC m=+26.333375839"
Sep 19 01:12:56.266701 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:56.266664    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dkpb2"
Sep 19 01:12:56.266907 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:56.266801    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dkpb2" podUID="3884e026-09ad-4bc8-bcb8-ae622d873aea"
Sep 19 01:12:57.266833 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:57.266668    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrgqz"
Sep 19 01:12:57.267286 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:57.266669    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lmcnn"
Sep 19 01:12:57.267286 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:57.266947    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrgqz" podUID="2ff7ef56-5c24-40a5-84d6-7259cf83a29c"
Sep 19 01:12:57.267286 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:57.267080    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lmcnn" podUID="b0be445f-6160-4532-8a71-55bfdf782241"
Sep 19 01:12:58.266385 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:58.266356    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dkpb2"
Sep 19 01:12:58.266521 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:58.266450    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dkpb2" podUID="3884e026-09ad-4bc8-bcb8-ae622d873aea"
Sep 19 01:12:58.435906 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:58.435882    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x785d_f759a12f-5b6f-4924-96f2-dd5e00fc1348/ovn-acl-logging/0.log"
Sep 19 01:12:58.436668 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:58.436292    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x785d" event={"ID":"f759a12f-5b6f-4924-96f2-dd5e00fc1348","Type":"ContainerStarted","Data":"5ba17192b44fbe2c136d38d76bb4acd7ded794c47bd4ed238ae522e559110557"}
Sep 19 01:12:58.436668 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:58.436512    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-x785d"
Sep 19 01:12:58.436668 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:58.436576    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-x785d"
Sep 19 01:12:58.436829 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:58.436746    2581 scope.go:117] "RemoveContainer" containerID="8fb7d14bd01fab4f14feca00d93ebd58220099e73fbdb4c76c91b688d96fe4f6"
Sep 19 01:12:58.438799 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:58.438779    2581 generic.go:356] "Generic (PLEG): container finished" podID="d16ba40c-ab77-4204-b80e-d58f3a57e49a" containerID="c48311a3c11fe989dd2f8a7e753d420904ebdb50e1f0885ba7dbb6ddde9e908a" exitCode=0
Sep 19 01:12:58.438909 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:58.438817    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-42mjj" event={"ID":"d16ba40c-ab77-4204-b80e-d58f3a57e49a","Type":"ContainerDied","Data":"c48311a3c11fe989dd2f8a7e753d420904ebdb50e1f0885ba7dbb6ddde9e908a"}
Sep 19 01:12:58.452489 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:58.452472    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x785d"
Sep 19 01:12:58.453827 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:58.453806    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x785d"
Sep 19 01:12:59.266519 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:59.266369    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrgqz"
Sep 19 01:12:59.266606 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:59.266428    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lmcnn"
Sep 19 01:12:59.266659 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:59.266616    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrgqz" podUID="2ff7ef56-5c24-40a5-84d6-7259cf83a29c"
Sep 19 01:12:59.266734 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:59.266710    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lmcnn" podUID="b0be445f-6160-4532-8a71-55bfdf782241"
Sep 19 01:12:59.443427 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:59.443391    2581 generic.go:356] "Generic (PLEG): container finished" podID="d16ba40c-ab77-4204-b80e-d58f3a57e49a" containerID="daa9217218811d874ecdb5340a99cab093d7cab7ae4b6bec24f6ba08286740b3" exitCode=0
Sep 19 01:12:59.443800 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:59.443481    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-42mjj" event={"ID":"d16ba40c-ab77-4204-b80e-d58f3a57e49a","Type":"ContainerDied","Data":"daa9217218811d874ecdb5340a99cab093d7cab7ae4b6bec24f6ba08286740b3"}
Sep 19 01:12:59.447571 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:59.447553    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x785d_f759a12f-5b6f-4924-96f2-dd5e00fc1348/ovn-acl-logging/0.log"
Sep 19 01:12:59.447942 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:59.447920    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x785d" event={"ID":"f759a12f-5b6f-4924-96f2-dd5e00fc1348","Type":"ContainerStarted","Data":"ff299f98a77664496bda97d9114a3278672d74cbe52de508a2fcdb197170eda2"}
Sep 19 01:12:59.448209 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:59.448195    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-x785d"
Sep 19 01:12:59.602788 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:59.602725    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-x785d" podStartSLOduration=10.676477453 podStartE2EDuration="28.6027058s" totalImagesPullingTime="17.926228347s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 01:12:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 01:12:59.503920519 +0000 UTC m=+30.391589609" watchObservedRunningTime="2026-09-19 01:12:59.6027058 +0000 UTC m=+30.490374878"
Sep 19 01:12:59.602970 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:59.602956    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hrgqz"]
Sep 19 01:12:59.603094 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:59.603082    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrgqz"
Sep 19 01:12:59.603224 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:59.603190    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrgqz" podUID="2ff7ef56-5c24-40a5-84d6-7259cf83a29c"
Sep 19 01:12:59.606515 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:59.606488    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dkpb2"]
Sep 19 01:12:59.606664 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:59.606610    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dkpb2"
Sep 19 01:12:59.606752 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:59.606719    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dkpb2" podUID="3884e026-09ad-4bc8-bcb8-ae622d873aea"
Sep 19 01:12:59.607157 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:59.607131    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lmcnn"]
Sep 19 01:12:59.607257 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:12:59.607226    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lmcnn"
Sep 19 01:12:59.607342 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:12:59.607323    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lmcnn" podUID="b0be445f-6160-4532-8a71-55bfdf782241"
Sep 19 01:13:00.452942 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:00.452912    2581 generic.go:356] "Generic (PLEG): container finished" podID="d16ba40c-ab77-4204-b80e-d58f3a57e49a" containerID="05e6e422c9477ffda652e0ff2b5f7c03ddc6e75c492032a20ae150ed71018c86" exitCode=0
Sep 19 01:13:00.453487 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:00.453004    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-42mjj" event={"ID":"d16ba40c-ab77-4204-b80e-d58f3a57e49a","Type":"ContainerDied","Data":"05e6e422c9477ffda652e0ff2b5f7c03ddc6e75c492032a20ae150ed71018c86"}
Sep 19 01:13:01.267032 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:01.266989    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrgqz"
Sep 19 01:13:01.267204 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:13:01.267100    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrgqz" podUID="2ff7ef56-5c24-40a5-84d6-7259cf83a29c"
Sep 19 01:13:01.267204 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:01.267183    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lmcnn"
Sep 19 01:13:01.267312 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:01.267294    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dkpb2"
Sep 19 01:13:01.267360 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:13:01.267327    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lmcnn" podUID="b0be445f-6160-4532-8a71-55bfdf782241"
Sep 19 01:13:01.267409 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:13:01.267384    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dkpb2" podUID="3884e026-09ad-4bc8-bcb8-ae622d873aea"
Sep 19 01:13:03.266204 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:03.266167    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dkpb2"
Sep 19 01:13:03.266577 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:03.266167    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrgqz"
Sep 19 01:13:03.266577 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:13:03.266312    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dkpb2" podUID="3884e026-09ad-4bc8-bcb8-ae622d873aea"
Sep 19 01:13:03.266577 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:13:03.266415    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrgqz" podUID="2ff7ef56-5c24-40a5-84d6-7259cf83a29c"
Sep 19 01:13:03.266577 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:03.266167    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lmcnn"
Sep 19 01:13:03.266577 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:13:03.266518    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lmcnn" podUID="b0be445f-6160-4532-8a71-55bfdf782241"
Sep 19 01:13:04.902227 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:04.902085    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3884e026-09ad-4bc8-bcb8-ae622d873aea-metrics-certs\") pod \"network-metrics-daemon-dkpb2\" (UID: \"3884e026-09ad-4bc8-bcb8-ae622d873aea\") " pod="openshift-multus/network-metrics-daemon-dkpb2"
Sep 19 01:13:04.902522 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:13:04.902231    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 01:13:04.902522 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:04.902272    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2ff7ef56-5c24-40a5-84d6-7259cf83a29c-original-pull-secret\") pod \"global-pull-secret-syncer-hrgqz\" (UID: \"2ff7ef56-5c24-40a5-84d6-7259cf83a29c\") " pod="kube-system/global-pull-secret-syncer-hrgqz"
Sep 19 01:13:04.902522 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:13:04.902306    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3884e026-09ad-4bc8-bcb8-ae622d873aea-metrics-certs podName:3884e026-09ad-4bc8-bcb8-ae622d873aea nodeName:}" failed. No retries permitted until 2026-09-19 01:13:36.902284203 +0000 UTC m=+67.789953268 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3884e026-09ad-4bc8-bcb8-ae622d873aea-metrics-certs") pod "network-metrics-daemon-dkpb2" (UID: "3884e026-09ad-4bc8-bcb8-ae622d873aea") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 01:13:04.902522 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:13:04.902349    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 19 01:13:04.902522 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:13:04.902397    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ff7ef56-5c24-40a5-84d6-7259cf83a29c-original-pull-secret podName:2ff7ef56-5c24-40a5-84d6-7259cf83a29c nodeName:}" failed. No retries permitted until 2026-09-19 01:13:36.902383882 +0000 UTC m=+67.790052941 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2ff7ef56-5c24-40a5-84d6-7259cf83a29c-original-pull-secret") pod "global-pull-secret-syncer-hrgqz" (UID: "2ff7ef56-5c24-40a5-84d6-7259cf83a29c") : object "kube-system"/"original-pull-secret" not registered
Sep 19 01:13:05.073398 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:05.073376    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-113.ec2.internal" event="NodeReady"
Sep 19 01:13:05.073518 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:05.073504    2581 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 19 01:13:05.103584 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:05.103558    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8c9j\" (UniqueName: \"kubernetes.io/projected/b0be445f-6160-4532-8a71-55bfdf782241-kube-api-access-g8c9j\") pod \"network-check-target-lmcnn\" (UID: \"b0be445f-6160-4532-8a71-55bfdf782241\") " pod="openshift-network-diagnostics/network-check-target-lmcnn"
Sep 19 01:13:05.103720 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:13:05.103704    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 19 01:13:05.103776 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:13:05.103764    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 19 01:13:05.103811 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:13:05.103782    2581 projected.go:196] Error preparing data for projected volume kube-api-access-g8c9j for pod openshift-network-diagnostics/network-check-target-lmcnn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 01:13:05.103843 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:13:05.103827    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b0be445f-6160-4532-8a71-55bfdf782241-kube-api-access-g8c9j podName:b0be445f-6160-4532-8a71-55bfdf782241 nodeName:}" failed. No retries permitted until 2026-09-19 01:13:37.103814751 +0000 UTC m=+67.991483806 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-g8c9j" (UniqueName: "kubernetes.io/projected/b0be445f-6160-4532-8a71-55bfdf782241-kube-api-access-g8c9j") pod "network-check-target-lmcnn" (UID: "b0be445f-6160-4532-8a71-55bfdf782241") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 01:13:05.182570 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:05.182547    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-v6mxp"]
Sep 19 01:13:05.199258 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:05.199237    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-587d7ff8fc-hhv2t"]
Sep 19 01:13:05.199390 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:05.199373    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-v6mxp"
Sep 19 01:13:05.210105 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:05.210089    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 19 01:13:05.212187 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:05.212166    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 19 01:13:05.213007 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:05.212994    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-h9l7c\""
Sep 19 01:13:05.213251 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:05.213238    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-9wqc6"]
Sep 19 01:13:05.213405 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:05.213378    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-587d7ff8fc-hhv2t"
Sep 19 01:13:05.221401 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:05.221386    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 19 01:13:05.224228 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:05.224215    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 19 01:13:05.224635 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:05.224615    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 19 01:13:05.224815 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:05.224803    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-5r66s\""
Sep 19 01:13:05.231760 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:05.231740    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-v6mxp"]
Sep 19 01:13:05.231848 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:05.231775    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9wqc6"]
Sep 19 01:13:05.231848 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:05.231789    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-587d7ff8fc-hhv2t"]
Sep 19 01:13:05.231943 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:05.231844    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9wqc6"
Sep 19 01:13:05.237453 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:05.237427    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 19 01:13:05.237539 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:05.237528    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 19 01:13:05.237589 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:05.237558    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-spc29\""
Sep 19 01:13:05.239204 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:05.239187    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 19 01:13:05.244014 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:05.244000    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 19 01:13:05.266422 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:05.266396    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrgqz"
Sep 19 01:13:05.266535 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:05.266399    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dkpb2"
Sep 19 01:13:05.266535 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:05.266513    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lmcnn"
Sep 19 01:13:05.269569 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:05.269533    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 19 01:13:05.269749 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:05.269733    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-f5jnh\""
Sep 19 01:13:05.270123 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:05.270109    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lthdr\""
Sep 19 01:13:05.270900 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:05.270886    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 19 01:13:05.271412 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:05.271395    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 19 01:13:05.275236 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:05.275218    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 19 01:13:05.304967 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:05.304947    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a4338a4a-7257-495a-8e73-952e81106c45-metrics-tls\") pod \"dns-default-v6mxp\" (UID: \"a4338a4a-7257-495a-8e73-952e81106c45\") " pod="openshift-dns/dns-default-v6mxp"
Sep 19 01:13:05.305081 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:05.304973    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/49b29fdc-bbdc-44ff-901f-384537c101e5-bound-sa-token\") pod \"image-registry-587d7ff8fc-hhv2t\" (UID: \"49b29fdc-bbdc-44ff-901f-384537c101e5\") " pod="openshift-image-registry/image-registry-587d7ff8fc-hhv2t"
Sep 19 01:13:05.305081 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:05.304995    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4338a4a-7257-495a-8e73-952e81106c45-tmp-dir\") pod \"dns-default-v6mxp\" (UID: \"a4338a4a-7257-495a-8e73-952e81106c45\") " pod="openshift-dns/dns-default-v6mxp"
Sep 19 01:13:05.305081 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:05.305012    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtqsl\" (UniqueName: \"kubernetes.io/projected/49b29fdc-bbdc-44ff-901f-384537c101e5-kube-api-access-xtqsl\") pod \"image-registry-587d7ff8fc-hhv2t\" (UID: \"49b29fdc-bbdc-44ff-901f-384537c101e5\") " pod="openshift-image-registry/image-registry-587d7ff8fc-hhv2t"
Sep 19 01:13:05.305081 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:05.305029    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/49b29fdc-bbdc-44ff-901f-384537c101e5-ca-trust-extracted\") pod \"image-registry-587d7ff8fc-hhv2t\" (UID: \"49b29fdc-bbdc-44ff-901f-384537c101e5\") " pod="openshift-image-registry/image-registry-587d7ff8fc-hhv2t"
Sep 19 01:13:05.305223 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:05.305124    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pk2s6\" (UniqueName: \"kubernetes.io/projected/a4338a4a-7257-495a-8e73-952e81106c45-kube-api-access-pk2s6\") pod \"dns-default-v6mxp\" (UID: \"a4338a4a-7257-495a-8e73-952e81106c45\") " pod="openshift-dns/dns-default-v6mxp"
Sep 19 01:13:05.305223 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:05.305151    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/49b29fdc-bbdc-44ff-901f-384537c101e5-installation-pull-secrets\") pod \"image-registry-587d7ff8fc-hhv2t\" (UID: \"49b29fdc-bbdc-44ff-901f-384537c101e5\") " pod="openshift-image-registry/image-registry-587d7ff8fc-hhv2t"
Sep 19 01:13:05.305223 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:05.305177    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/49b29fdc-bbdc-44ff-901f-384537c101e5-image-registry-private-configuration\") pod \"image-registry-587d7ff8fc-hhv2t\" (UID: \"49b29fdc-bbdc-44ff-901f-384537c101e5\") " pod="openshift-image-registry/image-registry-587d7ff8fc-hhv2t"
Sep 19 01:13:05.305223 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:05.305212    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a4338a4a-7257-495a-8e73-952e81106c45-config-volume\") pod \"dns-default-v6mxp\" (UID: \"a4338a4a-7257-495a-8e73-952e81106c45\") " pod="openshift-dns/dns-default-v6mxp"
Sep 19 01:13:05.305375 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:05.305228    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/49b29fdc-bbdc-44ff-901f-384537c101e5-registry-certificates\") pod \"image-registry-587d7ff8fc-hhv2t\" (UID: \"49b29fdc-bbdc-44ff-901f-384537c101e5\") " pod="openshift-image-registry/image-registry-587d7ff8fc-hhv2t"
Sep 19 01:13:05.305375 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:05.305252    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/49b29fdc-bbdc-44ff-901f-384537c101e5-registry-tls\") pod \"image-registry-587d7ff8fc-hhv2t\" (UID: \"49b29fdc-bbdc-44ff-901f-384537c101e5\") " pod="openshift-image-registry/image-registry-587d7ff8fc-hhv2t"
Sep 19 01:13:05.305375 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:05.305274    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/49b29fdc-bbdc-44ff-901f-384537c101e5-trusted-ca\") pod \"image-registry-587d7ff8fc-hhv2t\" (UID: \"49b29fdc-bbdc-44ff-901f-384537c101e5\") " pod="openshift-image-registry/image-registry-587d7ff8fc-hhv2t"
Sep 19 01:13:05.406354 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:05.406298    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4338a4a-7257-495a-8e73-952e81106c45-tmp-dir\") pod \"dns-default-v6mxp\" (UID: \"a4338a4a-7257-495a-8e73-952e81106c45\") " pod="openshift-dns/dns-default-v6mxp"
Sep 19 01:13:05.406354 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:05.406326    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtqsl\" (UniqueName: \"kubernetes.io/projected/49b29fdc-bbdc-44ff-901f-384537c101e5-kube-api-access-xtqsl\") pod \"image-registry-587d7ff8fc-hhv2t\" (UID: \"49b29fdc-bbdc-44ff-901f-384537c101e5\") " pod="openshift-image-registry/image-registry-587d7ff8fc-hhv2t"
Sep 19 01:13:05.406354 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:05.406345    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/49b29fdc-bbdc-44ff-901f-384537c101e5-ca-trust-extracted\") pod \"image-registry-587d7ff8fc-hhv2t\" (UID: \"49b29fdc-bbdc-44ff-901f-384537c101e5\") " pod="openshift-image-registry/image-registry-587d7ff8fc-hhv2t"
Sep 19 01:13:05.406524 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:05.406367    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pk2s6\" (UniqueName: \"kubernetes.io/projected/a4338a4a-7257-495a-8e73-952e81106c45-kube-api-access-pk2s6\") pod \"dns-default-v6mxp\" (UID: \"a4338a4a-7257-495a-8e73-952e81106c45\") " pod="openshift-dns/dns-default-v6mxp"
Sep 19 01:13:05.406524 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:05.406406    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/49b29fdc-bbdc-44ff-901f-384537c101e5-installation-pull-secrets\") pod \"image-registry-587d7ff8fc-hhv2t\" (UID: \"49b29fdc-bbdc-44ff-901f-384537c101e5\") " pod="openshift-image-registry/image-registry-587d7ff8fc-hhv2t"
Sep 19 01:13:05.406524 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:05.406438    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q78hm\" (UniqueName: \"kubernetes.io/projected/eef21c42-fb10-46ba-9ba9-fff960f461a6-kube-api-access-q78hm\") pod \"ingress-canary-9wqc6\" (UID: \"eef21c42-fb10-46ba-9ba9-fff960f461a6\") " pod="openshift-ingress-canary/ingress-canary-9wqc6"
Sep 19 01:13:05.406524 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:05.406482    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/49b29fdc-bbdc-44ff-901f-384537c101e5-image-registry-private-configuration\") pod \"image-registry-587d7ff8fc-hhv2t\" (UID: \"49b29fdc-bbdc-44ff-901f-384537c101e5\") " pod="openshift-image-registry/image-registry-587d7ff8fc-hhv2t"
Sep 19 01:13:05.406524 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:05.406508    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eef21c42-fb10-46ba-9ba9-fff960f461a6-cert\") pod \"ingress-canary-9wqc6\" (UID: \"eef21c42-fb10-46ba-9ba9-fff960f461a6\") " pod="openshift-ingress-canary/ingress-canary-9wqc6"
Sep 19 01:13:05.406752 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:05.406553    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a4338a4a-7257-495a-8e73-952e81106c45-config-volume\") pod \"dns-default-v6mxp\" (UID: \"a4338a4a-7257-495a-8e73-952e81106c45\") " pod="openshift-dns/dns-default-v6mxp"
Sep 19 01:13:05.406752 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:05.406577    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/49b29fdc-bbdc-44ff-901f-384537c101e5-registry-certificates\") pod \"image-registry-587d7ff8fc-hhv2t\" (UID: \"49b29fdc-bbdc-44ff-901f-384537c101e5\") " pod="openshift-image-registry/image-registry-587d7ff8fc-hhv2t"
Sep 19 01:13:05.406752 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:05.406616    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/49b29fdc-bbdc-44ff-901f-384537c101e5-registry-tls\") pod \"image-registry-587d7ff8fc-hhv2t\" (UID: \"49b29fdc-bbdc-44ff-901f-384537c101e5\") " pod="openshift-image-registry/image-registry-587d7ff8fc-hhv2t"
Sep 19 01:13:05.406752 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:05.406655    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/49b29fdc-bbdc-44ff-901f-384537c101e5-trusted-ca\") pod \"image-registry-587d7ff8fc-hhv2t\" (UID: \"49b29fdc-bbdc-44ff-901f-384537c101e5\") " pod="openshift-image-registry/image-registry-587d7ff8fc-hhv2t"
Sep 19 01:13:05.406752 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:05.406686    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4338a4a-7257-495a-8e73-952e81106c45-tmp-dir\") pod \"dns-default-v6mxp\" (UID: \"a4338a4a-7257-495a-8e73-952e81106c45\") " pod="openshift-dns/dns-default-v6mxp"
Sep 19 01:13:05.406752 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:05.406709    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a4338a4a-7257-495a-8e73-952e81106c45-metrics-tls\") pod \"dns-default-v6mxp\" (UID: \"a4338a4a-7257-495a-8e73-952e81106c45\") " pod="openshift-dns/dns-default-v6mxp"
Sep 19 01:13:05.406752 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:05.406736    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/49b29fdc-bbdc-44ff-901f-384537c101e5-bound-sa-token\") pod \"image-registry-587d7ff8fc-hhv2t\" (UID: \"49b29fdc-bbdc-44ff-901f-384537c101e5\") " pod="openshift-image-registry/image-registry-587d7ff8fc-hhv2t"
Sep 19 01:13:05.407154 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:13:05.406786    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 19 01:13:05.407154 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:13:05.406801    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-587d7ff8fc-hhv2t: secret "image-registry-tls" not found
Sep 19 01:13:05.407154 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:13:05.406856    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/49b29fdc-bbdc-44ff-901f-384537c101e5-registry-tls podName:49b29fdc-bbdc-44ff-901f-384537c101e5 nodeName:}" failed. No retries permitted until 2026-09-19 01:13:05.906835056 +0000 UTC m=+36.794504115 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/49b29fdc-bbdc-44ff-901f-384537c101e5-registry-tls") pod "image-registry-587d7ff8fc-hhv2t" (UID: "49b29fdc-bbdc-44ff-901f-384537c101e5") : secret "image-registry-tls" not found
Sep 19 01:13:05.407154 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:05.406919    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/49b29fdc-bbdc-44ff-901f-384537c101e5-ca-trust-extracted\") pod \"image-registry-587d7ff8fc-hhv2t\" (UID: \"49b29fdc-bbdc-44ff-901f-384537c101e5\") " pod="openshift-image-registry/image-registry-587d7ff8fc-hhv2t"
Sep 19 01:13:05.407154 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:13:05.407048    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 19 01:13:05.407154 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:13:05.407122    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4338a4a-7257-495a-8e73-952e81106c45-metrics-tls podName:a4338a4a-7257-495a-8e73-952e81106c45 nodeName:}" failed. No retries permitted until 2026-09-19 01:13:05.907104792 +0000 UTC m=+36.794773852 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a4338a4a-7257-495a-8e73-952e81106c45-metrics-tls") pod "dns-default-v6mxp" (UID: "a4338a4a-7257-495a-8e73-952e81106c45") : secret "dns-default-metrics-tls" not found
Sep 19 01:13:05.407397 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:05.407189    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a4338a4a-7257-495a-8e73-952e81106c45-config-volume\") pod \"dns-default-v6mxp\" (UID: \"a4338a4a-7257-495a-8e73-952e81106c45\") " pod="openshift-dns/dns-default-v6mxp"
Sep 19 01:13:05.407431 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:05.407408    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/49b29fdc-bbdc-44ff-901f-384537c101e5-registry-certificates\") pod \"image-registry-587d7ff8fc-hhv2t\" (UID: \"49b29fdc-bbdc-44ff-901f-384537c101e5\") " pod="openshift-image-registry/image-registry-587d7ff8fc-hhv2t"
Sep 19 01:13:05.407829 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:05.407809    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/49b29fdc-bbdc-44ff-901f-384537c101e5-trusted-ca\") pod \"image-registry-587d7ff8fc-hhv2t\" (UID: \"49b29fdc-bbdc-44ff-901f-384537c101e5\") " pod="openshift-image-registry/image-registry-587d7ff8fc-hhv2t"
Sep 19 01:13:05.410414 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:05.410397    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/49b29fdc-bbdc-44ff-901f-384537c101e5-image-registry-private-configuration\") pod \"image-registry-587d7ff8fc-hhv2t\" (UID: \"49b29fdc-bbdc-44ff-901f-384537c101e5\") " pod="openshift-image-registry/image-registry-587d7ff8fc-hhv2t"
Sep 19 01:13:05.410502 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:05.410450    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/49b29fdc-bbdc-44ff-901f-384537c101e5-installation-pull-secrets\") pod \"image-registry-587d7ff8fc-hhv2t\" (UID: \"49b29fdc-bbdc-44ff-901f-384537c101e5\") " pod="openshift-image-registry/image-registry-587d7ff8fc-hhv2t"
Sep 19 01:13:05.427191 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:05.427173    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtqsl\" (UniqueName: \"kubernetes.io/projected/49b29fdc-bbdc-44ff-901f-384537c101e5-kube-api-access-xtqsl\") pod \"image-registry-587d7ff8fc-hhv2t\" (UID: \"49b29fdc-bbdc-44ff-901f-384537c101e5\") " pod="openshift-image-registry/image-registry-587d7ff8fc-hhv2t"
Sep 19 01:13:05.429217 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:05.429187    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pk2s6\" (UniqueName: \"kubernetes.io/projected/a4338a4a-7257-495a-8e73-952e81106c45-kube-api-access-pk2s6\") pod \"dns-default-v6mxp\" (UID: \"a4338a4a-7257-495a-8e73-952e81106c45\") " pod="openshift-dns/dns-default-v6mxp"
Sep 19 01:13:05.430040 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:05.430020    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/49b29fdc-bbdc-44ff-901f-384537c101e5-bound-sa-token\") pod \"image-registry-587d7ff8fc-hhv2t\" (UID: \"49b29fdc-bbdc-44ff-901f-384537c101e5\") " pod="openshift-image-registry/image-registry-587d7ff8fc-hhv2t"
Sep 19 01:13:05.472119 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:05.472096    2581 generic.go:356] "Generic (PLEG): container finished" podID="d16ba40c-ab77-4204-b80e-d58f3a57e49a" containerID="03252005a8b67bea39d28b8ebe7047f08c2b841109f81843f01ab2a64fa95f38" exitCode=0
Sep 19 01:13:05.472221 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:05.472153    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-42mjj" event={"ID":"d16ba40c-ab77-4204-b80e-d58f3a57e49a","Type":"ContainerDied","Data":"03252005a8b67bea39d28b8ebe7047f08c2b841109f81843f01ab2a64fa95f38"}
Sep 19 01:13:05.507823 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:05.507793    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q78hm\" (UniqueName: \"kubernetes.io/projected/eef21c42-fb10-46ba-9ba9-fff960f461a6-kube-api-access-q78hm\") pod \"ingress-canary-9wqc6\" (UID: \"eef21c42-fb10-46ba-9ba9-fff960f461a6\") " pod="openshift-ingress-canary/ingress-canary-9wqc6"
Sep 19 01:13:05.507823 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:05.507828    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eef21c42-fb10-46ba-9ba9-fff960f461a6-cert\") pod \"ingress-canary-9wqc6\" (UID: \"eef21c42-fb10-46ba-9ba9-fff960f461a6\") " pod="openshift-ingress-canary/ingress-canary-9wqc6"
Sep 19 01:13:05.510300 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:05.510274    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eef21c42-fb10-46ba-9ba9-fff960f461a6-cert\") pod \"ingress-canary-9wqc6\" (UID: \"eef21c42-fb10-46ba-9ba9-fff960f461a6\") " pod="openshift-ingress-canary/ingress-canary-9wqc6"
Sep 19 01:13:05.530119 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:05.530099    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q78hm\" (UniqueName: \"kubernetes.io/projected/eef21c42-fb10-46ba-9ba9-fff960f461a6-kube-api-access-q78hm\") pod \"ingress-canary-9wqc6\" (UID: \"eef21c42-fb10-46ba-9ba9-fff960f461a6\") " pod="openshift-ingress-canary/ingress-canary-9wqc6"
Sep 19 01:13:05.540087 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:05.540061    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9wqc6"
Sep 19 01:13:05.872536 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:05.872503    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9wqc6"]
Sep 19 01:13:05.875464 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:13:05.875438    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeef21c42_fb10_46ba_9ba9_fff960f461a6.slice/crio-1de585634030e50c40cf31938b059f1f800c85173bef69f79c1ff550e9ab0a4c WatchSource:0}: Error finding container 1de585634030e50c40cf31938b059f1f800c85173bef69f79c1ff550e9ab0a4c: Status 404 returned error can't find the container with id 1de585634030e50c40cf31938b059f1f800c85173bef69f79c1ff550e9ab0a4c
Sep 19 01:13:05.910565 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:05.910504    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a4338a4a-7257-495a-8e73-952e81106c45-metrics-tls\") pod \"dns-default-v6mxp\" (UID: \"a4338a4a-7257-495a-8e73-952e81106c45\") " pod="openshift-dns/dns-default-v6mxp"
Sep 19 01:13:05.911119 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:05.910566    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/49b29fdc-bbdc-44ff-901f-384537c101e5-registry-tls\") pod \"image-registry-587d7ff8fc-hhv2t\" (UID: \"49b29fdc-bbdc-44ff-901f-384537c101e5\") " pod="openshift-image-registry/image-registry-587d7ff8fc-hhv2t"
Sep 19 01:13:05.911119 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:13:05.910656    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 19 01:13:05.911119 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:13:05.910659    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 19 01:13:05.911119 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:13:05.910680    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-587d7ff8fc-hhv2t: secret "image-registry-tls" not found
Sep 19 01:13:05.911119 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:13:05.910722    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/49b29fdc-bbdc-44ff-901f-384537c101e5-registry-tls podName:49b29fdc-bbdc-44ff-901f-384537c101e5 nodeName:}" failed. No retries permitted until 2026-09-19 01:13:06.91070301 +0000 UTC m=+37.798372087 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/49b29fdc-bbdc-44ff-901f-384537c101e5-registry-tls") pod "image-registry-587d7ff8fc-hhv2t" (UID: "49b29fdc-bbdc-44ff-901f-384537c101e5") : secret "image-registry-tls" not found
Sep 19 01:13:05.911119 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:13:05.910743    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4338a4a-7257-495a-8e73-952e81106c45-metrics-tls podName:a4338a4a-7257-495a-8e73-952e81106c45 nodeName:}" failed. No retries permitted until 2026-09-19 01:13:06.910735045 +0000 UTC m=+37.798404102 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a4338a4a-7257-495a-8e73-952e81106c45-metrics-tls") pod "dns-default-v6mxp" (UID: "a4338a4a-7257-495a-8e73-952e81106c45") : secret "dns-default-metrics-tls" not found
Sep 19 01:13:06.477323 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:06.477290    2581 generic.go:356] "Generic (PLEG): container finished" podID="d16ba40c-ab77-4204-b80e-d58f3a57e49a" containerID="68e9eb0e5f12056c658cb100ab46b81705937118ef6ed84c0e21d91028994e5b" exitCode=0
Sep 19 01:13:06.477482 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:06.477374    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-42mjj" event={"ID":"d16ba40c-ab77-4204-b80e-d58f3a57e49a","Type":"ContainerDied","Data":"68e9eb0e5f12056c658cb100ab46b81705937118ef6ed84c0e21d91028994e5b"}
Sep 19 01:13:06.478617 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:06.478594    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9wqc6" event={"ID":"eef21c42-fb10-46ba-9ba9-fff960f461a6","Type":"ContainerStarted","Data":"1de585634030e50c40cf31938b059f1f800c85173bef69f79c1ff550e9ab0a4c"}
Sep 19 01:13:06.919811 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:06.919604    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a4338a4a-7257-495a-8e73-952e81106c45-metrics-tls\") pod \"dns-default-v6mxp\" (UID: \"a4338a4a-7257-495a-8e73-952e81106c45\") " pod="openshift-dns/dns-default-v6mxp"
Sep 19 01:13:06.920292 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:13:06.919785    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 19 01:13:06.920292 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:06.919903    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/49b29fdc-bbdc-44ff-901f-384537c101e5-registry-tls\") pod \"image-registry-587d7ff8fc-hhv2t\" (UID: \"49b29fdc-bbdc-44ff-901f-384537c101e5\") " pod="openshift-image-registry/image-registry-587d7ff8fc-hhv2t"
Sep 19 01:13:06.920292 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:13:06.919957    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4338a4a-7257-495a-8e73-952e81106c45-metrics-tls podName:a4338a4a-7257-495a-8e73-952e81106c45 nodeName:}" failed. No retries permitted until 2026-09-19 01:13:08.919935123 +0000 UTC m=+39.807604197 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a4338a4a-7257-495a-8e73-952e81106c45-metrics-tls") pod "dns-default-v6mxp" (UID: "a4338a4a-7257-495a-8e73-952e81106c45") : secret "dns-default-metrics-tls" not found
Sep 19 01:13:06.920292 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:13:06.920029    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 19 01:13:06.920292 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:13:06.920043    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-587d7ff8fc-hhv2t: secret "image-registry-tls" not found
Sep 19 01:13:06.920292 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:13:06.920097    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/49b29fdc-bbdc-44ff-901f-384537c101e5-registry-tls podName:49b29fdc-bbdc-44ff-901f-384537c101e5 nodeName:}" failed. No retries permitted until 2026-09-19 01:13:08.920081763 +0000 UTC m=+39.807750818 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/49b29fdc-bbdc-44ff-901f-384537c101e5-registry-tls") pod "image-registry-587d7ff8fc-hhv2t" (UID: "49b29fdc-bbdc-44ff-901f-384537c101e5") : secret "image-registry-tls" not found
Sep 19 01:13:07.485686 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:07.485649    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-42mjj" event={"ID":"d16ba40c-ab77-4204-b80e-d58f3a57e49a","Type":"ContainerStarted","Data":"a3b78e49104b2467a2ea356a539aa878b44e54111f5713cd88536696f0928ba1"}
Sep 19 01:13:07.510854 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:07.510814    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-42mjj" podStartSLOduration=8.731843509 podStartE2EDuration="36.510798591s" totalImagesPullingTime="27.778955082s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 01:12:31 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 01:13:07.509725491 +0000 UTC m=+38.397394602" watchObservedRunningTime="2026-09-19 01:13:07.510798591 +0000 UTC m=+38.398467669"
Sep 19 01:13:08.936163 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:08.936125    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/49b29fdc-bbdc-44ff-901f-384537c101e5-registry-tls\") pod \"image-registry-587d7ff8fc-hhv2t\" (UID: \"49b29fdc-bbdc-44ff-901f-384537c101e5\") " pod="openshift-image-registry/image-registry-587d7ff8fc-hhv2t"
Sep 19 01:13:08.936548 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:08.936193    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a4338a4a-7257-495a-8e73-952e81106c45-metrics-tls\") pod \"dns-default-v6mxp\" (UID: \"a4338a4a-7257-495a-8e73-952e81106c45\") " pod="openshift-dns/dns-default-v6mxp"
Sep 19 01:13:08.936548 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:13:08.936280    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 19 01:13:08.936548 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:13:08.936297    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-587d7ff8fc-hhv2t: secret "image-registry-tls" not found
Sep 19 01:13:08.936548 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:13:08.936300    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 19 01:13:08.936548 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:13:08.936360    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4338a4a-7257-495a-8e73-952e81106c45-metrics-tls podName:a4338a4a-7257-495a-8e73-952e81106c45 nodeName:}" failed. No retries permitted until 2026-09-19 01:13:12.936345734 +0000 UTC m=+43.824014801 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a4338a4a-7257-495a-8e73-952e81106c45-metrics-tls") pod "dns-default-v6mxp" (UID: "a4338a4a-7257-495a-8e73-952e81106c45") : secret "dns-default-metrics-tls" not found
Sep 19 01:13:08.936548 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:13:08.936376    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/49b29fdc-bbdc-44ff-901f-384537c101e5-registry-tls podName:49b29fdc-bbdc-44ff-901f-384537c101e5 nodeName:}" failed. No retries permitted until 2026-09-19 01:13:12.936369661 +0000 UTC m=+43.824038743 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/49b29fdc-bbdc-44ff-901f-384537c101e5-registry-tls") pod "image-registry-587d7ff8fc-hhv2t" (UID: "49b29fdc-bbdc-44ff-901f-384537c101e5") : secret "image-registry-tls" not found
Sep 19 01:13:09.492329 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:09.492303    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9wqc6" event={"ID":"eef21c42-fb10-46ba-9ba9-fff960f461a6","Type":"ContainerStarted","Data":"3a7fcfdd610ea28b93b7d972e85e18b8eced76ab0526586aec5a19f5fa0d7dff"}
Sep 19 01:13:09.515794 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:09.515752    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-9wqc6" podStartSLOduration=1.8762808610000001 podStartE2EDuration="4.515740006s" totalImagesPullingTime="2.639459145s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 01:13:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 01:13:09.515598134 +0000 UTC m=+40.403267212" watchObservedRunningTime="2026-09-19 01:13:09.515740006 +0000 UTC m=+40.403409082"
Sep 19 01:13:11.920541 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:11.920508    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-7nbtf"]
Sep 19 01:13:11.922455 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:11.922439    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7nbtf"
Sep 19 01:13:11.924764 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:11.924742    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 19 01:13:11.925795 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:11.925766    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 19 01:13:11.926202 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:11.926159    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 19 01:13:11.926796 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:11.926767    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 19 01:13:11.929612 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:11.929592    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-pqfkn\""
Sep 19 01:13:11.948604 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:11.948481    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7nbtf"]
Sep 19 01:13:12.061489 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:12.061464    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/da1df6c8-9dc8-44c2-9093-66d5f87ffd35-crio-socket\") pod \"insights-runtime-extractor-7nbtf\" (UID: \"da1df6c8-9dc8-44c2-9093-66d5f87ffd35\") " pod="openshift-insights/insights-runtime-extractor-7nbtf"
Sep 19 01:13:12.061600 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:12.061510    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhmfp\" (UniqueName: \"kubernetes.io/projected/da1df6c8-9dc8-44c2-9093-66d5f87ffd35-kube-api-access-dhmfp\") pod \"insights-runtime-extractor-7nbtf\" (UID: \"da1df6c8-9dc8-44c2-9093-66d5f87ffd35\") " pod="openshift-insights/insights-runtime-extractor-7nbtf"
Sep 19 01:13:12.061600 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:12.061527    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/da1df6c8-9dc8-44c2-9093-66d5f87ffd35-data-volume\") pod \"insights-runtime-extractor-7nbtf\" (UID: \"da1df6c8-9dc8-44c2-9093-66d5f87ffd35\") " pod="openshift-insights/insights-runtime-extractor-7nbtf"
Sep 19 01:13:12.061600 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:12.061579    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/da1df6c8-9dc8-44c2-9093-66d5f87ffd35-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7nbtf\" (UID: \"da1df6c8-9dc8-44c2-9093-66d5f87ffd35\") " pod="openshift-insights/insights-runtime-extractor-7nbtf"
Sep 19 01:13:12.061708 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:12.061602    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/da1df6c8-9dc8-44c2-9093-66d5f87ffd35-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7nbtf\" (UID: \"da1df6c8-9dc8-44c2-9093-66d5f87ffd35\") " pod="openshift-insights/insights-runtime-extractor-7nbtf"
Sep 19 01:13:12.162265 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:12.162237    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/da1df6c8-9dc8-44c2-9093-66d5f87ffd35-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7nbtf\" (UID: \"da1df6c8-9dc8-44c2-9093-66d5f87ffd35\") " pod="openshift-insights/insights-runtime-extractor-7nbtf"
Sep 19 01:13:12.162349 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:12.162284    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/da1df6c8-9dc8-44c2-9093-66d5f87ffd35-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7nbtf\" (UID: \"da1df6c8-9dc8-44c2-9093-66d5f87ffd35\") " pod="openshift-insights/insights-runtime-extractor-7nbtf"
Sep 19 01:13:12.162349 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:12.162327    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/da1df6c8-9dc8-44c2-9093-66d5f87ffd35-crio-socket\") pod \"insights-runtime-extractor-7nbtf\" (UID: \"da1df6c8-9dc8-44c2-9093-66d5f87ffd35\") " pod="openshift-insights/insights-runtime-extractor-7nbtf"
Sep 19 01:13:12.162349 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:13:12.162336    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 19 01:13:12.162487 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:13:12.162383    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/da1df6c8-9dc8-44c2-9093-66d5f87ffd35-kube-rbac-proxy-cm podName:da1df6c8-9dc8-44c2-9093-66d5f87ffd35 nodeName:}" failed. No retries permitted until 2026-09-19 01:13:12.662370013 +0000 UTC m=+43.550039068 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/da1df6c8-9dc8-44c2-9093-66d5f87ffd35-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7nbtf" (UID: "da1df6c8-9dc8-44c2-9093-66d5f87ffd35") : configmap "kube-rbac-proxy" not found
Sep 19 01:13:12.162487 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:12.162397    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhmfp\" (UniqueName: \"kubernetes.io/projected/da1df6c8-9dc8-44c2-9093-66d5f87ffd35-kube-api-access-dhmfp\") pod \"insights-runtime-extractor-7nbtf\" (UID: \"da1df6c8-9dc8-44c2-9093-66d5f87ffd35\") " pod="openshift-insights/insights-runtime-extractor-7nbtf"
Sep 19 01:13:12.162487 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:12.162417    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/da1df6c8-9dc8-44c2-9093-66d5f87ffd35-data-volume\") pod \"insights-runtime-extractor-7nbtf\" (UID: \"da1df6c8-9dc8-44c2-9093-66d5f87ffd35\") " pod="openshift-insights/insights-runtime-extractor-7nbtf"
Sep 19 01:13:12.162487 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:13:12.162445    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 19 01:13:12.162643 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:13:12.162511    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da1df6c8-9dc8-44c2-9093-66d5f87ffd35-insights-runtime-extractor-tls podName:da1df6c8-9dc8-44c2-9093-66d5f87ffd35 nodeName:}" failed. No retries permitted until 2026-09-19 01:13:12.66249428 +0000 UTC m=+43.550163354 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/da1df6c8-9dc8-44c2-9093-66d5f87ffd35-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7nbtf" (UID: "da1df6c8-9dc8-44c2-9093-66d5f87ffd35") : secret "insights-runtime-extractor-tls" not found
Sep 19 01:13:12.162643 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:12.162523    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/da1df6c8-9dc8-44c2-9093-66d5f87ffd35-crio-socket\") pod \"insights-runtime-extractor-7nbtf\" (UID: \"da1df6c8-9dc8-44c2-9093-66d5f87ffd35\") " pod="openshift-insights/insights-runtime-extractor-7nbtf"
Sep 19 01:13:12.162711 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:12.162677    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/da1df6c8-9dc8-44c2-9093-66d5f87ffd35-data-volume\") pod \"insights-runtime-extractor-7nbtf\" (UID: \"da1df6c8-9dc8-44c2-9093-66d5f87ffd35\") " pod="openshift-insights/insights-runtime-extractor-7nbtf"
Sep 19 01:13:12.174548 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:12.174504    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhmfp\" (UniqueName: \"kubernetes.io/projected/da1df6c8-9dc8-44c2-9093-66d5f87ffd35-kube-api-access-dhmfp\") pod \"insights-runtime-extractor-7nbtf\" (UID: \"da1df6c8-9dc8-44c2-9093-66d5f87ffd35\") " pod="openshift-insights/insights-runtime-extractor-7nbtf"
Sep 19 01:13:12.665834 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:12.665804    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/da1df6c8-9dc8-44c2-9093-66d5f87ffd35-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7nbtf\" (UID: \"da1df6c8-9dc8-44c2-9093-66d5f87ffd35\") " pod="openshift-insights/insights-runtime-extractor-7nbtf"
Sep 19 01:13:12.666109 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:12.665845    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/da1df6c8-9dc8-44c2-9093-66d5f87ffd35-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7nbtf\" (UID: \"da1df6c8-9dc8-44c2-9093-66d5f87ffd35\") " pod="openshift-insights/insights-runtime-extractor-7nbtf"
Sep 19 01:13:12.666109 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:13:12.665946    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 19 01:13:12.666109 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:13:12.665980    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 19 01:13:12.666109 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:13:12.666032    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/da1df6c8-9dc8-44c2-9093-66d5f87ffd35-kube-rbac-proxy-cm podName:da1df6c8-9dc8-44c2-9093-66d5f87ffd35 nodeName:}" failed. No retries permitted until 2026-09-19 01:13:13.666010269 +0000 UTC m=+44.553679342 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/da1df6c8-9dc8-44c2-9093-66d5f87ffd35-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7nbtf" (UID: "da1df6c8-9dc8-44c2-9093-66d5f87ffd35") : configmap "kube-rbac-proxy" not found
Sep 19 01:13:12.666109 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:13:12.666051    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da1df6c8-9dc8-44c2-9093-66d5f87ffd35-insights-runtime-extractor-tls podName:da1df6c8-9dc8-44c2-9093-66d5f87ffd35 nodeName:}" failed. No retries permitted until 2026-09-19 01:13:13.666042475 +0000 UTC m=+44.553711535 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/da1df6c8-9dc8-44c2-9093-66d5f87ffd35-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7nbtf" (UID: "da1df6c8-9dc8-44c2-9093-66d5f87ffd35") : secret "insights-runtime-extractor-tls" not found
Sep 19 01:13:12.967693 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:12.967624    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a4338a4a-7257-495a-8e73-952e81106c45-metrics-tls\") pod \"dns-default-v6mxp\" (UID: \"a4338a4a-7257-495a-8e73-952e81106c45\") " pod="openshift-dns/dns-default-v6mxp"
Sep 19 01:13:12.968043 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:12.967704    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/49b29fdc-bbdc-44ff-901f-384537c101e5-registry-tls\") pod \"image-registry-587d7ff8fc-hhv2t\" (UID: \"49b29fdc-bbdc-44ff-901f-384537c101e5\") " pod="openshift-image-registry/image-registry-587d7ff8fc-hhv2t"
Sep 19 01:13:12.968043 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:13:12.967769    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 19 01:13:12.968043 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:13:12.967798    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 19 01:13:12.968043 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:13:12.967808    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-587d7ff8fc-hhv2t: secret "image-registry-tls" not found
Sep 19 01:13:12.968043 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:13:12.967828    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4338a4a-7257-495a-8e73-952e81106c45-metrics-tls podName:a4338a4a-7257-495a-8e73-952e81106c45 nodeName:}" failed. No retries permitted until 2026-09-19 01:13:20.967813935 +0000 UTC m=+51.855482990 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a4338a4a-7257-495a-8e73-952e81106c45-metrics-tls") pod "dns-default-v6mxp" (UID: "a4338a4a-7257-495a-8e73-952e81106c45") : secret "dns-default-metrics-tls" not found
Sep 19 01:13:12.968043 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:13:12.967848    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/49b29fdc-bbdc-44ff-901f-384537c101e5-registry-tls podName:49b29fdc-bbdc-44ff-901f-384537c101e5 nodeName:}" failed. No retries permitted until 2026-09-19 01:13:20.967836974 +0000 UTC m=+51.855506030 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/49b29fdc-bbdc-44ff-901f-384537c101e5-registry-tls") pod "image-registry-587d7ff8fc-hhv2t" (UID: "49b29fdc-bbdc-44ff-901f-384537c101e5") : secret "image-registry-tls" not found
Sep 19 01:13:13.673281 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:13.673253    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/da1df6c8-9dc8-44c2-9093-66d5f87ffd35-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7nbtf\" (UID: \"da1df6c8-9dc8-44c2-9093-66d5f87ffd35\") " pod="openshift-insights/insights-runtime-extractor-7nbtf"
Sep 19 01:13:13.673418 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:13.673294    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/da1df6c8-9dc8-44c2-9093-66d5f87ffd35-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7nbtf\" (UID: \"da1df6c8-9dc8-44c2-9093-66d5f87ffd35\") " pod="openshift-insights/insights-runtime-extractor-7nbtf"
Sep 19 01:13:13.673418 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:13:13.673368    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 19 01:13:13.673418 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:13:13.673408    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 19 01:13:13.673590 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:13:13.673431    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/da1df6c8-9dc8-44c2-9093-66d5f87ffd35-kube-rbac-proxy-cm podName:da1df6c8-9dc8-44c2-9093-66d5f87ffd35 nodeName:}" failed. No retries permitted until 2026-09-19 01:13:15.673415289 +0000 UTC m=+46.561084360 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/da1df6c8-9dc8-44c2-9093-66d5f87ffd35-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7nbtf" (UID: "da1df6c8-9dc8-44c2-9093-66d5f87ffd35") : configmap "kube-rbac-proxy" not found
Sep 19 01:13:13.673590 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:13:13.673454    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da1df6c8-9dc8-44c2-9093-66d5f87ffd35-insights-runtime-extractor-tls podName:da1df6c8-9dc8-44c2-9093-66d5f87ffd35 nodeName:}" failed. No retries permitted until 2026-09-19 01:13:15.673444202 +0000 UTC m=+46.561113258 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/da1df6c8-9dc8-44c2-9093-66d5f87ffd35-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7nbtf" (UID: "da1df6c8-9dc8-44c2-9093-66d5f87ffd35") : secret "insights-runtime-extractor-tls" not found
Sep 19 01:13:14.868007 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:14.867974    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-nx2sn_be4767d8-9458-4c5a-bc2b-d033a86f7017/dns-node-resolver/0.log"
Sep 19 01:13:15.684837 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:15.684811    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/da1df6c8-9dc8-44c2-9093-66d5f87ffd35-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7nbtf\" (UID: \"da1df6c8-9dc8-44c2-9093-66d5f87ffd35\") " pod="openshift-insights/insights-runtime-extractor-7nbtf"
Sep 19 01:13:15.685021 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:15.684846    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/da1df6c8-9dc8-44c2-9093-66d5f87ffd35-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7nbtf\" (UID: \"da1df6c8-9dc8-44c2-9093-66d5f87ffd35\") " pod="openshift-insights/insights-runtime-extractor-7nbtf"
Sep 19 01:13:15.685021 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:13:15.684941    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 19 01:13:15.685021 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:13:15.684952    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 19 01:13:15.685021 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:13:15.685007    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da1df6c8-9dc8-44c2-9093-66d5f87ffd35-insights-runtime-extractor-tls podName:da1df6c8-9dc8-44c2-9093-66d5f87ffd35 nodeName:}" failed. No retries permitted until 2026-09-19 01:13:19.684995004 +0000 UTC m=+50.572664065 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/da1df6c8-9dc8-44c2-9093-66d5f87ffd35-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7nbtf" (UID: "da1df6c8-9dc8-44c2-9093-66d5f87ffd35") : secret "insights-runtime-extractor-tls" not found
Sep 19 01:13:15.685021 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:13:15.685020    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/da1df6c8-9dc8-44c2-9093-66d5f87ffd35-kube-rbac-proxy-cm podName:da1df6c8-9dc8-44c2-9093-66d5f87ffd35 nodeName:}" failed. No retries permitted until 2026-09-19 01:13:19.685013418 +0000 UTC m=+50.572682473 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/da1df6c8-9dc8-44c2-9093-66d5f87ffd35-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7nbtf" (UID: "da1df6c8-9dc8-44c2-9093-66d5f87ffd35") : configmap "kube-rbac-proxy" not found
Sep 19 01:13:15.864125 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:15.864100    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vrlkw_ff802e32-5c32-466d-b76f-252275c6fb82/node-ca/0.log"
Sep 19 01:13:16.459693 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:16.459671    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9wqc6_eef21c42-fb10-46ba-9ba9-fff960f461a6/serve-healthcheck-canary/0.log"
Sep 19 01:13:19.714087 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:19.714048    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/da1df6c8-9dc8-44c2-9093-66d5f87ffd35-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7nbtf\" (UID: \"da1df6c8-9dc8-44c2-9093-66d5f87ffd35\") " pod="openshift-insights/insights-runtime-extractor-7nbtf"
Sep 19 01:13:19.714087 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:19.714097    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/da1df6c8-9dc8-44c2-9093-66d5f87ffd35-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7nbtf\" (UID: \"da1df6c8-9dc8-44c2-9093-66d5f87ffd35\") " pod="openshift-insights/insights-runtime-extractor-7nbtf"
Sep 19 01:13:19.714509 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:13:19.714208    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 19 01:13:19.714509 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:13:19.714262    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da1df6c8-9dc8-44c2-9093-66d5f87ffd35-insights-runtime-extractor-tls podName:da1df6c8-9dc8-44c2-9093-66d5f87ffd35 nodeName:}" failed. No retries permitted until 2026-09-19 01:13:27.714248429 +0000 UTC m=+58.601917505 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/da1df6c8-9dc8-44c2-9093-66d5f87ffd35-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7nbtf" (UID: "da1df6c8-9dc8-44c2-9093-66d5f87ffd35") : secret "insights-runtime-extractor-tls" not found
Sep 19 01:13:19.714611 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:19.714566    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/da1df6c8-9dc8-44c2-9093-66d5f87ffd35-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7nbtf\" (UID: \"da1df6c8-9dc8-44c2-9093-66d5f87ffd35\") " pod="openshift-insights/insights-runtime-extractor-7nbtf"
Sep 19 01:13:21.023468 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:21.023436    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/49b29fdc-bbdc-44ff-901f-384537c101e5-registry-tls\") pod \"image-registry-587d7ff8fc-hhv2t\" (UID: \"49b29fdc-bbdc-44ff-901f-384537c101e5\") " pod="openshift-image-registry/image-registry-587d7ff8fc-hhv2t"
Sep 19 01:13:21.023957 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:21.023492    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a4338a4a-7257-495a-8e73-952e81106c45-metrics-tls\") pod \"dns-default-v6mxp\" (UID: \"a4338a4a-7257-495a-8e73-952e81106c45\") " pod="openshift-dns/dns-default-v6mxp"
Sep 19 01:13:21.023957 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:13:21.023594    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 19 01:13:21.023957 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:13:21.023594    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 19 01:13:21.023957 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:13:21.023610    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-587d7ff8fc-hhv2t: secret "image-registry-tls" not found
Sep 19 01:13:21.023957 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:13:21.023669    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/49b29fdc-bbdc-44ff-901f-384537c101e5-registry-tls podName:49b29fdc-bbdc-44ff-901f-384537c101e5 nodeName:}" failed. No retries permitted until 2026-09-19 01:13:37.023650869 +0000 UTC m=+67.911319941 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/49b29fdc-bbdc-44ff-901f-384537c101e5-registry-tls") pod "image-registry-587d7ff8fc-hhv2t" (UID: "49b29fdc-bbdc-44ff-901f-384537c101e5") : secret "image-registry-tls" not found
Sep 19 01:13:21.023957 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:13:21.023686    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4338a4a-7257-495a-8e73-952e81106c45-metrics-tls podName:a4338a4a-7257-495a-8e73-952e81106c45 nodeName:}" failed. No retries permitted until 2026-09-19 01:13:37.023677989 +0000 UTC m=+67.911347057 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a4338a4a-7257-495a-8e73-952e81106c45-metrics-tls") pod "dns-default-v6mxp" (UID: "a4338a4a-7257-495a-8e73-952e81106c45") : secret "dns-default-metrics-tls" not found
Sep 19 01:13:27.771442 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:27.771400    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/da1df6c8-9dc8-44c2-9093-66d5f87ffd35-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7nbtf\" (UID: \"da1df6c8-9dc8-44c2-9093-66d5f87ffd35\") " pod="openshift-insights/insights-runtime-extractor-7nbtf"
Sep 19 01:13:27.774065 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:27.774037    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/da1df6c8-9dc8-44c2-9093-66d5f87ffd35-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7nbtf\" (UID: \"da1df6c8-9dc8-44c2-9093-66d5f87ffd35\") " pod="openshift-insights/insights-runtime-extractor-7nbtf"
Sep 19 01:13:27.834399 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:27.834375    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7nbtf"
Sep 19 01:13:28.141793 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:28.141773    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7nbtf"]
Sep 19 01:13:28.144738 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:13:28.144714    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda1df6c8_9dc8_44c2_9093_66d5f87ffd35.slice/crio-4b241a8cea79b9fcfa16121a24ebdf7fcb13112bed55f36046851202fe670c8a WatchSource:0}: Error finding container 4b241a8cea79b9fcfa16121a24ebdf7fcb13112bed55f36046851202fe670c8a: Status 404 returned error can't find the container with id 4b241a8cea79b9fcfa16121a24ebdf7fcb13112bed55f36046851202fe670c8a
Sep 19 01:13:28.535483 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:28.535408    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7nbtf" event={"ID":"da1df6c8-9dc8-44c2-9093-66d5f87ffd35","Type":"ContainerStarted","Data":"c9128bf6ce3ed5592653ed8e7d10eb6243b53b751b8a16d94c63df5b28ec9dc6"}
Sep 19 01:13:28.535483 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:28.535465    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7nbtf" event={"ID":"da1df6c8-9dc8-44c2-9093-66d5f87ffd35","Type":"ContainerStarted","Data":"4b241a8cea79b9fcfa16121a24ebdf7fcb13112bed55f36046851202fe670c8a"}
Sep 19 01:13:29.540102 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:29.540060    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7nbtf" event={"ID":"da1df6c8-9dc8-44c2-9093-66d5f87ffd35","Type":"ContainerStarted","Data":"3ae0303bb8f4d1a3021206da32f4ebbb3e0c421500b95316336c1b53747ae035"}
Sep 19 01:13:30.465342 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:30.465310    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x785d"
Sep 19 01:13:31.546332 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:31.546298    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7nbtf" event={"ID":"da1df6c8-9dc8-44c2-9093-66d5f87ffd35","Type":"ContainerStarted","Data":"b2e2b7bdaff2fc887743ee42bb07100cde8147feb5727d5d6f1eb807795b6e89"}
Sep 19 01:13:36.931939 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:36.931907    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3884e026-09ad-4bc8-bcb8-ae622d873aea-metrics-certs\") pod \"network-metrics-daemon-dkpb2\" (UID: \"3884e026-09ad-4bc8-bcb8-ae622d873aea\") " pod="openshift-multus/network-metrics-daemon-dkpb2"
Sep 19 01:13:36.932396 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:36.931969    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2ff7ef56-5c24-40a5-84d6-7259cf83a29c-original-pull-secret\") pod \"global-pull-secret-syncer-hrgqz\" (UID: \"2ff7ef56-5c24-40a5-84d6-7259cf83a29c\") " pod="kube-system/global-pull-secret-syncer-hrgqz"
Sep 19 01:13:36.934296 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:36.934280    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 19 01:13:36.934347 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:36.934280    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 19 01:13:36.944504 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:36.944477    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3884e026-09ad-4bc8-bcb8-ae622d873aea-metrics-certs\") pod \"network-metrics-daemon-dkpb2\" (UID: \"3884e026-09ad-4bc8-bcb8-ae622d873aea\") " pod="openshift-multus/network-metrics-daemon-dkpb2"
Sep 19 01:13:36.944606 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:36.944521    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2ff7ef56-5c24-40a5-84d6-7259cf83a29c-original-pull-secret\") pod \"global-pull-secret-syncer-hrgqz\" (UID: \"2ff7ef56-5c24-40a5-84d6-7259cf83a29c\") " pod="kube-system/global-pull-secret-syncer-hrgqz"
Sep 19 01:13:37.032978 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:37.032950    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a4338a4a-7257-495a-8e73-952e81106c45-metrics-tls\") pod \"dns-default-v6mxp\" (UID: \"a4338a4a-7257-495a-8e73-952e81106c45\") " pod="openshift-dns/dns-default-v6mxp"
Sep 19 01:13:37.033069 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:37.033008    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/49b29fdc-bbdc-44ff-901f-384537c101e5-registry-tls\") pod \"image-registry-587d7ff8fc-hhv2t\" (UID: \"49b29fdc-bbdc-44ff-901f-384537c101e5\") " pod="openshift-image-registry/image-registry-587d7ff8fc-hhv2t"
Sep 19 01:13:37.035064 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:37.035037    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a4338a4a-7257-495a-8e73-952e81106c45-metrics-tls\") pod \"dns-default-v6mxp\" (UID: \"a4338a4a-7257-495a-8e73-952e81106c45\") " pod="openshift-dns/dns-default-v6mxp"
Sep 19 01:13:37.035064 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:37.035041    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/49b29fdc-bbdc-44ff-901f-384537c101e5-registry-tls\") pod \"image-registry-587d7ff8fc-hhv2t\" (UID: \"49b29fdc-bbdc-44ff-901f-384537c101e5\") " pod="openshift-image-registry/image-registry-587d7ff8fc-hhv2t"
Sep 19 01:13:37.076006 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:37.075981    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrgqz"
Sep 19 01:13:37.086904 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:37.086689    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lthdr\""
Sep 19 01:13:37.092833 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:37.092817    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dkpb2"
Sep 19 01:13:37.133669 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:37.133629    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8c9j\" (UniqueName: \"kubernetes.io/projected/b0be445f-6160-4532-8a71-55bfdf782241-kube-api-access-g8c9j\") pod \"network-check-target-lmcnn\" (UID: \"b0be445f-6160-4532-8a71-55bfdf782241\") " pod="openshift-network-diagnostics/network-check-target-lmcnn"
Sep 19 01:13:37.138374 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:37.138351    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 19 01:13:37.148442 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:37.148309    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 19 01:13:37.156958 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:37.156914    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8c9j\" (UniqueName: \"kubernetes.io/projected/b0be445f-6160-4532-8a71-55bfdf782241-kube-api-access-g8c9j\") pod \"network-check-target-lmcnn\" (UID: \"b0be445f-6160-4532-8a71-55bfdf782241\") " pod="openshift-network-diagnostics/network-check-target-lmcnn"
Sep 19 01:13:37.310050 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:37.309983    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-h9l7c\""
Sep 19 01:13:37.317827 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:37.317809    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-v6mxp"
Sep 19 01:13:37.325585 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:37.325570    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-5r66s\""
Sep 19 01:13:37.332721 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:37.332702    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-587d7ff8fc-hhv2t"
Sep 19 01:13:37.362045 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:37.361966    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-7nbtf" podStartSLOduration=24.14930184 podStartE2EDuration="26.361944274s" totalImagesPullingTime="2.212642434s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 01:13:11 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 01:13:31.579551131 +0000 UTC m=+62.467220233" watchObservedRunningTime="2026-09-19 01:13:37.361944274 +0000 UTC m=+68.249613350"
Sep 19 01:13:37.363826 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:37.363129    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hrgqz"]
Sep 19 01:13:37.370245 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:13:37.367417    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ff7ef56_5c24_40a5_84d6_7259cf83a29c.slice/crio-859cbd95cbc8b0d9591a4343a4e96cfb6c74b05bde58128e001294a3ecbf613d WatchSource:0}: Error finding container 859cbd95cbc8b0d9591a4343a4e96cfb6c74b05bde58128e001294a3ecbf613d: Status 404 returned error can't find the container with id 859cbd95cbc8b0d9591a4343a4e96cfb6c74b05bde58128e001294a3ecbf613d
Sep 19 01:13:37.382544 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:37.382517    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dkpb2"]
Sep 19 01:13:37.386441 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:13:37.385956    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3884e026_09ad_4bc8_bcb8_ae622d873aea.slice/crio-2d65d01cae8c3264cf61ad9be2dc1b86f55005bfe18d53ce7e5c4a8cf56d89b1 WatchSource:0}: Error finding container 2d65d01cae8c3264cf61ad9be2dc1b86f55005bfe18d53ce7e5c4a8cf56d89b1: Status 404 returned error can't find the container with id 2d65d01cae8c3264cf61ad9be2dc1b86f55005bfe18d53ce7e5c4a8cf56d89b1
Sep 19 01:13:37.388755 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:37.388654    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-f5jnh\""
Sep 19 01:13:37.396207 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:37.396140    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lmcnn"
Sep 19 01:13:37.560658 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:37.560579    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dkpb2" event={"ID":"3884e026-09ad-4bc8-bcb8-ae622d873aea","Type":"ContainerStarted","Data":"2d65d01cae8c3264cf61ad9be2dc1b86f55005bfe18d53ce7e5c4a8cf56d89b1"}
Sep 19 01:13:37.561678 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:37.561656    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hrgqz" event={"ID":"2ff7ef56-5c24-40a5-84d6-7259cf83a29c","Type":"ContainerStarted","Data":"859cbd95cbc8b0d9591a4343a4e96cfb6c74b05bde58128e001294a3ecbf613d"}
Sep 19 01:13:37.587885 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:37.587838    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-v6mxp"]
Sep 19 01:13:37.590593 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:13:37.590567    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4338a4a_7257_495a_8e73_952e81106c45.slice/crio-71f5b7b751feb39a4e259fd14ca0935df211b72f74ced3af6a710c58f792299e WatchSource:0}: Error finding container 71f5b7b751feb39a4e259fd14ca0935df211b72f74ced3af6a710c58f792299e: Status 404 returned error can't find the container with id 71f5b7b751feb39a4e259fd14ca0935df211b72f74ced3af6a710c58f792299e
Sep 19 01:13:37.613484 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:37.613461    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-587d7ff8fc-hhv2t"]
Sep 19 01:13:37.615819 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:13:37.615798    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49b29fdc_bbdc_44ff_901f_384537c101e5.slice/crio-936d6b0b5d2a86173ade67dd9d216b4d5f25033f92b4887108fce68fd3036174 WatchSource:0}: Error finding container 936d6b0b5d2a86173ade67dd9d216b4d5f25033f92b4887108fce68fd3036174: Status 404 returned error can't find the container with id 936d6b0b5d2a86173ade67dd9d216b4d5f25033f92b4887108fce68fd3036174
Sep 19 01:13:37.662552 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:37.662532    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lmcnn"]
Sep 19 01:13:37.665389 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:13:37.665360    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0be445f_6160_4532_8a71_55bfdf782241.slice/crio-24def1ad6434eac7865721869dea52078f171522ce7a41c678ad3ab5722a5fbc WatchSource:0}: Error finding container 24def1ad6434eac7865721869dea52078f171522ce7a41c678ad3ab5722a5fbc: Status 404 returned error can't find the container with id 24def1ad6434eac7865721869dea52078f171522ce7a41c678ad3ab5722a5fbc
Sep 19 01:13:38.571306 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:38.571196    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-587d7ff8fc-hhv2t" event={"ID":"49b29fdc-bbdc-44ff-901f-384537c101e5","Type":"ContainerStarted","Data":"50cfeef2723b4eca36ee1689fb5e81b6823718f242eaf47e3885a41a29e608e5"}
Sep 19 01:13:38.571306 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:38.571256    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-587d7ff8fc-hhv2t" event={"ID":"49b29fdc-bbdc-44ff-901f-384537c101e5","Type":"ContainerStarted","Data":"936d6b0b5d2a86173ade67dd9d216b4d5f25033f92b4887108fce68fd3036174"}
Sep 19 01:13:38.571852 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:38.571832    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-587d7ff8fc-hhv2t"
Sep 19 01:13:38.573907 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:38.573856    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-v6mxp" event={"ID":"a4338a4a-7257-495a-8e73-952e81106c45","Type":"ContainerStarted","Data":"71f5b7b751feb39a4e259fd14ca0935df211b72f74ced3af6a710c58f792299e"}
Sep 19 01:13:38.576603 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:38.576556    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lmcnn" event={"ID":"b0be445f-6160-4532-8a71-55bfdf782241","Type":"ContainerStarted","Data":"24def1ad6434eac7865721869dea52078f171522ce7a41c678ad3ab5722a5fbc"}
Sep 19 01:13:38.599175 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:38.598275    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-587d7ff8fc-hhv2t" podStartSLOduration=43.598257911 podStartE2EDuration="43.598257911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 01:12:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 01:13:38.597418653 +0000 UTC m=+69.485087733" watchObservedRunningTime="2026-09-19 01:13:38.598257911 +0000 UTC m=+69.485926991"
Sep 19 01:13:39.583502 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:39.583450    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dkpb2" event={"ID":"3884e026-09ad-4bc8-bcb8-ae622d873aea","Type":"ContainerStarted","Data":"5259fd049e4014008f4dbc742ea004fdb9de89555036129a645580c0586f6e78"}
Sep 19 01:13:39.583946 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:39.583524    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dkpb2" event={"ID":"3884e026-09ad-4bc8-bcb8-ae622d873aea","Type":"ContainerStarted","Data":"afc987d4d41901c619aebdd2b902c1f510136f6206c4a513634d8d9cc721b0cf"}
Sep 19 01:13:39.623794 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:39.623731    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-dkpb2" podStartSLOduration=67.487079089 podStartE2EDuration="1m8.623708278s" totalImagesPullingTime="1.136629189s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 01:12:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 01:13:39.623302864 +0000 UTC m=+70.510971967" watchObservedRunningTime="2026-09-19 01:13:39.623708278 +0000 UTC m=+70.511377356"
Sep 19 01:13:40.593259 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:40.593211    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-v6mxp" event={"ID":"a4338a4a-7257-495a-8e73-952e81106c45","Type":"ContainerStarted","Data":"95f43835972825ad72a65702887f6b1162cf14df5f4f2b7af30d109852d79d04"}
Sep 19 01:13:40.593686 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:40.593308    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-v6mxp" event={"ID":"a4338a4a-7257-495a-8e73-952e81106c45","Type":"ContainerStarted","Data":"7da1218715283ea624695ed4ff118eb3c8478555ecf64005257ebd6d0366f415"}
Sep 19 01:13:40.619369 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:40.619228    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-v6mxp" podStartSLOduration=33.479834056 podStartE2EDuration="35.619205711s" totalImagesPullingTime="2.139371655s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 01:13:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 01:13:40.617892231 +0000 UTC m=+71.505561328" watchObservedRunningTime="2026-09-19 01:13:40.619205711 +0000 UTC m=+71.506874791"
Sep 19 01:13:41.056762 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:41.056687    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-769fbb98bc-jzj5g"]
Sep 19 01:13:41.059014 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:41.058995    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-769fbb98bc-jzj5g"
Sep 19 01:13:41.061806 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:41.061775    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 19 01:13:41.062457 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:41.062436    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 19 01:13:41.062572 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:41.062490    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 19 01:13:41.062572 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:41.062520    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 19 01:13:41.071215 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:41.071193    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bdf66c7c6-zjwvm"]
Sep 19 01:13:41.073044 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:41.073030    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bdf66c7c6-zjwvm"
Sep 19 01:13:41.079449 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:41.079228    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 19 01:13:41.079449 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:41.079268    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 19 01:13:41.079449 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:41.079306    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 19 01:13:41.079449 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:41.079251    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 19 01:13:41.082174 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:41.082120    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-769fbb98bc-jzj5g"]
Sep 19 01:13:41.107468 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:41.107398    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bdf66c7c6-zjwvm"]
Sep 19 01:13:41.109569 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:41.109546    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-587d7ff8fc-hhv2t"]
Sep 19 01:13:41.165616 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:41.165587    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/abaf3f48-1acc-4ca1-9ba3-b720a0176105-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6bdf66c7c6-zjwvm\" (UID: \"abaf3f48-1acc-4ca1-9ba3-b720a0176105\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bdf66c7c6-zjwvm"
Sep 19 01:13:41.165752 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:41.165676    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szwpn\" (UniqueName: \"kubernetes.io/projected/870d381a-3621-4841-9847-4a4dc2a470c4-kube-api-access-szwpn\") pod \"klusterlet-addon-workmgr-769fbb98bc-jzj5g\" (UID: \"870d381a-3621-4841-9847-4a4dc2a470c4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-769fbb98bc-jzj5g"
Sep 19 01:13:41.165752 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:41.165709    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/abaf3f48-1acc-4ca1-9ba3-b720a0176105-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6bdf66c7c6-zjwvm\" (UID: \"abaf3f48-1acc-4ca1-9ba3-b720a0176105\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bdf66c7c6-zjwvm"
Sep 19 01:13:41.165752 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:41.165747    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/870d381a-3621-4841-9847-4a4dc2a470c4-klusterlet-config\") pod \"klusterlet-addon-workmgr-769fbb98bc-jzj5g\" (UID: \"870d381a-3621-4841-9847-4a4dc2a470c4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-769fbb98bc-jzj5g"
Sep 19 01:13:41.165946 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:41.165770    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/abaf3f48-1acc-4ca1-9ba3-b720a0176105-ca\") pod \"cluster-proxy-proxy-agent-6bdf66c7c6-zjwvm\" (UID: \"abaf3f48-1acc-4ca1-9ba3-b720a0176105\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bdf66c7c6-zjwvm"
Sep 19 01:13:41.165946 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:41.165796    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/abaf3f48-1acc-4ca1-9ba3-b720a0176105-hub\") pod \"cluster-proxy-proxy-agent-6bdf66c7c6-zjwvm\" (UID: \"abaf3f48-1acc-4ca1-9ba3-b720a0176105\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bdf66c7c6-zjwvm"
Sep 19 01:13:41.165946 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:41.165819    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsfmg\" (UniqueName: \"kubernetes.io/projected/abaf3f48-1acc-4ca1-9ba3-b720a0176105-kube-api-access-nsfmg\") pod \"cluster-proxy-proxy-agent-6bdf66c7c6-zjwvm\" (UID: \"abaf3f48-1acc-4ca1-9ba3-b720a0176105\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bdf66c7c6-zjwvm"
Sep 19 01:13:41.165946 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:41.165897    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/870d381a-3621-4841-9847-4a4dc2a470c4-tmp\") pod \"klusterlet-addon-workmgr-769fbb98bc-jzj5g\" (UID: \"870d381a-3621-4841-9847-4a4dc2a470c4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-769fbb98bc-jzj5g"
Sep 19 01:13:41.165946 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:41.165931    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/abaf3f48-1acc-4ca1-9ba3-b720a0176105-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6bdf66c7c6-zjwvm\" (UID: \"abaf3f48-1acc-4ca1-9ba3-b720a0176105\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bdf66c7c6-zjwvm"
Sep 19 01:13:41.266648 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:41.266617    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szwpn\" (UniqueName: \"kubernetes.io/projected/870d381a-3621-4841-9847-4a4dc2a470c4-kube-api-access-szwpn\") pod \"klusterlet-addon-workmgr-769fbb98bc-jzj5g\" (UID: \"870d381a-3621-4841-9847-4a4dc2a470c4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-769fbb98bc-jzj5g"
Sep 19 01:13:41.266803 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:41.266664    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/abaf3f48-1acc-4ca1-9ba3-b720a0176105-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6bdf66c7c6-zjwvm\" (UID: \"abaf3f48-1acc-4ca1-9ba3-b720a0176105\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bdf66c7c6-zjwvm"
Sep 19 01:13:41.266803 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:41.266704    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/870d381a-3621-4841-9847-4a4dc2a470c4-klusterlet-config\") pod \"klusterlet-addon-workmgr-769fbb98bc-jzj5g\" (UID: \"870d381a-3621-4841-9847-4a4dc2a470c4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-769fbb98bc-jzj5g"
Sep 19 01:13:41.266803 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:41.266733    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/abaf3f48-1acc-4ca1-9ba3-b720a0176105-ca\") pod \"cluster-proxy-proxy-agent-6bdf66c7c6-zjwvm\" (UID: \"abaf3f48-1acc-4ca1-9ba3-b720a0176105\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bdf66c7c6-zjwvm"
Sep 19 01:13:41.266803 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:41.266757    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/abaf3f48-1acc-4ca1-9ba3-b720a0176105-hub\") pod \"cluster-proxy-proxy-agent-6bdf66c7c6-zjwvm\" (UID: \"abaf3f48-1acc-4ca1-9ba3-b720a0176105\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bdf66c7c6-zjwvm"
Sep 19 01:13:41.266803 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:41.266780    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsfmg\" (UniqueName: \"kubernetes.io/projected/abaf3f48-1acc-4ca1-9ba3-b720a0176105-kube-api-access-nsfmg\") pod \"cluster-proxy-proxy-agent-6bdf66c7c6-zjwvm\" (UID: \"abaf3f48-1acc-4ca1-9ba3-b720a0176105\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bdf66c7c6-zjwvm"
Sep 19 01:13:41.267074 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:41.266806    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/870d381a-3621-4841-9847-4a4dc2a470c4-tmp\") pod \"klusterlet-addon-workmgr-769fbb98bc-jzj5g\" (UID: \"870d381a-3621-4841-9847-4a4dc2a470c4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-769fbb98bc-jzj5g"
Sep 19 01:13:41.267074 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:41.266830    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/abaf3f48-1acc-4ca1-9ba3-b720a0176105-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6bdf66c7c6-zjwvm\" (UID: \"abaf3f48-1acc-4ca1-9ba3-b720a0176105\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bdf66c7c6-zjwvm"
Sep 19 01:13:41.267074 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:41.266880    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/abaf3f48-1acc-4ca1-9ba3-b720a0176105-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6bdf66c7c6-zjwvm\" (UID: \"abaf3f48-1acc-4ca1-9ba3-b720a0176105\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bdf66c7c6-zjwvm"
Sep 19 01:13:41.267605 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:41.267586    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/870d381a-3621-4841-9847-4a4dc2a470c4-tmp\") pod \"klusterlet-addon-workmgr-769fbb98bc-jzj5g\" (UID: \"870d381a-3621-4841-9847-4a4dc2a470c4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-769fbb98bc-jzj5g"
Sep 19 01:13:41.268118 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:41.268057    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/abaf3f48-1acc-4ca1-9ba3-b720a0176105-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6bdf66c7c6-zjwvm\" (UID: \"abaf3f48-1acc-4ca1-9ba3-b720a0176105\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bdf66c7c6-zjwvm"
Sep 19 01:13:41.270668 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:41.270606    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/870d381a-3621-4841-9847-4a4dc2a470c4-klusterlet-config\") pod \"klusterlet-addon-workmgr-769fbb98bc-jzj5g\" (UID: \"870d381a-3621-4841-9847-4a4dc2a470c4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-769fbb98bc-jzj5g"
Sep 19 01:13:41.270668 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:41.270606    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/abaf3f48-1acc-4ca1-9ba3-b720a0176105-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6bdf66c7c6-zjwvm\" (UID: \"abaf3f48-1acc-4ca1-9ba3-b720a0176105\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bdf66c7c6-zjwvm"
Sep 19 01:13:41.270827 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:41.270801    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/abaf3f48-1acc-4ca1-9ba3-b720a0176105-ca\") pod \"cluster-proxy-proxy-agent-6bdf66c7c6-zjwvm\" (UID: \"abaf3f48-1acc-4ca1-9ba3-b720a0176105\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bdf66c7c6-zjwvm"
Sep 19 01:13:41.271502 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:41.271473    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/abaf3f48-1acc-4ca1-9ba3-b720a0176105-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6bdf66c7c6-zjwvm\" (UID: \"abaf3f48-1acc-4ca1-9ba3-b720a0176105\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bdf66c7c6-zjwvm"
Sep 19 01:13:41.271904 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:41.271885    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/abaf3f48-1acc-4ca1-9ba3-b720a0176105-hub\") pod \"cluster-proxy-proxy-agent-6bdf66c7c6-zjwvm\" (UID: \"abaf3f48-1acc-4ca1-9ba3-b720a0176105\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bdf66c7c6-zjwvm"
Sep 19 01:13:41.287069 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:41.287045    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-szwpn\" (UniqueName: \"kubernetes.io/projected/870d381a-3621-4841-9847-4a4dc2a470c4-kube-api-access-szwpn\") pod \"klusterlet-addon-workmgr-769fbb98bc-jzj5g\" (UID: \"870d381a-3621-4841-9847-4a4dc2a470c4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-769fbb98bc-jzj5g"
Sep 19 01:13:41.288124 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:41.288099    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsfmg\" (UniqueName: \"kubernetes.io/projected/abaf3f48-1acc-4ca1-9ba3-b720a0176105-kube-api-access-nsfmg\") pod \"cluster-proxy-proxy-agent-6bdf66c7c6-zjwvm\" (UID: \"abaf3f48-1acc-4ca1-9ba3-b720a0176105\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bdf66c7c6-zjwvm"
Sep 19 01:13:41.368016 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:41.367986    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-769fbb98bc-jzj5g"
Sep 19 01:13:41.389526 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:41.389504    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bdf66c7c6-zjwvm"
Sep 19 01:13:41.597616 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:41.597579    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-v6mxp"
Sep 19 01:13:42.601431 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:42.601401    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hrgqz" event={"ID":"2ff7ef56-5c24-40a5-84d6-7259cf83a29c","Type":"ContainerStarted","Data":"4d1ec1aeb6ee596bc27d8b6a1103a7decf41de2d30ad4ba4d0e29c0346815207"}
Sep 19 01:13:42.603063 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:42.603036    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lmcnn" event={"ID":"b0be445f-6160-4532-8a71-55bfdf782241","Type":"ContainerStarted","Data":"dc4208a7cd4d7f4e7b3eb169fc8931ebd45c4ce82b28fc15b8513c8a79bad5b8"}
Sep 19 01:13:42.603167 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:42.603102    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-lmcnn"
Sep 19 01:13:42.630008 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:42.629950    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-hrgqz" podStartSLOduration=66.513479735 podStartE2EDuration="1m11.629930757s" totalImagesPullingTime="5.116451022s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 01:12:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 01:13:42.629122266 +0000 UTC m=+73.516791355" watchObservedRunningTime="2026-09-19 01:13:42.629930757 +0000 UTC m=+73.517599862"
Sep 19 01:13:42.679383 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:42.679333    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-lmcnn" podStartSLOduration=66.869949332 podStartE2EDuration="1m11.679314598s" totalImagesPullingTime="4.809365266s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 01:12:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 01:13:42.677175053 +0000 UTC m=+73.564844129" watchObservedRunningTime="2026-09-19 01:13:42.679314598 +0000 UTC m=+73.566983678"
Sep 19 01:13:42.788958 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:42.788806    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-769fbb98bc-jzj5g"]
Sep 19 01:13:42.791542 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:13:42.791497    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod870d381a_3621_4841_9847_4a4dc2a470c4.slice/crio-cb4f60b07c55d5858255754df468193a8b5eb503540c83efcb1c92f8c0e8aafc WatchSource:0}: Error finding container cb4f60b07c55d5858255754df468193a8b5eb503540c83efcb1c92f8c0e8aafc: Status 404 returned error can't find the container with id cb4f60b07c55d5858255754df468193a8b5eb503540c83efcb1c92f8c0e8aafc
Sep 19 01:13:42.793805 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:42.793777    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bdf66c7c6-zjwvm"]
Sep 19 01:13:42.796733 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:13:42.796712    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podabaf3f48_1acc_4ca1_9ba3_b720a0176105.slice/crio-e05a0674c432510e4255824d8c841f924fa0fc459d19b579fb8703617da0e82f WatchSource:0}: Error finding container e05a0674c432510e4255824d8c841f924fa0fc459d19b579fb8703617da0e82f: Status 404 returned error can't find the container with id e05a0674c432510e4255824d8c841f924fa0fc459d19b579fb8703617da0e82f
Sep 19 01:13:43.608278 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:43.608239    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-769fbb98bc-jzj5g" event={"ID":"870d381a-3621-4841-9847-4a4dc2a470c4","Type":"ContainerStarted","Data":"cb4f60b07c55d5858255754df468193a8b5eb503540c83efcb1c92f8c0e8aafc"}
Sep 19 01:13:43.609580 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:43.609549    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bdf66c7c6-zjwvm" event={"ID":"abaf3f48-1acc-4ca1-9ba3-b720a0176105","Type":"ContainerStarted","Data":"e05a0674c432510e4255824d8c841f924fa0fc459d19b579fb8703617da0e82f"}
Sep 19 01:13:48.633485 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:48.633449    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bdf66c7c6-zjwvm" event={"ID":"abaf3f48-1acc-4ca1-9ba3-b720a0176105","Type":"ContainerStarted","Data":"ebac555301eca0edd67ee445a2f4da8d9265e467397ebd2fe2fe40a2975c741a"}
Sep 19 01:13:48.635964 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:48.635431    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-769fbb98bc-jzj5g" event={"ID":"870d381a-3621-4841-9847-4a4dc2a470c4","Type":"ContainerStarted","Data":"985c1270e0d5c9b3ac5fc52d4aa9416df7041f69a426fbe9b6ec8fe8cd36be63"}
Sep 19 01:13:48.636087 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:48.636059    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-769fbb98bc-jzj5g"
Sep 19 01:13:48.637342 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:48.637297    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-769fbb98bc-jzj5g" podUID="870d381a-3621-4841-9847-4a4dc2a470c4" containerName="acm-agent" probeResult="failure" output="Get \"http://10.133.0.10:8000/readyz\": dial tcp 10.133.0.10:8000: connect: connection refused"
Sep 19 01:13:48.690611 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:48.690561    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-769fbb98bc-jzj5g" podStartSLOduration=1.9677330309999999 podStartE2EDuration="7.690541855s" totalImagesPullingTime="5.722808824s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 01:13:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 01:13:48.689041895 +0000 UTC m=+79.576710991" watchObservedRunningTime="2026-09-19 01:13:48.690541855 +0000 UTC m=+79.578210935"
Sep 19 01:13:49.640522 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:49.640481    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bdf66c7c6-zjwvm" event={"ID":"abaf3f48-1acc-4ca1-9ba3-b720a0176105","Type":"ContainerStarted","Data":"965af9bf99b46456aa2db9ad66eab7ae6cde08f686d7f1b0558fd5c90ae55dec"}
Sep 19 01:13:49.640906 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:49.640535    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bdf66c7c6-zjwvm" event={"ID":"abaf3f48-1acc-4ca1-9ba3-b720a0176105","Type":"ContainerStarted","Data":"a69a872386b5ff41cf5bf9513f28fb768e8c169e41e2d39231b12924e12599be"}
Sep 19 01:13:49.641553 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:49.641532    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-769fbb98bc-jzj5g"
Sep 19 01:13:49.669767 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:49.669709    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bdf66c7c6-zjwvm" podStartSLOduration=2.9494693119999997 podStartE2EDuration="8.669690567s" totalImagesPullingTime="5.720221255s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 01:13:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 01:13:49.667624478 +0000 UTC m=+80.555293555" watchObservedRunningTime="2026-09-19 01:13:49.669690567 +0000 UTC m=+80.557359647"
Sep 19 01:13:51.604881 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:51.604842    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-v6mxp"
Sep 19 01:13:56.789530 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:56.789495    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-cdprj"]
Sep 19 01:13:56.792237 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:56.792221    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-cdprj"
Sep 19 01:13:56.797132 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:56.797106    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 19 01:13:56.797448 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:56.797431    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 19 01:13:56.797585 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:56.797570    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 19 01:13:56.797645 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:56.797592    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 19 01:13:56.798172 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:56.798148    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 19 01:13:56.798172 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:56.798168    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 19 01:13:56.798354 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:56.798150    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-hn4x7\""
Sep 19 01:13:56.873988 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:56.873963    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8e1d7cd1-1fa0-422e-a512-fef514857934-sys\") pod \"node-exporter-cdprj\" (UID: \"8e1d7cd1-1fa0-422e-a512-fef514857934\") " pod="openshift-monitoring/node-exporter-cdprj"
Sep 19 01:13:56.873988 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:56.873989    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8e1d7cd1-1fa0-422e-a512-fef514857934-root\") pod \"node-exporter-cdprj\" (UID: \"8e1d7cd1-1fa0-422e-a512-fef514857934\") " pod="openshift-monitoring/node-exporter-cdprj"
Sep 19 01:13:56.874126 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:56.874011    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8e1d7cd1-1fa0-422e-a512-fef514857934-node-exporter-wtmp\") pod \"node-exporter-cdprj\" (UID: \"8e1d7cd1-1fa0-422e-a512-fef514857934\") " pod="openshift-monitoring/node-exporter-cdprj"
Sep 19 01:13:56.874126 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:56.874031    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8e1d7cd1-1fa0-422e-a512-fef514857934-node-exporter-tls\") pod \"node-exporter-cdprj\" (UID: \"8e1d7cd1-1fa0-422e-a512-fef514857934\") " pod="openshift-monitoring/node-exporter-cdprj"
Sep 19 01:13:56.874126 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:56.874092    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8e1d7cd1-1fa0-422e-a512-fef514857934-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cdprj\" (UID: \"8e1d7cd1-1fa0-422e-a512-fef514857934\") " pod="openshift-monitoring/node-exporter-cdprj"
Sep 19 01:13:56.874126 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:56.874120    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hdvk\" (UniqueName: \"kubernetes.io/projected/8e1d7cd1-1fa0-422e-a512-fef514857934-kube-api-access-2hdvk\") pod \"node-exporter-cdprj\" (UID: \"8e1d7cd1-1fa0-422e-a512-fef514857934\") " pod="openshift-monitoring/node-exporter-cdprj"
Sep 19 01:13:56.874277 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:56.874146    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8e1d7cd1-1fa0-422e-a512-fef514857934-node-exporter-textfile\") pod \"node-exporter-cdprj\" (UID: \"8e1d7cd1-1fa0-422e-a512-fef514857934\") " pod="openshift-monitoring/node-exporter-cdprj"
Sep 19 01:13:56.874277 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:56.874190    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8e1d7cd1-1fa0-422e-a512-fef514857934-metrics-client-ca\") pod \"node-exporter-cdprj\" (UID: \"8e1d7cd1-1fa0-422e-a512-fef514857934\") " pod="openshift-monitoring/node-exporter-cdprj"
Sep 19 01:13:56.874277 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:56.874211    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8e1d7cd1-1fa0-422e-a512-fef514857934-node-exporter-accelerators-collector-config\") pod \"node-exporter-cdprj\" (UID: \"8e1d7cd1-1fa0-422e-a512-fef514857934\") " pod="openshift-monitoring/node-exporter-cdprj"
Sep 19 01:13:56.975396 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:56.975375    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8e1d7cd1-1fa0-422e-a512-fef514857934-metrics-client-ca\") pod \"node-exporter-cdprj\" (UID: \"8e1d7cd1-1fa0-422e-a512-fef514857934\") " pod="openshift-monitoring/node-exporter-cdprj"
Sep 19 01:13:56.975521 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:56.975404    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8e1d7cd1-1fa0-422e-a512-fef514857934-node-exporter-accelerators-collector-config\") pod \"node-exporter-cdprj\" (UID: \"8e1d7cd1-1fa0-422e-a512-fef514857934\") " pod="openshift-monitoring/node-exporter-cdprj"
Sep 19 01:13:56.975521 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:56.975423    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8e1d7cd1-1fa0-422e-a512-fef514857934-sys\") pod \"node-exporter-cdprj\" (UID: \"8e1d7cd1-1fa0-422e-a512-fef514857934\") " pod="openshift-monitoring/node-exporter-cdprj"
Sep 19 01:13:56.975521 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:56.975441    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8e1d7cd1-1fa0-422e-a512-fef514857934-root\") pod \"node-exporter-cdprj\" (UID: \"8e1d7cd1-1fa0-422e-a512-fef514857934\") " pod="openshift-monitoring/node-exporter-cdprj"
Sep 19 01:13:56.975521 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:56.975462    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8e1d7cd1-1fa0-422e-a512-fef514857934-node-exporter-wtmp\") pod \"node-exporter-cdprj\" (UID: \"8e1d7cd1-1fa0-422e-a512-fef514857934\") " pod="openshift-monitoring/node-exporter-cdprj"
Sep 19 01:13:56.975521 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:56.975480    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8e1d7cd1-1fa0-422e-a512-fef514857934-node-exporter-tls\") pod \"node-exporter-cdprj\" (UID: \"8e1d7cd1-1fa0-422e-a512-fef514857934\") " pod="openshift-monitoring/node-exporter-cdprj"
Sep 19 01:13:56.975521 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:56.975504    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8e1d7cd1-1fa0-422e-a512-fef514857934-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cdprj\" (UID: \"8e1d7cd1-1fa0-422e-a512-fef514857934\") " pod="openshift-monitoring/node-exporter-cdprj"
Sep 19 01:13:56.975521 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:56.975507    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8e1d7cd1-1fa0-422e-a512-fef514857934-sys\") pod \"node-exporter-cdprj\" (UID: \"8e1d7cd1-1fa0-422e-a512-fef514857934\") " pod="openshift-monitoring/node-exporter-cdprj"
Sep 19 01:13:56.975853 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:56.975520    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8e1d7cd1-1fa0-422e-a512-fef514857934-root\") pod \"node-exporter-cdprj\" (UID: \"8e1d7cd1-1fa0-422e-a512-fef514857934\") " pod="openshift-monitoring/node-exporter-cdprj"
Sep 19 01:13:56.975853 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:56.975528    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hdvk\" (UniqueName: \"kubernetes.io/projected/8e1d7cd1-1fa0-422e-a512-fef514857934-kube-api-access-2hdvk\") pod \"node-exporter-cdprj\" (UID: \"8e1d7cd1-1fa0-422e-a512-fef514857934\") " pod="openshift-monitoring/node-exporter-cdprj"
Sep 19 01:13:56.975853 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:56.975577    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8e1d7cd1-1fa0-422e-a512-fef514857934-node-exporter-textfile\") pod \"node-exporter-cdprj\" (UID: \"8e1d7cd1-1fa0-422e-a512-fef514857934\") " pod="openshift-monitoring/node-exporter-cdprj"
Sep 19 01:13:56.975853 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:56.975607    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8e1d7cd1-1fa0-422e-a512-fef514857934-node-exporter-wtmp\") pod \"node-exporter-cdprj\" (UID: \"8e1d7cd1-1fa0-422e-a512-fef514857934\") " pod="openshift-monitoring/node-exporter-cdprj"
Sep 19 01:13:56.975853 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:13:56.975622    2581 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 19 01:13:56.975853 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:13:56.975690    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e1d7cd1-1fa0-422e-a512-fef514857934-node-exporter-tls podName:8e1d7cd1-1fa0-422e-a512-fef514857934 nodeName:}" failed. No retries permitted until 2026-09-19 01:13:57.475666601 +0000 UTC m=+88.363335671 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/8e1d7cd1-1fa0-422e-a512-fef514857934-node-exporter-tls") pod "node-exporter-cdprj" (UID: "8e1d7cd1-1fa0-422e-a512-fef514857934") : secret "node-exporter-tls" not found
Sep 19 01:13:56.975853 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:56.975885    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8e1d7cd1-1fa0-422e-a512-fef514857934-node-exporter-textfile\") pod \"node-exporter-cdprj\" (UID: \"8e1d7cd1-1fa0-422e-a512-fef514857934\") " pod="openshift-monitoring/node-exporter-cdprj"
Sep 19 01:13:56.976261 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:56.976073    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8e1d7cd1-1fa0-422e-a512-fef514857934-metrics-client-ca\") pod \"node-exporter-cdprj\" (UID: \"8e1d7cd1-1fa0-422e-a512-fef514857934\") " pod="openshift-monitoring/node-exporter-cdprj"
Sep 19 01:13:56.976361 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:56.976338    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8e1d7cd1-1fa0-422e-a512-fef514857934-node-exporter-accelerators-collector-config\") pod \"node-exporter-cdprj\" (UID: \"8e1d7cd1-1fa0-422e-a512-fef514857934\") " pod="openshift-monitoring/node-exporter-cdprj"
Sep 19 01:13:56.977877 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:56.977841    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8e1d7cd1-1fa0-422e-a512-fef514857934-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cdprj\" (UID: \"8e1d7cd1-1fa0-422e-a512-fef514857934\") " pod="openshift-monitoring/node-exporter-cdprj"
Sep 19 01:13:56.991409 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:56.991386    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hdvk\" (UniqueName: \"kubernetes.io/projected/8e1d7cd1-1fa0-422e-a512-fef514857934-kube-api-access-2hdvk\") pod \"node-exporter-cdprj\" (UID: \"8e1d7cd1-1fa0-422e-a512-fef514857934\") " pod="openshift-monitoring/node-exporter-cdprj"
Sep 19 01:13:57.479148 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:57.479106    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8e1d7cd1-1fa0-422e-a512-fef514857934-node-exporter-tls\") pod \"node-exporter-cdprj\" (UID: \"8e1d7cd1-1fa0-422e-a512-fef514857934\") " pod="openshift-monitoring/node-exporter-cdprj"
Sep 19 01:13:57.481490 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:57.481464    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8e1d7cd1-1fa0-422e-a512-fef514857934-node-exporter-tls\") pod \"node-exporter-cdprj\" (UID: \"8e1d7cd1-1fa0-422e-a512-fef514857934\") " pod="openshift-monitoring/node-exporter-cdprj"
Sep 19 01:13:57.700390 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:57.700359    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-cdprj"
Sep 19 01:13:57.710595 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:13:57.710570    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e1d7cd1_1fa0_422e_a512_fef514857934.slice/crio-f7ad270443c5ad88b9b65d4a4a0edc54f1080ecc4e2edd16aef4627a464c8430 WatchSource:0}: Error finding container f7ad270443c5ad88b9b65d4a4a0edc54f1080ecc4e2edd16aef4627a464c8430: Status 404 returned error can't find the container with id f7ad270443c5ad88b9b65d4a4a0edc54f1080ecc4e2edd16aef4627a464c8430
Sep 19 01:13:58.671338 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:58.671297    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cdprj" event={"ID":"8e1d7cd1-1fa0-422e-a512-fef514857934","Type":"ContainerStarted","Data":"f7ad270443c5ad88b9b65d4a4a0edc54f1080ecc4e2edd16aef4627a464c8430"}
Sep 19 01:13:59.675952 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:59.675916    2581 generic.go:356] "Generic (PLEG): container finished" podID="8e1d7cd1-1fa0-422e-a512-fef514857934" containerID="7a9af984ef3009b8cc826fc4fa5b9a39df24800a752c3657922c0389e2b2feda" exitCode=0
Sep 19 01:13:59.676297 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:13:59.676003    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cdprj" event={"ID":"8e1d7cd1-1fa0-422e-a512-fef514857934","Type":"ContainerDied","Data":"7a9af984ef3009b8cc826fc4fa5b9a39df24800a752c3657922c0389e2b2feda"}
Sep 19 01:14:00.681003 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:14:00.680968    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cdprj" event={"ID":"8e1d7cd1-1fa0-422e-a512-fef514857934","Type":"ContainerStarted","Data":"15f2f3b44d22ac09c53cfe796f15eddb2d662056d5209ef64dd9e707279889f8"}
Sep 19 01:14:00.681400 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:14:00.681018    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cdprj" event={"ID":"8e1d7cd1-1fa0-422e-a512-fef514857934","Type":"ContainerStarted","Data":"91f13003f191d9104da61fe44e2f28ddb3f68954c8a8b55672d28ce7fd3b1f15"}
Sep 19 01:14:00.710846 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:14:00.710798    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-cdprj" podStartSLOduration=3.629539006 podStartE2EDuration="4.710783356s" totalImagesPullingTime="1.010759397s" totalInitContainerRuntime="70.484953ms" isStatefulPod=true podCreationTimestamp="2026-09-19 01:13:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 01:14:00.709188624 +0000 UTC m=+91.596857732" watchObservedRunningTime="2026-09-19 01:14:00.710783356 +0000 UTC m=+91.598452443"
Sep 19 01:14:01.378198 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:14:01.378168    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6497c8758-jhscm"]
Sep 19 01:14:01.381030 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:14:01.381014    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6497c8758-jhscm"
Sep 19 01:14:01.385074 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:14:01.385051    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 19 01:14:01.385074 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:14:01.385065    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-e86ifn5n8ct81\""
Sep 19 01:14:01.385250 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:14:01.385058    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 19 01:14:01.385250 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:14:01.385101    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 19 01:14:01.385250 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:14:01.385050    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 19 01:14:01.385943 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:14:01.385926    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-4mr7j\""
Sep 19 01:14:01.395933 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:14:01.395910    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6497c8758-jhscm"]
Sep 19 01:14:01.508741 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:14:01.508717    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54dmc\" (UniqueName: \"kubernetes.io/projected/c4bbe6b9-a10c-442c-baed-8f3a102c36df-kube-api-access-54dmc\") pod \"metrics-server-6497c8758-jhscm\" (UID: \"c4bbe6b9-a10c-442c-baed-8f3a102c36df\") " pod="openshift-monitoring/metrics-server-6497c8758-jhscm"
Sep 19 01:14:01.508883 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:14:01.508756    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c4bbe6b9-a10c-442c-baed-8f3a102c36df-client-ca-bundle\") pod \"metrics-server-6497c8758-jhscm\" (UID: \"c4bbe6b9-a10c-442c-baed-8f3a102c36df\") " pod="openshift-monitoring/metrics-server-6497c8758-jhscm"
Sep 19 01:14:01.508883 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:14:01.508782    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4bbe6b9-a10c-442c-baed-8f3a102c36df-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6497c8758-jhscm\" (UID: \"c4bbe6b9-a10c-442c-baed-8f3a102c36df\") " pod="openshift-monitoring/metrics-server-6497c8758-jhscm"
Sep 19 01:14:01.508883 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:14:01.508806    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c4bbe6b9-a10c-442c-baed-8f3a102c36df-audit-log\") pod \"metrics-server-6497c8758-jhscm\" (UID: \"c4bbe6b9-a10c-442c-baed-8f3a102c36df\") " pod="openshift-monitoring/metrics-server-6497c8758-jhscm"
Sep 19 01:14:01.509042 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:14:01.508906    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c4bbe6b9-a10c-442c-baed-8f3a102c36df-metrics-server-audit-profiles\") pod \"metrics-server-6497c8758-jhscm\" (UID: \"c4bbe6b9-a10c-442c-baed-8f3a102c36df\") " pod="openshift-monitoring/metrics-server-6497c8758-jhscm"
Sep 19 01:14:01.509042 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:14:01.508949    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c4bbe6b9-a10c-442c-baed-8f3a102c36df-secret-metrics-server-client-certs\") pod \"metrics-server-6497c8758-jhscm\" (UID: \"c4bbe6b9-a10c-442c-baed-8f3a102c36df\") " pod="openshift-monitoring/metrics-server-6497c8758-jhscm"
Sep 19 01:14:01.509042 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:14:01.508967    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c4bbe6b9-a10c-442c-baed-8f3a102c36df-secret-metrics-server-tls\") pod \"metrics-server-6497c8758-jhscm\" (UID: \"c4bbe6b9-a10c-442c-baed-8f3a102c36df\") " pod="openshift-monitoring/metrics-server-6497c8758-jhscm"
Sep 19 01:14:01.568350 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:14:01.568324    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-dtn4x"]
Sep 19 01:14:01.571178 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:14:01.571160    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-dtn4x"
Sep 19 01:14:01.575037 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:14:01.575020    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 19 01:14:01.575509 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:14:01.575495    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-9skw9\""
Sep 19 01:14:01.585785 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:14:01.585764    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-dtn4x"]
Sep 19 01:14:01.602496 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:14:01.602474    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-587d7ff8fc-hhv2t"
Sep 19 01:14:01.609740 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:14:01.609721    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54dmc\" (UniqueName: \"kubernetes.io/projected/c4bbe6b9-a10c-442c-baed-8f3a102c36df-kube-api-access-54dmc\") pod \"metrics-server-6497c8758-jhscm\" (UID: \"c4bbe6b9-a10c-442c-baed-8f3a102c36df\") " pod="openshift-monitoring/metrics-server-6497c8758-jhscm"
Sep 19 01:14:01.609807 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:14:01.609751    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/55f739af-85c0-4d5a-87cc-e104fcbd1a86-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-dtn4x\" (UID: \"55f739af-85c0-4d5a-87cc-e104fcbd1a86\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-dtn4x"
Sep 19 01:14:01.609807 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:14:01.609770    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c4bbe6b9-a10c-442c-baed-8f3a102c36df-client-ca-bundle\") pod \"metrics-server-6497c8758-jhscm\" (UID: \"c4bbe6b9-a10c-442c-baed-8f3a102c36df\") " pod="openshift-monitoring/metrics-server-6497c8758-jhscm"
Sep 19 01:14:01.609930 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:14:01.609896    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4bbe6b9-a10c-442c-baed-8f3a102c36df-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6497c8758-jhscm\" (UID: \"c4bbe6b9-a10c-442c-baed-8f3a102c36df\") " pod="openshift-monitoring/metrics-server-6497c8758-jhscm"
Sep 19 01:14:01.609968 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:14:01.609946    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c4bbe6b9-a10c-442c-baed-8f3a102c36df-audit-log\") pod \"metrics-server-6497c8758-jhscm\" (UID: \"c4bbe6b9-a10c-442c-baed-8f3a102c36df\") " pod="openshift-monitoring/metrics-server-6497c8758-jhscm"
Sep 19 01:14:01.610001 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:14:01.609971    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8g4d\" (UniqueName: \"kubernetes.io/projected/55f739af-85c0-4d5a-87cc-e104fcbd1a86-kube-api-access-p8g4d\") pod \"monitoring-plugin-5984459ff8-dtn4x\" (UID: \"55f739af-85c0-4d5a-87cc-e104fcbd1a86\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-dtn4x"
Sep 19 01:14:01.610033 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:14:01.610020    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c4bbe6b9-a10c-442c-baed-8f3a102c36df-metrics-server-audit-profiles\") pod \"metrics-server-6497c8758-jhscm\" (UID: \"c4bbe6b9-a10c-442c-baed-8f3a102c36df\") " pod="openshift-monitoring/metrics-server-6497c8758-jhscm"
Sep 19 01:14:01.610093 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:14:01.610080    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c4bbe6b9-a10c-442c-baed-8f3a102c36df-secret-metrics-server-client-certs\") pod \"metrics-server-6497c8758-jhscm\" (UID: \"c4bbe6b9-a10c-442c-baed-8f3a102c36df\") " pod="openshift-monitoring/metrics-server-6497c8758-jhscm"
Sep 19 01:14:01.610146 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:14:01.610107    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c4bbe6b9-a10c-442c-baed-8f3a102c36df-secret-metrics-server-tls\") pod \"metrics-server-6497c8758-jhscm\" (UID: \"c4bbe6b9-a10c-442c-baed-8f3a102c36df\") " pod="openshift-monitoring/metrics-server-6497c8758-jhscm"
Sep 19 01:14:01.610481 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:14:01.610405    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c4bbe6b9-a10c-442c-baed-8f3a102c36df-audit-log\") pod \"metrics-server-6497c8758-jhscm\" (UID: \"c4bbe6b9-a10c-442c-baed-8f3a102c36df\") " pod="openshift-monitoring/metrics-server-6497c8758-jhscm"
Sep 19 01:14:01.610630 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:14:01.610605    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4bbe6b9-a10c-442c-baed-8f3a102c36df-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6497c8758-jhscm\" (UID: \"c4bbe6b9-a10c-442c-baed-8f3a102c36df\") " pod="openshift-monitoring/metrics-server-6497c8758-jhscm"
Sep 19 01:14:01.611150 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:14:01.611125    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c4bbe6b9-a10c-442c-baed-8f3a102c36df-metrics-server-audit-profiles\") pod \"metrics-server-6497c8758-jhscm\" (UID: \"c4bbe6b9-a10c-442c-baed-8f3a102c36df\") " pod="openshift-monitoring/metrics-server-6497c8758-jhscm"
Sep 19 01:14:01.612441 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:14:01.612418    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c4bbe6b9-a10c-442c-baed-8f3a102c36df-client-ca-bundle\") pod \"metrics-server-6497c8758-jhscm\" (UID: \"c4bbe6b9-a10c-442c-baed-8f3a102c36df\") " pod="openshift-monitoring/metrics-server-6497c8758-jhscm"
Sep 19 01:14:01.612535 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:14:01.612506    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c4bbe6b9-a10c-442c-baed-8f3a102c36df-secret-metrics-server-client-certs\") pod \"metrics-server-6497c8758-jhscm\" (UID: \"c4bbe6b9-a10c-442c-baed-8f3a102c36df\") " pod="openshift-monitoring/metrics-server-6497c8758-jhscm"
Sep 19 01:14:01.612625 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:14:01.612610    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c4bbe6b9-a10c-442c-baed-8f3a102c36df-secret-metrics-server-tls\") pod \"metrics-server-6497c8758-jhscm\" (UID: \"c4bbe6b9-a10c-442c-baed-8f3a102c36df\") " pod="openshift-monitoring/metrics-server-6497c8758-jhscm"
Sep 19 01:14:01.617816 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:14:01.617786    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54dmc\" (UniqueName: \"kubernetes.io/projected/c4bbe6b9-a10c-442c-baed-8f3a102c36df-kube-api-access-54dmc\") pod \"metrics-server-6497c8758-jhscm\" (UID: \"c4bbe6b9-a10c-442c-baed-8f3a102c36df\") " pod="openshift-monitoring/metrics-server-6497c8758-jhscm"
Sep 19 01:14:01.690291 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:14:01.690235    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6497c8758-jhscm"
Sep 19 01:14:01.710847 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:14:01.710813    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8g4d\" (UniqueName: \"kubernetes.io/projected/55f739af-85c0-4d5a-87cc-e104fcbd1a86-kube-api-access-p8g4d\") pod \"monitoring-plugin-5984459ff8-dtn4x\" (UID: \"55f739af-85c0-4d5a-87cc-e104fcbd1a86\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-dtn4x"
Sep 19 01:14:01.710964 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:14:01.710937    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/55f739af-85c0-4d5a-87cc-e104fcbd1a86-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-dtn4x\" (UID: \"55f739af-85c0-4d5a-87cc-e104fcbd1a86\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-dtn4x"
Sep 19 01:14:01.713383 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:14:01.713362    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/55f739af-85c0-4d5a-87cc-e104fcbd1a86-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-dtn4x\" (UID: \"55f739af-85c0-4d5a-87cc-e104fcbd1a86\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-dtn4x"
Sep 19 01:14:01.719081 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:14:01.719063    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8g4d\" (UniqueName: \"kubernetes.io/projected/55f739af-85c0-4d5a-87cc-e104fcbd1a86-kube-api-access-p8g4d\") pod \"monitoring-plugin-5984459ff8-dtn4x\" (UID: \"55f739af-85c0-4d5a-87cc-e104fcbd1a86\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-dtn4x"
Sep 19 01:14:01.879620 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:14:01.879591    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-dtn4x"
Sep 19 01:14:01.962422 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:14:01.962297    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6497c8758-jhscm"]
Sep 19 01:14:01.964810 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:14:01.964786    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4bbe6b9_a10c_442c_baed_8f3a102c36df.slice/crio-8cb450495d2be8a83d55bf7165b4450281b863db4a33ff2057ba21de3e686a75 WatchSource:0}: Error finding container 8cb450495d2be8a83d55bf7165b4450281b863db4a33ff2057ba21de3e686a75: Status 404 returned error can't find the container with id 8cb450495d2be8a83d55bf7165b4450281b863db4a33ff2057ba21de3e686a75
Sep 19 01:14:02.149168 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:14:02.149131    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-dtn4x"]
Sep 19 01:14:02.151641 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:14:02.151616    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55f739af_85c0_4d5a_87cc_e104fcbd1a86.slice/crio-7d1d1fa398b95e0c7d685f05d1a6b2218fa21debde2e0131a0dfa6f3c8e1fa6c WatchSource:0}: Error finding container 7d1d1fa398b95e0c7d685f05d1a6b2218fa21debde2e0131a0dfa6f3c8e1fa6c: Status 404 returned error can't find the container with id 7d1d1fa398b95e0c7d685f05d1a6b2218fa21debde2e0131a0dfa6f3c8e1fa6c
Sep 19 01:14:02.689666 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:14:02.689583    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-dtn4x" event={"ID":"55f739af-85c0-4d5a-87cc-e104fcbd1a86","Type":"ContainerStarted","Data":"7d1d1fa398b95e0c7d685f05d1a6b2218fa21debde2e0131a0dfa6f3c8e1fa6c"}
Sep 19 01:14:02.691171 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:14:02.691137    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6497c8758-jhscm" event={"ID":"c4bbe6b9-a10c-442c-baed-8f3a102c36df","Type":"ContainerStarted","Data":"8cb450495d2be8a83d55bf7165b4450281b863db4a33ff2057ba21de3e686a75"}
Sep 19 01:14:04.699932 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:14:04.699889    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-dtn4x" event={"ID":"55f739af-85c0-4d5a-87cc-e104fcbd1a86","Type":"ContainerStarted","Data":"621f236946336a833feaf89980bf6c692383b9b85b20c11119409825268e983f"}
Sep 19 01:14:04.700401 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:14:04.700069    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-dtn4x"
Sep 19 01:14:04.701755 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:14:04.701717    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6497c8758-jhscm" event={"ID":"c4bbe6b9-a10c-442c-baed-8f3a102c36df","Type":"ContainerStarted","Data":"b2b4391e4c141b8d5ecf2c8638d9ecee9c69ab38e3eb0c09a819c41d32f38ebf"}
Sep 19 01:14:04.704956 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:14:04.704926    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-dtn4x"
Sep 19 01:14:04.724002 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:14:04.723963    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5984459ff8-dtn4x" podStartSLOduration=2.135426801 podStartE2EDuration="3.72394869s" totalImagesPullingTime="1.588521889s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 01:14:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 01:14:04.722848179 +0000 UTC m=+95.610517257" watchObservedRunningTime="2026-09-19 01:14:04.72394869 +0000 UTC m=+95.611617778"
Sep 19 01:14:04.761669 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:14:04.761624    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6497c8758-jhscm" podStartSLOduration=1.989218185 podStartE2EDuration="3.761610784s" totalImagesPullingTime="1.772392599s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 01:14:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 01:14:04.745212001 +0000 UTC m=+95.632881087" watchObservedRunningTime="2026-09-19 01:14:04.761610784 +0000 UTC m=+95.649279901"
Sep 19 01:14:06.617314 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:14:06.617273    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-587d7ff8fc-hhv2t" podUID="49b29fdc-bbdc-44ff-901f-384537c101e5" containerName="registry" containerID="cri-o://50cfeef2723b4eca36ee1689fb5e81b6823718f242eaf47e3885a41a29e608e5" gracePeriod=30
Sep 19 01:14:06.856405 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:14:06.856377    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-587d7ff8fc-hhv2t"
Sep 19 01:14:06.947964 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:14:06.947885    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/49b29fdc-bbdc-44ff-901f-384537c101e5-ca-trust-extracted\") pod \"49b29fdc-bbdc-44ff-901f-384537c101e5\" (UID: \"49b29fdc-bbdc-44ff-901f-384537c101e5\") "
Sep 19 01:14:06.947964 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:14:06.947929    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/49b29fdc-bbdc-44ff-901f-384537c101e5-installation-pull-secrets\") pod \"49b29fdc-bbdc-44ff-901f-384537c101e5\" (UID: \"49b29fdc-bbdc-44ff-901f-384537c101e5\") "
Sep 19 01:14:06.947964 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:14:06.947952    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/49b29fdc-bbdc-44ff-901f-384537c101e5-image-registry-private-configuration\") pod \"49b29fdc-bbdc-44ff-901f-384537c101e5\" (UID: \"49b29fdc-bbdc-44ff-901f-384537c101e5\") "
Sep 19 01:14:06.948215 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:14:06.948007    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/49b29fdc-bbdc-44ff-901f-384537c101e5-bound-sa-token\") pod \"49b29fdc-bbdc-44ff-901f-384537c101e5\" (UID: \"49b29fdc-bbdc-44ff-901f-384537c101e5\") "
Sep 19 01:14:06.948215 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:14:06.948024    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/49b29fdc-bbdc-44ff-901f-384537c101e5-registry-certificates\") pod \"49b29fdc-bbdc-44ff-901f-384537c101e5\" (UID: \"49b29fdc-bbdc-44ff-901f-384537c101e5\") "
Sep 19 01:14:06.948215 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:14:06.948055    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/49b29fdc-bbdc-44ff-901f-384537c101e5-trusted-ca\") pod \"49b29fdc-bbdc-44ff-901f-384537c101e5\" (UID: \"49b29fdc-bbdc-44ff-901f-384537c101e5\") "
Sep 19 01:14:06.948362 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:14:06.948176    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xtqsl\" (UniqueName: \"kubernetes.io/projected/49b29fdc-bbdc-44ff-901f-384537c101e5-kube-api-access-xtqsl\") pod \"49b29fdc-bbdc-44ff-901f-384537c101e5\" (UID: \"49b29fdc-bbdc-44ff-901f-384537c101e5\") "
Sep 19 01:14:06.948362 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:14:06.948262    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/49b29fdc-bbdc-44ff-901f-384537c101e5-registry-tls\") pod \"49b29fdc-bbdc-44ff-901f-384537c101e5\" (UID: \"49b29fdc-bbdc-44ff-901f-384537c101e5\") "
Sep 19 01:14:06.948504 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:14:06.948477    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49b29fdc-bbdc-44ff-901f-384537c101e5-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "49b29fdc-bbdc-44ff-901f-384537c101e5" (UID: "49b29fdc-bbdc-44ff-901f-384537c101e5"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 01:14:06.948504 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:14:06.948492    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49b29fdc-bbdc-44ff-901f-384537c101e5-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "49b29fdc-bbdc-44ff-901f-384537c101e5" (UID: "49b29fdc-bbdc-44ff-901f-384537c101e5"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 01:14:06.950476 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:14:06.950445    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49b29fdc-bbdc-44ff-901f-384537c101e5-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "49b29fdc-bbdc-44ff-901f-384537c101e5" (UID: "49b29fdc-bbdc-44ff-901f-384537c101e5"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 01:14:06.950614 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:14:06.950499    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49b29fdc-bbdc-44ff-901f-384537c101e5-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "49b29fdc-bbdc-44ff-901f-384537c101e5" (UID: "49b29fdc-bbdc-44ff-901f-384537c101e5"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 01:14:06.950942 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:14:06.950907    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49b29fdc-bbdc-44ff-901f-384537c101e5-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "49b29fdc-bbdc-44ff-901f-384537c101e5" (UID: "49b29fdc-bbdc-44ff-901f-384537c101e5"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 01:14:06.951207 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:14:06.951176    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49b29fdc-bbdc-44ff-901f-384537c101e5-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "49b29fdc-bbdc-44ff-901f-384537c101e5" (UID: "49b29fdc-bbdc-44ff-901f-384537c101e5"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 01:14:06.951270 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:14:06.951255    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49b29fdc-bbdc-44ff-901f-384537c101e5-kube-api-access-xtqsl" (OuterVolumeSpecName: "kube-api-access-xtqsl") pod "49b29fdc-bbdc-44ff-901f-384537c101e5" (UID: "49b29fdc-bbdc-44ff-901f-384537c101e5"). InnerVolumeSpecName "kube-api-access-xtqsl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 01:14:06.957148 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:14:06.957125    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49b29fdc-bbdc-44ff-901f-384537c101e5-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "49b29fdc-bbdc-44ff-901f-384537c101e5" (UID: "49b29fdc-bbdc-44ff-901f-384537c101e5"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 01:14:07.049590 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:14:07.049567    2581 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/49b29fdc-bbdc-44ff-901f-384537c101e5-bound-sa-token\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:14:07.049590 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:14:07.049586    2581 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/49b29fdc-bbdc-44ff-901f-384537c101e5-registry-certificates\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:14:07.049709 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:14:07.049598    2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/49b29fdc-bbdc-44ff-901f-384537c101e5-trusted-ca\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:14:07.049709 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:14:07.049606    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtqsl\" (UniqueName: \"kubernetes.io/projected/49b29fdc-bbdc-44ff-901f-384537c101e5-kube-api-access-xtqsl\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:14:07.049709 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:14:07.049613    2581 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/49b29fdc-bbdc-44ff-901f-384537c101e5-registry-tls\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:14:07.049709 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:14:07.049622    2581 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/49b29fdc-bbdc-44ff-901f-384537c101e5-ca-trust-extracted\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:14:07.049709 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:14:07.049631    2581 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/49b29fdc-bbdc-44ff-901f-384537c101e5-installation-pull-secrets\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:14:07.049709 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:14:07.049641    2581 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/49b29fdc-bbdc-44ff-901f-384537c101e5-image-registry-private-configuration\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:14:07.713147 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:14:07.713119    2581 generic.go:356] "Generic (PLEG): container finished" podID="49b29fdc-bbdc-44ff-901f-384537c101e5" containerID="50cfeef2723b4eca36ee1689fb5e81b6823718f242eaf47e3885a41a29e608e5" exitCode=0
Sep 19 01:14:07.713532 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:14:07.713166    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-587d7ff8fc-hhv2t"
Sep 19 01:14:07.713532 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:14:07.713189    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-587d7ff8fc-hhv2t" event={"ID":"49b29fdc-bbdc-44ff-901f-384537c101e5","Type":"ContainerDied","Data":"50cfeef2723b4eca36ee1689fb5e81b6823718f242eaf47e3885a41a29e608e5"}
Sep 19 01:14:07.713532 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:14:07.713233    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-587d7ff8fc-hhv2t" event={"ID":"49b29fdc-bbdc-44ff-901f-384537c101e5","Type":"ContainerDied","Data":"936d6b0b5d2a86173ade67dd9d216b4d5f25033f92b4887108fce68fd3036174"}
Sep 19 01:14:07.713532 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:14:07.713257    2581 scope.go:117] "RemoveContainer" containerID="50cfeef2723b4eca36ee1689fb5e81b6823718f242eaf47e3885a41a29e608e5"
Sep 19 01:14:07.723058 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:14:07.723025    2581 scope.go:117] "RemoveContainer" containerID="50cfeef2723b4eca36ee1689fb5e81b6823718f242eaf47e3885a41a29e608e5"
Sep 19 01:14:07.723496 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:14:07.723472    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50cfeef2723b4eca36ee1689fb5e81b6823718f242eaf47e3885a41a29e608e5\": container with ID starting with 50cfeef2723b4eca36ee1689fb5e81b6823718f242eaf47e3885a41a29e608e5 not found: ID does not exist" containerID="50cfeef2723b4eca36ee1689fb5e81b6823718f242eaf47e3885a41a29e608e5"
Sep 19 01:14:07.723590 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:14:07.723505    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50cfeef2723b4eca36ee1689fb5e81b6823718f242eaf47e3885a41a29e608e5"} err="failed to get container status \"50cfeef2723b4eca36ee1689fb5e81b6823718f242eaf47e3885a41a29e608e5\": rpc error: code = NotFound desc = could not find container \"50cfeef2723b4eca36ee1689fb5e81b6823718f242eaf47e3885a41a29e608e5\": container with ID starting with 50cfeef2723b4eca36ee1689fb5e81b6823718f242eaf47e3885a41a29e608e5 not found: ID does not exist"
Sep 19 01:14:07.739578 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:14:07.739555    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-587d7ff8fc-hhv2t"]
Sep 19 01:14:07.742736 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:14:07.742715    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-587d7ff8fc-hhv2t"]
Sep 19 01:14:09.270304 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:14:09.270274    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49b29fdc-bbdc-44ff-901f-384537c101e5" path="/var/lib/kubelet/pods/49b29fdc-bbdc-44ff-901f-384537c101e5/volumes"
Sep 19 01:14:13.611928 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:14:13.611891    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-lmcnn"
Sep 19 01:14:21.690605 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:14:21.690572    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-6497c8758-jhscm"
Sep 19 01:14:21.691061 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:14:21.690630    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-6497c8758-jhscm"
Sep 19 01:14:41.696343 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:14:41.696314    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-6497c8758-jhscm"
Sep 19 01:14:41.700264 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:14:41.700235    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6497c8758-jhscm"
Sep 19 01:16:38.549412 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:16:38.549373    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-7t6km"]
Sep 19 01:16:38.549882 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:16:38.549613    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49b29fdc-bbdc-44ff-901f-384537c101e5" containerName="registry"
Sep 19 01:16:38.549882 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:16:38.549625    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="49b29fdc-bbdc-44ff-901f-384537c101e5" containerName="registry"
Sep 19 01:16:38.549882 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:16:38.549665    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="49b29fdc-bbdc-44ff-901f-384537c101e5" containerName="registry"
Sep 19 01:16:38.552354 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:16:38.552337    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-7t6km"
Sep 19 01:16:38.554708 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:16:38.554687    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 19 01:16:38.555515 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:16:38.555492    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 19 01:16:38.555610 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:16:38.555518    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-swqcj\""
Sep 19 01:16:38.566031 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:16:38.566008    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-7t6km"]
Sep 19 01:16:38.622332 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:16:38.622297    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cln5h\" (UniqueName: \"kubernetes.io/projected/c6c7a231-efcd-4a32-9f92-e4e9c74e10fc-kube-api-access-cln5h\") pod \"cert-manager-cainjector-559cdd8c87-7t6km\" (UID: \"c6c7a231-efcd-4a32-9f92-e4e9c74e10fc\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-7t6km"
Sep 19 01:16:38.622500 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:16:38.622354    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c6c7a231-efcd-4a32-9f92-e4e9c74e10fc-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-7t6km\" (UID: \"c6c7a231-efcd-4a32-9f92-e4e9c74e10fc\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-7t6km"
Sep 19 01:16:38.723690 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:16:38.723663    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cln5h\" (UniqueName: \"kubernetes.io/projected/c6c7a231-efcd-4a32-9f92-e4e9c74e10fc-kube-api-access-cln5h\") pod \"cert-manager-cainjector-559cdd8c87-7t6km\" (UID: \"c6c7a231-efcd-4a32-9f92-e4e9c74e10fc\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-7t6km"
Sep 19 01:16:38.723807 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:16:38.723711    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c6c7a231-efcd-4a32-9f92-e4e9c74e10fc-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-7t6km\" (UID: \"c6c7a231-efcd-4a32-9f92-e4e9c74e10fc\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-7t6km"
Sep 19 01:16:38.732597 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:16:38.732571    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c6c7a231-efcd-4a32-9f92-e4e9c74e10fc-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-7t6km\" (UID: \"c6c7a231-efcd-4a32-9f92-e4e9c74e10fc\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-7t6km"
Sep 19 01:16:38.732734 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:16:38.732716    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cln5h\" (UniqueName: \"kubernetes.io/projected/c6c7a231-efcd-4a32-9f92-e4e9c74e10fc-kube-api-access-cln5h\") pod \"cert-manager-cainjector-559cdd8c87-7t6km\" (UID: \"c6c7a231-efcd-4a32-9f92-e4e9c74e10fc\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-7t6km"
Sep 19 01:16:38.861598 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:16:38.861572    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-7t6km"
Sep 19 01:16:39.158256 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:16:39.158228    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-7t6km"]
Sep 19 01:16:39.159848 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:16:39.159816    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc6c7a231_efcd_4a32_9f92_e4e9c74e10fc.slice/crio-3d80765990b850bfc228dd1c3f581617ac1eb7a38ebf651451071406549cac03 WatchSource:0}: Error finding container 3d80765990b850bfc228dd1c3f581617ac1eb7a38ebf651451071406549cac03: Status 404 returned error can't find the container with id 3d80765990b850bfc228dd1c3f581617ac1eb7a38ebf651451071406549cac03
Sep 19 01:16:39.206321 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:16:39.206290    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-7t6km" event={"ID":"c6c7a231-efcd-4a32-9f92-e4e9c74e10fc","Type":"ContainerStarted","Data":"3d80765990b850bfc228dd1c3f581617ac1eb7a38ebf651451071406549cac03"}
Sep 19 01:16:43.222326 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:16:43.222291    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-7t6km" event={"ID":"c6c7a231-efcd-4a32-9f92-e4e9c74e10fc","Type":"ContainerStarted","Data":"e1ad4d3d94fec741c5b2a5ca693af894b3384f1ca668aa533cc410d94430e272"}
Sep 19 01:16:43.238843 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:16:43.238792    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-7t6km" podStartSLOduration=2.145983589 podStartE2EDuration="5.238774603s" totalImagesPullingTime="3.092791014s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 01:16:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 01:16:43.237494735 +0000 UTC m=+254.125163814" watchObservedRunningTime="2026-09-19 01:16:43.238774603 +0000 UTC m=+254.126443679"
Sep 19 01:17:12.369662 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:17:12.369631    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-74c67d8b6f-tt6jk"]
Sep 19 01:17:12.372650 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:17:12.372627    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-74c67d8b6f-tt6jk"
Sep 19 01:17:12.375696 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:17:12.375671    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 19 01:17:12.375849 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:17:12.375828    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 19 01:17:12.376408 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:17:12.376388    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 19 01:17:12.376514 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:17:12.376471    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-nzkht\""
Sep 19 01:17:12.376569 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:17:12.376533    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 19 01:17:12.376905 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:17:12.376888    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 19 01:17:12.413085 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:17:12.413062    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-74c67d8b6f-tt6jk"]
Sep 19 01:17:12.544752 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:17:12.544726    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c74h2\" (UniqueName: \"kubernetes.io/projected/d18b558d-3914-4240-a629-4df992413e22-kube-api-access-c74h2\") pod \"lws-controller-manager-74c67d8b6f-tt6jk\" (UID: \"d18b558d-3914-4240-a629-4df992413e22\") " pod="openshift-lws-operator/lws-controller-manager-74c67d8b6f-tt6jk"
Sep 19 01:17:12.544896 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:17:12.544759    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d18b558d-3914-4240-a629-4df992413e22-manager-config\") pod \"lws-controller-manager-74c67d8b6f-tt6jk\" (UID: \"d18b558d-3914-4240-a629-4df992413e22\") " pod="openshift-lws-operator/lws-controller-manager-74c67d8b6f-tt6jk"
Sep 19 01:17:12.544896 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:17:12.544782    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d18b558d-3914-4240-a629-4df992413e22-cert\") pod \"lws-controller-manager-74c67d8b6f-tt6jk\" (UID: \"d18b558d-3914-4240-a629-4df992413e22\") " pod="openshift-lws-operator/lws-controller-manager-74c67d8b6f-tt6jk"
Sep 19 01:17:12.544896 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:17:12.544807    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d18b558d-3914-4240-a629-4df992413e22-metrics-cert\") pod \"lws-controller-manager-74c67d8b6f-tt6jk\" (UID: \"d18b558d-3914-4240-a629-4df992413e22\") " pod="openshift-lws-operator/lws-controller-manager-74c67d8b6f-tt6jk"
Sep 19 01:17:12.645638 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:17:12.645576    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c74h2\" (UniqueName: \"kubernetes.io/projected/d18b558d-3914-4240-a629-4df992413e22-kube-api-access-c74h2\") pod \"lws-controller-manager-74c67d8b6f-tt6jk\" (UID: \"d18b558d-3914-4240-a629-4df992413e22\") " pod="openshift-lws-operator/lws-controller-manager-74c67d8b6f-tt6jk"
Sep 19 01:17:12.645638 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:17:12.645604    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d18b558d-3914-4240-a629-4df992413e22-manager-config\") pod \"lws-controller-manager-74c67d8b6f-tt6jk\" (UID: \"d18b558d-3914-4240-a629-4df992413e22\") " pod="openshift-lws-operator/lws-controller-manager-74c67d8b6f-tt6jk"
Sep 19 01:17:12.645638 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:17:12.645623    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d18b558d-3914-4240-a629-4df992413e22-cert\") pod \"lws-controller-manager-74c67d8b6f-tt6jk\" (UID: \"d18b558d-3914-4240-a629-4df992413e22\") " pod="openshift-lws-operator/lws-controller-manager-74c67d8b6f-tt6jk"
Sep 19 01:17:12.645638 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:17:12.645638    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d18b558d-3914-4240-a629-4df992413e22-metrics-cert\") pod \"lws-controller-manager-74c67d8b6f-tt6jk\" (UID: \"d18b558d-3914-4240-a629-4df992413e22\") " pod="openshift-lws-operator/lws-controller-manager-74c67d8b6f-tt6jk"
Sep 19 01:17:12.646271 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:17:12.646243    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d18b558d-3914-4240-a629-4df992413e22-manager-config\") pod \"lws-controller-manager-74c67d8b6f-tt6jk\" (UID: \"d18b558d-3914-4240-a629-4df992413e22\") " pod="openshift-lws-operator/lws-controller-manager-74c67d8b6f-tt6jk"
Sep 19 01:17:12.648322 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:17:12.648302    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d18b558d-3914-4240-a629-4df992413e22-metrics-cert\") pod \"lws-controller-manager-74c67d8b6f-tt6jk\" (UID: \"d18b558d-3914-4240-a629-4df992413e22\") " pod="openshift-lws-operator/lws-controller-manager-74c67d8b6f-tt6jk"
Sep 19 01:17:12.648416 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:17:12.648358    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d18b558d-3914-4240-a629-4df992413e22-cert\") pod \"lws-controller-manager-74c67d8b6f-tt6jk\" (UID: \"d18b558d-3914-4240-a629-4df992413e22\") " pod="openshift-lws-operator/lws-controller-manager-74c67d8b6f-tt6jk"
Sep 19 01:17:12.674540 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:17:12.674515    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c74h2\" (UniqueName: \"kubernetes.io/projected/d18b558d-3914-4240-a629-4df992413e22-kube-api-access-c74h2\") pod \"lws-controller-manager-74c67d8b6f-tt6jk\" (UID: \"d18b558d-3914-4240-a629-4df992413e22\") " pod="openshift-lws-operator/lws-controller-manager-74c67d8b6f-tt6jk"
Sep 19 01:17:12.681326 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:17:12.681303    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-74c67d8b6f-tt6jk"
Sep 19 01:17:12.958627 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:17:12.958536    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-74c67d8b6f-tt6jk"]
Sep 19 01:17:12.961147 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:17:12.961124    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd18b558d_3914_4240_a629_4df992413e22.slice/crio-5f348b3af9cc4f995ce888658049e955890efb715b2f0704b3c5c380209ab5f3 WatchSource:0}: Error finding container 5f348b3af9cc4f995ce888658049e955890efb715b2f0704b3c5c380209ab5f3: Status 404 returned error can't find the container with id 5f348b3af9cc4f995ce888658049e955890efb715b2f0704b3c5c380209ab5f3
Sep 19 01:17:13.320121 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:17:13.320039    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-74c67d8b6f-tt6jk" event={"ID":"d18b558d-3914-4240-a629-4df992413e22","Type":"ContainerStarted","Data":"5f348b3af9cc4f995ce888658049e955890efb715b2f0704b3c5c380209ab5f3"}
Sep 19 01:17:14.801140 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:17:14.801113    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-748475db69-bw4wd"]
Sep 19 01:17:14.813119 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:17:14.813096    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-748475db69-bw4wd"
Sep 19 01:17:14.816097 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:17:14.816073    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 19 01:17:14.816226 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:17:14.816147    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 19 01:17:14.816226 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:17:14.816218    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 19 01:17:14.816594 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:17:14.816574    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-h4tpm\""
Sep 19 01:17:14.821557 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:17:14.821534    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 19 01:17:14.843558 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:17:14.843535    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-748475db69-bw4wd"]
Sep 19 01:17:14.963874 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:17:14.963829    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/53e08af2-a376-4c3e-a17f-efabef25eca8-tmp\") pod \"opendatahub-operator-controller-manager-748475db69-bw4wd\" (UID: \"53e08af2-a376-4c3e-a17f-efabef25eca8\") " pod="opendatahub/opendatahub-operator-controller-manager-748475db69-bw4wd"
Sep 19 01:17:14.964046 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:17:14.963882    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/53e08af2-a376-4c3e-a17f-efabef25eca8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-748475db69-bw4wd\" (UID: \"53e08af2-a376-4c3e-a17f-efabef25eca8\") " pod="opendatahub/opendatahub-operator-controller-manager-748475db69-bw4wd"
Sep 19 01:17:14.964046 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:17:14.963944    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/53e08af2-a376-4c3e-a17f-efabef25eca8-manifests\") pod \"opendatahub-operator-controller-manager-748475db69-bw4wd\" (UID: \"53e08af2-a376-4c3e-a17f-efabef25eca8\") " pod="opendatahub/opendatahub-operator-controller-manager-748475db69-bw4wd"
Sep 19 01:17:14.964046 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:17:14.963968    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/53e08af2-a376-4c3e-a17f-efabef25eca8-webhook-cert\") pod \"opendatahub-operator-controller-manager-748475db69-bw4wd\" (UID: \"53e08af2-a376-4c3e-a17f-efabef25eca8\") " pod="opendatahub/opendatahub-operator-controller-manager-748475db69-bw4wd"
Sep 19 01:17:14.964046 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:17:14.964001    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxxhj\" (UniqueName: \"kubernetes.io/projected/53e08af2-a376-4c3e-a17f-efabef25eca8-kube-api-access-kxxhj\") pod \"opendatahub-operator-controller-manager-748475db69-bw4wd\" (UID: \"53e08af2-a376-4c3e-a17f-efabef25eca8\") " pod="opendatahub/opendatahub-operator-controller-manager-748475db69-bw4wd"
Sep 19 01:17:15.064839 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:17:15.064763    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/53e08af2-a376-4c3e-a17f-efabef25eca8-manifests\") pod \"opendatahub-operator-controller-manager-748475db69-bw4wd\" (UID: \"53e08af2-a376-4c3e-a17f-efabef25eca8\") " pod="opendatahub/opendatahub-operator-controller-manager-748475db69-bw4wd"
Sep 19 01:17:15.064839 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:17:15.064815    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/53e08af2-a376-4c3e-a17f-efabef25eca8-webhook-cert\") pod \"opendatahub-operator-controller-manager-748475db69-bw4wd\" (UID: \"53e08af2-a376-4c3e-a17f-efabef25eca8\") " pod="opendatahub/opendatahub-operator-controller-manager-748475db69-bw4wd"
Sep 19 01:17:15.065086 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:17:15.064852    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxxhj\" (UniqueName: \"kubernetes.io/projected/53e08af2-a376-4c3e-a17f-efabef25eca8-kube-api-access-kxxhj\") pod \"opendatahub-operator-controller-manager-748475db69-bw4wd\" (UID: \"53e08af2-a376-4c3e-a17f-efabef25eca8\") " pod="opendatahub/opendatahub-operator-controller-manager-748475db69-bw4wd"
Sep 19 01:17:15.065086 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:17:15.064932    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/53e08af2-a376-4c3e-a17f-efabef25eca8-tmp\") pod \"opendatahub-operator-controller-manager-748475db69-bw4wd\" (UID: \"53e08af2-a376-4c3e-a17f-efabef25eca8\") " pod="opendatahub/opendatahub-operator-controller-manager-748475db69-bw4wd"
Sep 19 01:17:15.065086 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:17:15.064960    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/53e08af2-a376-4c3e-a17f-efabef25eca8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-748475db69-bw4wd\" (UID: \"53e08af2-a376-4c3e-a17f-efabef25eca8\") " pod="opendatahub/opendatahub-operator-controller-manager-748475db69-bw4wd"
Sep 19 01:17:15.065245 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:17:15.065185    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/53e08af2-a376-4c3e-a17f-efabef25eca8-manifests\") pod \"opendatahub-operator-controller-manager-748475db69-bw4wd\" (UID: \"53e08af2-a376-4c3e-a17f-efabef25eca8\") " pod="opendatahub/opendatahub-operator-controller-manager-748475db69-bw4wd"
Sep 19 01:17:15.067700 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:17:15.067670    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/53e08af2-a376-4c3e-a17f-efabef25eca8-tmp\") pod \"opendatahub-operator-controller-manager-748475db69-bw4wd\" (UID: \"53e08af2-a376-4c3e-a17f-efabef25eca8\") " pod="opendatahub/opendatahub-operator-controller-manager-748475db69-bw4wd"
Sep 19 01:17:15.067832 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:17:15.067717    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/53e08af2-a376-4c3e-a17f-efabef25eca8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-748475db69-bw4wd\" (UID: \"53e08af2-a376-4c3e-a17f-efabef25eca8\") " pod="opendatahub/opendatahub-operator-controller-manager-748475db69-bw4wd"
Sep 19 01:17:15.067965 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:17:15.067945    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/53e08af2-a376-4c3e-a17f-efabef25eca8-webhook-cert\") pod \"opendatahub-operator-controller-manager-748475db69-bw4wd\" (UID: \"53e08af2-a376-4c3e-a17f-efabef25eca8\") " pod="opendatahub/opendatahub-operator-controller-manager-748475db69-bw4wd"
Sep 19 01:17:15.073769 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:17:15.073737    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxxhj\" (UniqueName: \"kubernetes.io/projected/53e08af2-a376-4c3e-a17f-efabef25eca8-kube-api-access-kxxhj\") pod \"opendatahub-operator-controller-manager-748475db69-bw4wd\" (UID: \"53e08af2-a376-4c3e-a17f-efabef25eca8\") " pod="opendatahub/opendatahub-operator-controller-manager-748475db69-bw4wd"
Sep 19 01:17:15.124680 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:17:15.124653    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-748475db69-bw4wd"
Sep 19 01:17:15.956756 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:17:15.956727    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-748475db69-bw4wd"]
Sep 19 01:17:16.057777 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:17:16.057740    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53e08af2_a376_4c3e_a17f_efabef25eca8.slice/crio-4a2551c1c729e34dc6cc4a80dd1f504b9b6cb0046d266aac8fef05d837ebd480 WatchSource:0}: Error finding container 4a2551c1c729e34dc6cc4a80dd1f504b9b6cb0046d266aac8fef05d837ebd480: Status 404 returned error can't find the container with id 4a2551c1c729e34dc6cc4a80dd1f504b9b6cb0046d266aac8fef05d837ebd480
Sep 19 01:17:16.333336 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:17:16.333262    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-74c67d8b6f-tt6jk" event={"ID":"d18b558d-3914-4240-a629-4df992413e22","Type":"ContainerStarted","Data":"84b8269fdff17271ed220a6df4261a832b507efd3fff2a5b43f6790894197210"}
Sep 19 01:17:16.333476 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:17:16.333366    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-74c67d8b6f-tt6jk"
Sep 19 01:17:16.334623 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:17:16.334601    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-748475db69-bw4wd" event={"ID":"53e08af2-a376-4c3e-a17f-efabef25eca8","Type":"ContainerStarted","Data":"4a2551c1c729e34dc6cc4a80dd1f504b9b6cb0046d266aac8fef05d837ebd480"}
Sep 19 01:17:16.360137 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:17:16.360096    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-74c67d8b6f-tt6jk" podStartSLOduration=1.235853127 podStartE2EDuration="4.36008174s" totalImagesPullingTime="3.124228613s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 01:17:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 01:17:16.358367545 +0000 UTC m=+287.246036642" watchObservedRunningTime="2026-09-19 01:17:16.36008174 +0000 UTC m=+287.247750895"
Sep 19 01:17:19.348412 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:17:19.348382    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-748475db69-bw4wd" event={"ID":"53e08af2-a376-4c3e-a17f-efabef25eca8","Type":"ContainerStarted","Data":"0cde3c5736d0737aa69b2dd0e4da61e2ed755f63f55c2b5f749658a8248ca50e"}
Sep 19 01:17:20.353810 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:17:20.353779    2581 generic.go:356] "Generic (PLEG): container finished" podID="53e08af2-a376-4c3e-a17f-efabef25eca8" containerID="0cde3c5736d0737aa69b2dd0e4da61e2ed755f63f55c2b5f749658a8248ca50e" exitCode=0
Sep 19 01:17:20.354170 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:17:20.353840    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-748475db69-bw4wd" event={"ID":"53e08af2-a376-4c3e-a17f-efabef25eca8","Type":"ContainerDied","Data":"0cde3c5736d0737aa69b2dd0e4da61e2ed755f63f55c2b5f749658a8248ca50e"}
Sep 19 01:17:21.363642 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:17:21.363604    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-748475db69-bw4wd" event={"ID":"53e08af2-a376-4c3e-a17f-efabef25eca8","Type":"ContainerStarted","Data":"8f34fb0fef51cdc7d943c2a8db92f960a3552b2dd43e371a5cc2016c5fe64877"}
Sep 19 01:17:21.364082 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:17:21.363713    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-748475db69-bw4wd"
Sep 19 01:17:21.389645 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:17:21.389598    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-748475db69-bw4wd" podStartSLOduration=3.208907158 podStartE2EDuration="7.389584377s" totalImagesPullingTime="3.145836801s" totalInitContainerRuntime="1.034840418s" isStatefulPod=false podCreationTimestamp="2026-09-19 01:17:14 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 01:17:21.388711073 +0000 UTC m=+292.276380151" watchObservedRunningTime="2026-09-19 01:17:21.389584377 +0000 UTC m=+292.277253516"
Sep 19 01:17:27.343095 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:17:27.343064    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-74c67d8b6f-tt6jk"
Sep 19 01:17:31.147668 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:17:31.147637    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x785d_f759a12f-5b6f-4924-96f2-dd5e00fc1348/ovn-acl-logging/0.log"
Sep 19 01:17:31.151616 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:17:31.151595    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x785d_f759a12f-5b6f-4924-96f2-dd5e00fc1348/ovn-acl-logging/0.log"
Sep 19 01:17:31.155537 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:17:31.155515    2581 kubelet.go:1648] "Image garbage collection succeeded"
Sep 19 01:17:32.368913 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:17:32.368879    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-748475db69-bw4wd"
Sep 19 01:17:35.039434 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:17:35.039401    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5c6796f956-5wjrr"]
Sep 19 01:17:35.042305 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:17:35.042289    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5c6796f956-5wjrr"
Sep 19 01:17:35.046725 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:17:35.046703    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 19 01:17:35.049804 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:17:35.049786    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 19 01:17:35.049899 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:17:35.049850    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-6k42h\""
Sep 19 01:17:35.050187 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:17:35.050167    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 19 01:17:35.050283 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:17:35.050195    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 19 01:17:35.077081 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:17:35.077059    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5c6796f956-5wjrr"]
Sep 19 01:17:35.207235 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:17:35.207203    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88237eb3-d014-4bf8-9d7f-05ea6cdc1768-tls-certs\") pod \"kube-auth-proxy-5c6796f956-5wjrr\" (UID: \"88237eb3-d014-4bf8-9d7f-05ea6cdc1768\") " pod="openshift-ingress/kube-auth-proxy-5c6796f956-5wjrr"
Sep 19 01:17:35.207235 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:17:35.207239    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6l7r5\" (UniqueName: \"kubernetes.io/projected/88237eb3-d014-4bf8-9d7f-05ea6cdc1768-kube-api-access-6l7r5\") pod \"kube-auth-proxy-5c6796f956-5wjrr\" (UID: \"88237eb3-d014-4bf8-9d7f-05ea6cdc1768\") " pod="openshift-ingress/kube-auth-proxy-5c6796f956-5wjrr"
Sep 19 01:17:35.207468 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:17:35.207263    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/88237eb3-d014-4bf8-9d7f-05ea6cdc1768-tmp\") pod \"kube-auth-proxy-5c6796f956-5wjrr\" (UID: \"88237eb3-d014-4bf8-9d7f-05ea6cdc1768\") " pod="openshift-ingress/kube-auth-proxy-5c6796f956-5wjrr"
Sep 19 01:17:35.308547 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:17:35.308465    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88237eb3-d014-4bf8-9d7f-05ea6cdc1768-tls-certs\") pod \"kube-auth-proxy-5c6796f956-5wjrr\" (UID: \"88237eb3-d014-4bf8-9d7f-05ea6cdc1768\") " pod="openshift-ingress/kube-auth-proxy-5c6796f956-5wjrr"
Sep 19 01:17:35.308547 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:17:35.308498    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6l7r5\" (UniqueName: \"kubernetes.io/projected/88237eb3-d014-4bf8-9d7f-05ea6cdc1768-kube-api-access-6l7r5\") pod \"kube-auth-proxy-5c6796f956-5wjrr\" (UID: \"88237eb3-d014-4bf8-9d7f-05ea6cdc1768\") " pod="openshift-ingress/kube-auth-proxy-5c6796f956-5wjrr"
Sep 19 01:17:35.308547 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:17:35.308522    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/88237eb3-d014-4bf8-9d7f-05ea6cdc1768-tmp\") pod \"kube-auth-proxy-5c6796f956-5wjrr\" (UID: \"88237eb3-d014-4bf8-9d7f-05ea6cdc1768\") " pod="openshift-ingress/kube-auth-proxy-5c6796f956-5wjrr"
Sep 19 01:17:35.310956 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:17:35.310925    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/88237eb3-d014-4bf8-9d7f-05ea6cdc1768-tmp\") pod \"kube-auth-proxy-5c6796f956-5wjrr\" (UID: \"88237eb3-d014-4bf8-9d7f-05ea6cdc1768\") " pod="openshift-ingress/kube-auth-proxy-5c6796f956-5wjrr"
Sep 19 01:17:35.311108 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:17:35.311091    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88237eb3-d014-4bf8-9d7f-05ea6cdc1768-tls-certs\") pod \"kube-auth-proxy-5c6796f956-5wjrr\" (UID: \"88237eb3-d014-4bf8-9d7f-05ea6cdc1768\") " pod="openshift-ingress/kube-auth-proxy-5c6796f956-5wjrr"
Sep 19 01:17:35.328748 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:17:35.328719    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6l7r5\" (UniqueName: \"kubernetes.io/projected/88237eb3-d014-4bf8-9d7f-05ea6cdc1768-kube-api-access-6l7r5\") pod \"kube-auth-proxy-5c6796f956-5wjrr\" (UID: \"88237eb3-d014-4bf8-9d7f-05ea6cdc1768\") " pod="openshift-ingress/kube-auth-proxy-5c6796f956-5wjrr"
Sep 19 01:17:35.351587 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:17:35.351562    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5c6796f956-5wjrr"
Sep 19 01:17:35.627591 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:17:35.627500    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5c6796f956-5wjrr"]
Sep 19 01:17:35.630237 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:17:35.630214    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88237eb3_d014_4bf8_9d7f_05ea6cdc1768.slice/crio-670731c7769b0f9b7de0059a490f7360661afcfe4d93e5bbf4e9d9dbe095a73b WatchSource:0}: Error finding container 670731c7769b0f9b7de0059a490f7360661afcfe4d93e5bbf4e9d9dbe095a73b: Status 404 returned error can't find the container with id 670731c7769b0f9b7de0059a490f7360661afcfe4d93e5bbf4e9d9dbe095a73b
Sep 19 01:17:35.631939 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:17:35.631925    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 19 01:17:36.418322 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:17:36.418273    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5c6796f956-5wjrr" event={"ID":"88237eb3-d014-4bf8-9d7f-05ea6cdc1768","Type":"ContainerStarted","Data":"670731c7769b0f9b7de0059a490f7360661afcfe4d93e5bbf4e9d9dbe095a73b"}
Sep 19 01:17:39.430996 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:17:39.430964    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5c6796f956-5wjrr" event={"ID":"88237eb3-d014-4bf8-9d7f-05ea6cdc1768","Type":"ContainerStarted","Data":"51ca18c0bb4d075512ca5a407e06524281f4a4a675702892dd750f59e26b5192"}
Sep 19 01:17:39.453417 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:17:39.453371    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5c6796f956-5wjrr" podStartSLOduration=1.5420777540000001 podStartE2EDuration="4.453357051s" totalImagesPullingTime="2.911279297s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 01:17:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 01:17:39.452227103 +0000 UTC m=+310.339896193" watchObservedRunningTime="2026-09-19 01:17:39.453357051 +0000 UTC m=+310.341026128"
Sep 19 01:19:16.532374 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:19:16.532335    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-8swpc"]
Sep 19 01:19:16.534310 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:19:16.534294    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8swpc"
Sep 19 01:19:16.537105 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:19:16.537081    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-p297g\""
Sep 19 01:19:16.537240 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:19:16.537110    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 19 01:19:16.537447 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:19:16.537433    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 19 01:19:16.537691 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:19:16.537677    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 19 01:19:16.537795 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:19:16.537778    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 19 01:19:16.549663 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:19:16.549633    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-8swpc"]
Sep 19 01:19:16.661902 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:19:16.661852    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6hpx\" (UniqueName: \"kubernetes.io/projected/028bd24e-116f-4004-a0e2-de3b4943146f-kube-api-access-s6hpx\") pod \"kuadrant-console-plugin-fbdd4f445-8swpc\" (UID: \"028bd24e-116f-4004-a0e2-de3b4943146f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8swpc"
Sep 19 01:19:16.662075 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:19:16.661924    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/028bd24e-116f-4004-a0e2-de3b4943146f-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-8swpc\" (UID: \"028bd24e-116f-4004-a0e2-de3b4943146f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8swpc"
Sep 19 01:19:16.662075 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:19:16.661950    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/028bd24e-116f-4004-a0e2-de3b4943146f-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-8swpc\" (UID: \"028bd24e-116f-4004-a0e2-de3b4943146f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8swpc"
Sep 19 01:19:16.763157 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:19:16.763117    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6hpx\" (UniqueName: \"kubernetes.io/projected/028bd24e-116f-4004-a0e2-de3b4943146f-kube-api-access-s6hpx\") pod \"kuadrant-console-plugin-fbdd4f445-8swpc\" (UID: \"028bd24e-116f-4004-a0e2-de3b4943146f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8swpc"
Sep 19 01:19:16.763265 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:19:16.763172    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/028bd24e-116f-4004-a0e2-de3b4943146f-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-8swpc\" (UID: \"028bd24e-116f-4004-a0e2-de3b4943146f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8swpc"
Sep 19 01:19:16.763265 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:19:16.763194    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/028bd24e-116f-4004-a0e2-de3b4943146f-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-8swpc\" (UID: \"028bd24e-116f-4004-a0e2-de3b4943146f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8swpc"
Sep 19 01:19:16.763336 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:19:16.763277    2581 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Sep 19 01:19:16.763336 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:19:16.763332    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/028bd24e-116f-4004-a0e2-de3b4943146f-plugin-serving-cert podName:028bd24e-116f-4004-a0e2-de3b4943146f nodeName:}" failed. No retries permitted until 2026-09-19 01:19:17.263315564 +0000 UTC m=+408.150984619 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/028bd24e-116f-4004-a0e2-de3b4943146f-plugin-serving-cert") pod "kuadrant-console-plugin-fbdd4f445-8swpc" (UID: "028bd24e-116f-4004-a0e2-de3b4943146f") : secret "plugin-serving-cert" not found
Sep 19 01:19:16.763902 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:19:16.763856    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/028bd24e-116f-4004-a0e2-de3b4943146f-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-8swpc\" (UID: \"028bd24e-116f-4004-a0e2-de3b4943146f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8swpc"
Sep 19 01:19:16.773782 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:19:16.773760    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6hpx\" (UniqueName: \"kubernetes.io/projected/028bd24e-116f-4004-a0e2-de3b4943146f-kube-api-access-s6hpx\") pod \"kuadrant-console-plugin-fbdd4f445-8swpc\" (UID: \"028bd24e-116f-4004-a0e2-de3b4943146f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8swpc"
Sep 19 01:19:17.266597 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:19:17.266563    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/028bd24e-116f-4004-a0e2-de3b4943146f-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-8swpc\" (UID: \"028bd24e-116f-4004-a0e2-de3b4943146f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8swpc"
Sep 19 01:19:17.269082 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:19:17.269059    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/028bd24e-116f-4004-a0e2-de3b4943146f-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-8swpc\" (UID: \"028bd24e-116f-4004-a0e2-de3b4943146f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8swpc"
Sep 19 01:19:17.442892 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:19:17.442851    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8swpc"
Sep 19 01:19:17.716695 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:19:17.716666    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-8swpc"]
Sep 19 01:19:17.719163 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:19:17.719140    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod028bd24e_116f_4004_a0e2_de3b4943146f.slice/crio-4beddab00892ba0d6acef37f170ae9f0f87cb4c5e485f65295408f4bd1162635 WatchSource:0}: Error finding container 4beddab00892ba0d6acef37f170ae9f0f87cb4c5e485f65295408f4bd1162635: Status 404 returned error can't find the container with id 4beddab00892ba0d6acef37f170ae9f0f87cb4c5e485f65295408f4bd1162635
Sep 19 01:19:17.779893 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:19:17.779842    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8swpc" event={"ID":"028bd24e-116f-4004-a0e2-de3b4943146f","Type":"ContainerStarted","Data":"4beddab00892ba0d6acef37f170ae9f0f87cb4c5e485f65295408f4bd1162635"}
Sep 19 01:19:22.804244 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:19:22.804208    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8swpc" event={"ID":"028bd24e-116f-4004-a0e2-de3b4943146f","Type":"ContainerStarted","Data":"5b27b82f2d2cff9947d042526b501821b0f5edf0305b9948242f0a1df0edc76b"}
Sep 19 01:19:22.822171 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:19:22.822119    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8swpc" podStartSLOduration=2.303713405 podStartE2EDuration="6.822103559s" totalImagesPullingTime="4.518390154s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 01:19:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 01:19:22.821901295 +0000 UTC m=+413.709570373" watchObservedRunningTime="2026-09-19 01:19:22.822103559 +0000 UTC m=+413.709772637"
Sep 19 01:19:58.498475 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:19:58.498400    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c9nsz"]
Sep 19 01:19:58.511920 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:19:58.511897    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c9nsz"]
Sep 19 01:19:58.512048 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:19:58.511980    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-c9nsz"
Sep 19 01:19:58.514322 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:19:58.514298    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 19 01:19:58.545400 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:19:58.545377    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c9nsz"]
Sep 19 01:19:58.557696 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:19:58.557680    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4a21d745-92cb-4219-986d-d7bb9fb324d3-config-file\") pod \"limitador-limitador-84994f7ffc-c9nsz\" (UID: \"4a21d745-92cb-4219-986d-d7bb9fb324d3\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-c9nsz"
Sep 19 01:19:58.557766 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:19:58.557714    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnvnq\" (UniqueName: \"kubernetes.io/projected/4a21d745-92cb-4219-986d-d7bb9fb324d3-kube-api-access-qnvnq\") pod \"limitador-limitador-84994f7ffc-c9nsz\" (UID: \"4a21d745-92cb-4219-986d-d7bb9fb324d3\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-c9nsz"
Sep 19 01:19:58.658231 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:19:58.658201    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4a21d745-92cb-4219-986d-d7bb9fb324d3-config-file\") pod \"limitador-limitador-84994f7ffc-c9nsz\" (UID: \"4a21d745-92cb-4219-986d-d7bb9fb324d3\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-c9nsz"
Sep 19 01:19:58.658328 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:19:58.658245    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnvnq\" (UniqueName: \"kubernetes.io/projected/4a21d745-92cb-4219-986d-d7bb9fb324d3-kube-api-access-qnvnq\") pod \"limitador-limitador-84994f7ffc-c9nsz\" (UID: \"4a21d745-92cb-4219-986d-d7bb9fb324d3\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-c9nsz"
Sep 19 01:19:58.658770 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:19:58.658753    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4a21d745-92cb-4219-986d-d7bb9fb324d3-config-file\") pod \"limitador-limitador-84994f7ffc-c9nsz\" (UID: \"4a21d745-92cb-4219-986d-d7bb9fb324d3\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-c9nsz"
Sep 19 01:19:58.666597 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:19:58.666574    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnvnq\" (UniqueName: \"kubernetes.io/projected/4a21d745-92cb-4219-986d-d7bb9fb324d3-kube-api-access-qnvnq\") pod \"limitador-limitador-84994f7ffc-c9nsz\" (UID: \"4a21d745-92cb-4219-986d-d7bb9fb324d3\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-c9nsz"
Sep 19 01:19:58.821892 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:19:58.821775    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-c9nsz"
Sep 19 01:19:59.093482 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:19:59.093346    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c9nsz"]
Sep 19 01:19:59.096898 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:19:59.096844    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a21d745_92cb_4219_986d_d7bb9fb324d3.slice/crio-cd34192dea3775aaffac450e719e769754b04bcefd8550ae64235332a1696f62 WatchSource:0}: Error finding container cd34192dea3775aaffac450e719e769754b04bcefd8550ae64235332a1696f62: Status 404 returned error can't find the container with id cd34192dea3775aaffac450e719e769754b04bcefd8550ae64235332a1696f62
Sep 19 01:19:59.170026 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:19:59.169997    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-dt7ph"]
Sep 19 01:19:59.172677 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:19:59.172660    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-dt7ph"
Sep 19 01:19:59.179015 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:19:59.178995    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-m97cq\""
Sep 19 01:19:59.191513 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:19:59.191487    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-dt7ph"]
Sep 19 01:19:59.263354 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:19:59.263328    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2fjh\" (UniqueName: \"kubernetes.io/projected/5b648ceb-651a-460b-b2a4-0e57b4dc61aa-kube-api-access-l2fjh\") pod \"authorino-59c974bd48-dt7ph\" (UID: \"5b648ceb-651a-460b-b2a4-0e57b4dc61aa\") " pod="kuadrant-system/authorino-59c974bd48-dt7ph"
Sep 19 01:19:59.363970 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:19:59.363903    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2fjh\" (UniqueName: \"kubernetes.io/projected/5b648ceb-651a-460b-b2a4-0e57b4dc61aa-kube-api-access-l2fjh\") pod \"authorino-59c974bd48-dt7ph\" (UID: \"5b648ceb-651a-460b-b2a4-0e57b4dc61aa\") " pod="kuadrant-system/authorino-59c974bd48-dt7ph"
Sep 19 01:19:59.388564 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:19:59.388541    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2fjh\" (UniqueName: \"kubernetes.io/projected/5b648ceb-651a-460b-b2a4-0e57b4dc61aa-kube-api-access-l2fjh\") pod \"authorino-59c974bd48-dt7ph\" (UID: \"5b648ceb-651a-460b-b2a4-0e57b4dc61aa\") " pod="kuadrant-system/authorino-59c974bd48-dt7ph"
Sep 19 01:19:59.481670 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:19:59.481638    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-dt7ph"
Sep 19 01:19:59.760372 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:19:59.760297    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-dt7ph"]
Sep 19 01:19:59.764482 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:19:59.764454    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5b648ceb_651a_460b_b2a4_0e57b4dc61aa.slice/crio-8edc50f2522d9cc6f3e1cd5b4976a894d7bf01cf054634d1351486caa3c55d4d WatchSource:0}: Error finding container 8edc50f2522d9cc6f3e1cd5b4976a894d7bf01cf054634d1351486caa3c55d4d: Status 404 returned error can't find the container with id 8edc50f2522d9cc6f3e1cd5b4976a894d7bf01cf054634d1351486caa3c55d4d
Sep 19 01:19:59.932406 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:19:59.932373    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-c9nsz" event={"ID":"4a21d745-92cb-4219-986d-d7bb9fb324d3","Type":"ContainerStarted","Data":"cd34192dea3775aaffac450e719e769754b04bcefd8550ae64235332a1696f62"}
Sep 19 01:19:59.933475 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:19:59.933451    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-dt7ph" event={"ID":"5b648ceb-651a-460b-b2a4-0e57b4dc61aa","Type":"ContainerStarted","Data":"8edc50f2522d9cc6f3e1cd5b4976a894d7bf01cf054634d1351486caa3c55d4d"}
Sep 19 01:20:02.948080 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:20:02.948045    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-c9nsz" event={"ID":"4a21d745-92cb-4219-986d-d7bb9fb324d3","Type":"ContainerStarted","Data":"4b93c296c3a2d319ae637502f5d619526d54bfc86a57b1b26160541c31898058"}
Sep 19 01:20:02.948489 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:20:02.948208    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-c9nsz"
Sep 19 01:20:02.949464 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:20:02.949443    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-dt7ph" event={"ID":"5b648ceb-651a-460b-b2a4-0e57b4dc61aa","Type":"ContainerStarted","Data":"57f34436c8081933ea1cce5194c36fffab4b1ff6123c1124fed7c80b1d1e65d5"}
Sep 19 01:20:02.967357 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:20:02.967313    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-c9nsz" podStartSLOduration=1.748120925 podStartE2EDuration="4.967299734s" totalImagesPullingTime="3.219178809s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 01:19:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 01:20:02.965832599 +0000 UTC m=+453.853501679" watchObservedRunningTime="2026-09-19 01:20:02.967299734 +0000 UTC m=+453.854968812"
Sep 19 01:20:13.954814 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:20:13.954786    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-c9nsz"
Sep 19 01:20:13.972443 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:20:13.972395    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-dt7ph" podStartSLOduration=12.413519067 podStartE2EDuration="14.972380431s" totalImagesPullingTime="2.558861364s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 01:19:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 01:20:02.981161548 +0000 UTC m=+453.868830625" watchObservedRunningTime="2026-09-19 01:20:13.972380431 +0000 UTC m=+464.860049508"
Sep 19 01:20:34.905516 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:20:34.905480    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-dt7ph"]
Sep 19 01:20:34.906070 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:20:34.905708    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-dt7ph" podUID="5b648ceb-651a-460b-b2a4-0e57b4dc61aa" containerName="authorino" containerID="cri-o://57f34436c8081933ea1cce5194c36fffab4b1ff6123c1124fed7c80b1d1e65d5" gracePeriod=30
Sep 19 01:20:35.075885 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:20:35.075831    2581 generic.go:356] "Generic (PLEG): container finished" podID="5b648ceb-651a-460b-b2a4-0e57b4dc61aa" containerID="57f34436c8081933ea1cce5194c36fffab4b1ff6123c1124fed7c80b1d1e65d5" exitCode=0
Sep 19 01:20:35.076032 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:20:35.075904    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-dt7ph" event={"ID":"5b648ceb-651a-460b-b2a4-0e57b4dc61aa","Type":"ContainerDied","Data":"57f34436c8081933ea1cce5194c36fffab4b1ff6123c1124fed7c80b1d1e65d5"}
Sep 19 01:20:35.161210 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:20:35.161153    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-dt7ph"
Sep 19 01:20:35.226374 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:20:35.226343    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l2fjh\" (UniqueName: \"kubernetes.io/projected/5b648ceb-651a-460b-b2a4-0e57b4dc61aa-kube-api-access-l2fjh\") pod \"5b648ceb-651a-460b-b2a4-0e57b4dc61aa\" (UID: \"5b648ceb-651a-460b-b2a4-0e57b4dc61aa\") "
Sep 19 01:20:35.228544 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:20:35.228517    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b648ceb-651a-460b-b2a4-0e57b4dc61aa-kube-api-access-l2fjh" (OuterVolumeSpecName: "kube-api-access-l2fjh") pod "5b648ceb-651a-460b-b2a4-0e57b4dc61aa" (UID: "5b648ceb-651a-460b-b2a4-0e57b4dc61aa"). InnerVolumeSpecName "kube-api-access-l2fjh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 01:20:35.327655 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:20:35.327627    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2fjh\" (UniqueName: \"kubernetes.io/projected/5b648ceb-651a-460b-b2a4-0e57b4dc61aa-kube-api-access-l2fjh\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:20:36.081414 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:20:36.081379    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-dt7ph" event={"ID":"5b648ceb-651a-460b-b2a4-0e57b4dc61aa","Type":"ContainerDied","Data":"8edc50f2522d9cc6f3e1cd5b4976a894d7bf01cf054634d1351486caa3c55d4d"}
Sep 19 01:20:36.081414 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:20:36.081398    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-dt7ph"
Sep 19 01:20:36.081958 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:20:36.081428    2581 scope.go:117] "RemoveContainer" containerID="57f34436c8081933ea1cce5194c36fffab4b1ff6123c1124fed7c80b1d1e65d5"
Sep 19 01:20:36.101182 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:20:36.101162    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-dt7ph"]
Sep 19 01:20:36.110167 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:20:36.110150    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-dt7ph"]
Sep 19 01:20:37.271349 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:20:37.271319    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b648ceb-651a-460b-b2a4-0e57b4dc61aa" path="/var/lib/kubelet/pods/5b648ceb-651a-460b-b2a4-0e57b4dc61aa/volumes"
Sep 19 01:21:18.071304 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:21:18.071226    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c9nsz"]
Sep 19 01:21:37.290451 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:21:37.290412    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c9nsz"]
Sep 19 01:21:46.983274 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:21:46.983241    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-dnrsw"]
Sep 19 01:21:46.983738 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:21:46.983638    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b648ceb-651a-460b-b2a4-0e57b4dc61aa" containerName="authorino"
Sep 19 01:21:46.983738 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:21:46.983656    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b648ceb-651a-460b-b2a4-0e57b4dc61aa" containerName="authorino"
Sep 19 01:21:46.983807 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:21:46.983746    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b648ceb-651a-460b-b2a4-0e57b4dc61aa" containerName="authorino"
Sep 19 01:21:46.985539 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:21:46.985520    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dnrsw"
Sep 19 01:21:46.988125 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:21:46.988107    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 19 01:21:46.988812 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:21:46.988790    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 19 01:21:46.988965 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:21:46.988826    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-z77ck\""
Sep 19 01:21:46.988965 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:21:46.988895    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Sep 19 01:21:46.999471 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:21:46.999450    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-dnrsw"]
Sep 19 01:21:47.006343 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:21:47.006318    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7847b3fe-4b9a-4416-93ec-5c5ecbce7687-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dnrsw\" (UID: \"7847b3fe-4b9a-4416-93ec-5c5ecbce7687\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dnrsw"
Sep 19 01:21:47.006453 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:21:47.006347    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7847b3fe-4b9a-4416-93ec-5c5ecbce7687-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dnrsw\" (UID: \"7847b3fe-4b9a-4416-93ec-5c5ecbce7687\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dnrsw"
Sep 19 01:21:47.006453 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:21:47.006363    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7847b3fe-4b9a-4416-93ec-5c5ecbce7687-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dnrsw\" (UID: \"7847b3fe-4b9a-4416-93ec-5c5ecbce7687\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dnrsw"
Sep 19 01:21:47.006453 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:21:47.006387    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7847b3fe-4b9a-4416-93ec-5c5ecbce7687-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dnrsw\" (UID: \"7847b3fe-4b9a-4416-93ec-5c5ecbce7687\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dnrsw"
Sep 19 01:21:47.006453 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:21:47.006439    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7847b3fe-4b9a-4416-93ec-5c5ecbce7687-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dnrsw\" (UID: \"7847b3fe-4b9a-4416-93ec-5c5ecbce7687\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dnrsw"
Sep 19 01:21:47.006669 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:21:47.006455    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4r4b4\" (UniqueName: \"kubernetes.io/projected/7847b3fe-4b9a-4416-93ec-5c5ecbce7687-kube-api-access-4r4b4\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dnrsw\" (UID: \"7847b3fe-4b9a-4416-93ec-5c5ecbce7687\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dnrsw"
Sep 19 01:21:47.107692 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:21:47.107663    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7847b3fe-4b9a-4416-93ec-5c5ecbce7687-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dnrsw\" (UID: \"7847b3fe-4b9a-4416-93ec-5c5ecbce7687\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dnrsw"
Sep 19 01:21:47.107692 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:21:47.107694    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7847b3fe-4b9a-4416-93ec-5c5ecbce7687-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dnrsw\" (UID: \"7847b3fe-4b9a-4416-93ec-5c5ecbce7687\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dnrsw"
Sep 19 01:21:47.107849 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:21:47.107709    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7847b3fe-4b9a-4416-93ec-5c5ecbce7687-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dnrsw\" (UID: \"7847b3fe-4b9a-4416-93ec-5c5ecbce7687\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dnrsw"
Sep 19 01:21:47.107849 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:21:47.107733    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7847b3fe-4b9a-4416-93ec-5c5ecbce7687-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dnrsw\" (UID: \"7847b3fe-4b9a-4416-93ec-5c5ecbce7687\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dnrsw"
Sep 19 01:21:47.107849 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:21:47.107775    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7847b3fe-4b9a-4416-93ec-5c5ecbce7687-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dnrsw\" (UID: \"7847b3fe-4b9a-4416-93ec-5c5ecbce7687\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dnrsw"
Sep 19 01:21:47.107849 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:21:47.107797    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r4b4\" (UniqueName: \"kubernetes.io/projected/7847b3fe-4b9a-4416-93ec-5c5ecbce7687-kube-api-access-4r4b4\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dnrsw\" (UID: \"7847b3fe-4b9a-4416-93ec-5c5ecbce7687\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dnrsw"
Sep 19 01:21:47.108075 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:21:47.108056    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7847b3fe-4b9a-4416-93ec-5c5ecbce7687-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dnrsw\" (UID: \"7847b3fe-4b9a-4416-93ec-5c5ecbce7687\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dnrsw"
Sep 19 01:21:47.108135 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:21:47.108119    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7847b3fe-4b9a-4416-93ec-5c5ecbce7687-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dnrsw\" (UID: \"7847b3fe-4b9a-4416-93ec-5c5ecbce7687\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dnrsw"
Sep 19 01:21:47.108249 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:21:47.108231    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7847b3fe-4b9a-4416-93ec-5c5ecbce7687-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dnrsw\" (UID: \"7847b3fe-4b9a-4416-93ec-5c5ecbce7687\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dnrsw"
Sep 19 01:21:47.110159 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:21:47.110133    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7847b3fe-4b9a-4416-93ec-5c5ecbce7687-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dnrsw\" (UID: \"7847b3fe-4b9a-4416-93ec-5c5ecbce7687\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dnrsw"
Sep 19 01:21:47.110374 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:21:47.110357    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7847b3fe-4b9a-4416-93ec-5c5ecbce7687-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dnrsw\" (UID: \"7847b3fe-4b9a-4416-93ec-5c5ecbce7687\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dnrsw"
Sep 19 01:21:47.118886 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:21:47.118850    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4r4b4\" (UniqueName: \"kubernetes.io/projected/7847b3fe-4b9a-4416-93ec-5c5ecbce7687-kube-api-access-4r4b4\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dnrsw\" (UID: \"7847b3fe-4b9a-4416-93ec-5c5ecbce7687\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dnrsw"
Sep 19 01:21:47.295471 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:21:47.295408    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dnrsw"
Sep 19 01:21:47.566909 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:21:47.566880    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-dnrsw"]
Sep 19 01:21:47.568744 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:21:47.568721    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7847b3fe_4b9a_4416_93ec_5c5ecbce7687.slice/crio-e892d99ad1520c1b641189fc7d7bf8ecb67c2d652581fc6c7b2ad05ed21b0e4f WatchSource:0}: Error finding container e892d99ad1520c1b641189fc7d7bf8ecb67c2d652581fc6c7b2ad05ed21b0e4f: Status 404 returned error can't find the container with id e892d99ad1520c1b641189fc7d7bf8ecb67c2d652581fc6c7b2ad05ed21b0e4f
Sep 19 01:21:47.700220 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:21:47.700193    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c9nsz"]
Sep 19 01:21:48.361565 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:21:48.361532    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dnrsw" event={"ID":"7847b3fe-4b9a-4416-93ec-5c5ecbce7687","Type":"ContainerStarted","Data":"e892d99ad1520c1b641189fc7d7bf8ecb67c2d652581fc6c7b2ad05ed21b0e4f"}
Sep 19 01:21:49.367318 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:21:49.367283    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dnrsw" event={"ID":"7847b3fe-4b9a-4416-93ec-5c5ecbce7687","Type":"ContainerStarted","Data":"94be4fb62d9abe8ea3ff4e8392293eba8bbafe53ff31cf725e61788a5385a8af"}
Sep 19 01:21:49.386331 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:21:49.386285    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dnrsw" podStartSLOduration=2.419070347 podStartE2EDuration="3.386268671s" totalImagesPullingTime="967.198324ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 01:21:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 01:21:49.385311819 +0000 UTC m=+560.272980911" watchObservedRunningTime="2026-09-19 01:21:49.386268671 +0000 UTC m=+560.273937747"
Sep 19 01:21:52.994529 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:21:52.994494    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c9nsz"]
Sep 19 01:21:57.296314 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:21:57.296283    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dnrsw"
Sep 19 01:21:57.296757 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:21:57.296336    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dnrsw"
Sep 19 01:21:57.308741 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:21:57.308722    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dnrsw"
Sep 19 01:21:57.408458 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:21:57.408432    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dnrsw"
Sep 19 01:22:18.792691 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:22:18.792652    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-9j4jn"]
Sep 19 01:22:18.798327 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:22:18.798310    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9j4jn"
Sep 19 01:22:18.801532 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:22:18.801514    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Sep 19 01:22:18.807002 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:22:18.806979    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-9j4jn"]
Sep 19 01:22:18.936498 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:22:18.936461    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4520d6a-eb0f-44ac-9005-b19e6b54fc55-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9j4jn\" (UID: \"e4520d6a-eb0f-44ac-9005-b19e6b54fc55\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9j4jn"
Sep 19 01:22:18.936498 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:22:18.936503    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvpwn\" (UniqueName: \"kubernetes.io/projected/e4520d6a-eb0f-44ac-9005-b19e6b54fc55-kube-api-access-xvpwn\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9j4jn\" (UID: \"e4520d6a-eb0f-44ac-9005-b19e6b54fc55\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9j4jn"
Sep 19 01:22:18.936696 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:22:18.936558    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4520d6a-eb0f-44ac-9005-b19e6b54fc55-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9j4jn\" (UID: \"e4520d6a-eb0f-44ac-9005-b19e6b54fc55\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9j4jn"
Sep 19 01:22:18.936696 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:22:18.936575    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4520d6a-eb0f-44ac-9005-b19e6b54fc55-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9j4jn\" (UID: \"e4520d6a-eb0f-44ac-9005-b19e6b54fc55\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9j4jn"
Sep 19 01:22:18.936696 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:22:18.936591    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4520d6a-eb0f-44ac-9005-b19e6b54fc55-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9j4jn\" (UID: \"e4520d6a-eb0f-44ac-9005-b19e6b54fc55\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9j4jn"
Sep 19 01:22:18.936696 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:22:18.936607    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4520d6a-eb0f-44ac-9005-b19e6b54fc55-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9j4jn\" (UID: \"e4520d6a-eb0f-44ac-9005-b19e6b54fc55\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9j4jn"
Sep 19 01:22:19.037694 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:22:19.037669    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4520d6a-eb0f-44ac-9005-b19e6b54fc55-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9j4jn\" (UID: \"e4520d6a-eb0f-44ac-9005-b19e6b54fc55\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9j4jn"
Sep 19 01:22:19.037694 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:22:19.037694    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4520d6a-eb0f-44ac-9005-b19e6b54fc55-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9j4jn\" (UID: \"e4520d6a-eb0f-44ac-9005-b19e6b54fc55\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9j4jn"
Sep 19 01:22:19.037838 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:22:19.037709    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4520d6a-eb0f-44ac-9005-b19e6b54fc55-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9j4jn\" (UID: \"e4520d6a-eb0f-44ac-9005-b19e6b54fc55\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9j4jn"
Sep 19 01:22:19.037838 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:22:19.037727    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4520d6a-eb0f-44ac-9005-b19e6b54fc55-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9j4jn\" (UID: \"e4520d6a-eb0f-44ac-9005-b19e6b54fc55\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9j4jn"
Sep 19 01:22:19.037838 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:22:19.037762    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4520d6a-eb0f-44ac-9005-b19e6b54fc55-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9j4jn\" (UID: \"e4520d6a-eb0f-44ac-9005-b19e6b54fc55\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9j4jn"
Sep 19 01:22:19.037838 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:22:19.037793    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvpwn\" (UniqueName: \"kubernetes.io/projected/e4520d6a-eb0f-44ac-9005-b19e6b54fc55-kube-api-access-xvpwn\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9j4jn\" (UID: \"e4520d6a-eb0f-44ac-9005-b19e6b54fc55\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9j4jn"
Sep 19 01:22:19.038185 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:22:19.038152    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4520d6a-eb0f-44ac-9005-b19e6b54fc55-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9j4jn\" (UID: \"e4520d6a-eb0f-44ac-9005-b19e6b54fc55\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9j4jn"
Sep 19 01:22:19.038311 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:22:19.038199    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4520d6a-eb0f-44ac-9005-b19e6b54fc55-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9j4jn\" (UID: \"e4520d6a-eb0f-44ac-9005-b19e6b54fc55\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9j4jn"
Sep 19 01:22:19.038311 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:22:19.038243    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4520d6a-eb0f-44ac-9005-b19e6b54fc55-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9j4jn\" (UID: \"e4520d6a-eb0f-44ac-9005-b19e6b54fc55\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9j4jn"
Sep 19 01:22:19.040168 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:22:19.040151    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4520d6a-eb0f-44ac-9005-b19e6b54fc55-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9j4jn\" (UID: \"e4520d6a-eb0f-44ac-9005-b19e6b54fc55\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9j4jn"
Sep 19 01:22:19.040483 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:22:19.040463    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4520d6a-eb0f-44ac-9005-b19e6b54fc55-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9j4jn\" (UID: \"e4520d6a-eb0f-44ac-9005-b19e6b54fc55\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9j4jn"
Sep 19 01:22:19.045809 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:22:19.045746    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvpwn\" (UniqueName: \"kubernetes.io/projected/e4520d6a-eb0f-44ac-9005-b19e6b54fc55-kube-api-access-xvpwn\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9j4jn\" (UID: \"e4520d6a-eb0f-44ac-9005-b19e6b54fc55\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9j4jn"
Sep 19 01:22:19.108197 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:22:19.108179    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9j4jn"
Sep 19 01:22:19.380974 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:22:19.380937    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-9j4jn"]
Sep 19 01:22:19.384201 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:22:19.384177    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4520d6a_eb0f_44ac_9005_b19e6b54fc55.slice/crio-1103e5bab216bbcd550a025b2f4b58fdbf8230332580cda541c1f4e792049351 WatchSource:0}: Error finding container 1103e5bab216bbcd550a025b2f4b58fdbf8230332580cda541c1f4e792049351: Status 404 returned error can't find the container with id 1103e5bab216bbcd550a025b2f4b58fdbf8230332580cda541c1f4e792049351
Sep 19 01:22:19.478521 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:22:19.478495    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9j4jn" event={"ID":"e4520d6a-eb0f-44ac-9005-b19e6b54fc55","Type":"ContainerStarted","Data":"1103e5bab216bbcd550a025b2f4b58fdbf8230332580cda541c1f4e792049351"}
Sep 19 01:22:20.113892 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:22:20.111108    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c9nsz"]
Sep 19 01:22:20.484078 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:22:20.484000    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9j4jn" event={"ID":"e4520d6a-eb0f-44ac-9005-b19e6b54fc55","Type":"ContainerStarted","Data":"9ee190d2efa722ef74226c045e0c611b7532d9e1bf46ca4c6ac69b7877c44fe7"}
Sep 19 01:22:20.516571 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:22:20.516524    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9j4jn" podStartSLOduration=2.342415651 podStartE2EDuration="2.51650873s" totalImagesPullingTime="174.093079ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 01:22:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 01:22:20.515645836 +0000 UTC m=+591.403314934" watchObservedRunningTime="2026-09-19 01:22:20.51650873 +0000 UTC m=+591.404177807"
Sep 19 01:22:29.108651 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:22:29.108609    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9j4jn"
Sep 19 01:22:29.109094 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:22:29.108786    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9j4jn"
Sep 19 01:22:29.121114 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:22:29.121094    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9j4jn"
Sep 19 01:22:29.529552 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:22:29.529478    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9j4jn"
Sep 19 01:22:31.172535 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:22:31.172506    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x785d_f759a12f-5b6f-4924-96f2-dd5e00fc1348/ovn-acl-logging/0.log"
Sep 19 01:22:31.173689 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:22:31.173666    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x785d_f759a12f-5b6f-4924-96f2-dd5e00fc1348/ovn-acl-logging/0.log"
Sep 19 01:22:37.729293 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:22:37.729257    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c9nsz"]
Sep 19 01:22:42.198232 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:22:42.198156    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-hjzvw"]
Sep 19 01:22:42.201786 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:22:42.201765    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hjzvw"
Sep 19 01:22:42.204812 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:22:42.204792    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 19 01:22:42.217294 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:22:42.217274    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-hjzvw"]
Sep 19 01:22:42.299961 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:22:42.299939    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea0f8eff-42f5-468d-a817-bd481182721f-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hjzvw\" (UID: \"ea0f8eff-42f5-468d-a817-bd481182721f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hjzvw"
Sep 19 01:22:42.300048 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:22:42.299972    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea0f8eff-42f5-468d-a817-bd481182721f-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hjzvw\" (UID: \"ea0f8eff-42f5-468d-a817-bd481182721f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hjzvw"
Sep 19 01:22:42.300048 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:22:42.300035    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea0f8eff-42f5-468d-a817-bd481182721f-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hjzvw\" (UID: \"ea0f8eff-42f5-468d-a817-bd481182721f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hjzvw"
Sep 19 01:22:42.300117 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:22:42.300057    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea0f8eff-42f5-468d-a817-bd481182721f-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hjzvw\" (UID: \"ea0f8eff-42f5-468d-a817-bd481182721f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hjzvw"
Sep 19 01:22:42.300117 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:22:42.300084    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea0f8eff-42f5-468d-a817-bd481182721f-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hjzvw\" (UID: \"ea0f8eff-42f5-468d-a817-bd481182721f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hjzvw"
Sep 19 01:22:42.300183 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:22:42.300132    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlkbf\" (UniqueName: \"kubernetes.io/projected/ea0f8eff-42f5-468d-a817-bd481182721f-kube-api-access-vlkbf\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hjzvw\" (UID: \"ea0f8eff-42f5-468d-a817-bd481182721f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hjzvw"
Sep 19 01:22:42.400767 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:22:42.400745    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea0f8eff-42f5-468d-a817-bd481182721f-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hjzvw\" (UID: \"ea0f8eff-42f5-468d-a817-bd481182721f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hjzvw"
Sep 19 01:22:42.400850 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:22:42.400800    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea0f8eff-42f5-468d-a817-bd481182721f-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hjzvw\" (UID: \"ea0f8eff-42f5-468d-a817-bd481182721f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hjzvw"
Sep 19 01:22:42.400850 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:22:42.400825    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea0f8eff-42f5-468d-a817-bd481182721f-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hjzvw\" (UID: \"ea0f8eff-42f5-468d-a817-bd481182721f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hjzvw"
Sep 19 01:22:42.400850 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:22:42.400842    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea0f8eff-42f5-468d-a817-bd481182721f-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hjzvw\" (UID: \"ea0f8eff-42f5-468d-a817-bd481182721f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hjzvw"
Sep 19 01:22:42.401019 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:22:42.400879    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlkbf\" (UniqueName: \"kubernetes.io/projected/ea0f8eff-42f5-468d-a817-bd481182721f-kube-api-access-vlkbf\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hjzvw\" (UID: \"ea0f8eff-42f5-468d-a817-bd481182721f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hjzvw"
Sep 19 01:22:42.401019 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:22:42.400911    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea0f8eff-42f5-468d-a817-bd481182721f-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hjzvw\" (UID: \"ea0f8eff-42f5-468d-a817-bd481182721f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hjzvw"
Sep 19 01:22:42.402361 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:22:42.402340    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea0f8eff-42f5-468d-a817-bd481182721f-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hjzvw\" (UID: \"ea0f8eff-42f5-468d-a817-bd481182721f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hjzvw"
Sep 19 01:22:42.402452 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:22:42.402436    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea0f8eff-42f5-468d-a817-bd481182721f-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hjzvw\" (UID: \"ea0f8eff-42f5-468d-a817-bd481182721f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hjzvw"
Sep 19 01:22:42.403354 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:22:42.403332    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea0f8eff-42f5-468d-a817-bd481182721f-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hjzvw\" (UID: \"ea0f8eff-42f5-468d-a817-bd481182721f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hjzvw"
Sep 19 01:22:42.404111 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:22:42.404093    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea0f8eff-42f5-468d-a817-bd481182721f-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hjzvw\" (UID: \"ea0f8eff-42f5-468d-a817-bd481182721f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hjzvw"
Sep 19 01:22:42.404523 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:22:42.404497    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea0f8eff-42f5-468d-a817-bd481182721f-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hjzvw\" (UID: \"ea0f8eff-42f5-468d-a817-bd481182721f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hjzvw"
Sep 19 01:22:42.410254 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:22:42.410229    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlkbf\" (UniqueName: \"kubernetes.io/projected/ea0f8eff-42f5-468d-a817-bd481182721f-kube-api-access-vlkbf\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hjzvw\" (UID: \"ea0f8eff-42f5-468d-a817-bd481182721f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hjzvw"
Sep 19 01:22:42.512381 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:22:42.512325    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hjzvw"
Sep 19 01:22:42.784786 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:22:42.784718    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-hjzvw"]
Sep 19 01:22:42.787784 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:22:42.787760    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea0f8eff_42f5_468d_a817_bd481182721f.slice/crio-e55ed63ee88c02cf03402ba0d5de4bb0d31e90b755062ac28280ae8b111c6567 WatchSource:0}: Error finding container e55ed63ee88c02cf03402ba0d5de4bb0d31e90b755062ac28280ae8b111c6567: Status 404 returned error can't find the container with id e55ed63ee88c02cf03402ba0d5de4bb0d31e90b755062ac28280ae8b111c6567
Sep 19 01:22:42.789378 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:22:42.789358    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 19 01:22:43.576574 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:22:43.576542    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hjzvw" event={"ID":"ea0f8eff-42f5-468d-a817-bd481182721f","Type":"ContainerStarted","Data":"3b3d20e0179e959fd372168cd525df51fd95534da9a015c08fea2a3af844e4e9"}
Sep 19 01:22:43.576941 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:22:43.576591    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hjzvw" event={"ID":"ea0f8eff-42f5-468d-a817-bd481182721f","Type":"ContainerStarted","Data":"e55ed63ee88c02cf03402ba0d5de4bb0d31e90b755062ac28280ae8b111c6567"}
Sep 19 01:22:43.595995 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:22:43.595951    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hjzvw" podStartSLOduration=1.4324554489999999 podStartE2EDuration="1.595937113s" totalImagesPullingTime="163.481664ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 01:22:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 01:22:43.594840366 +0000 UTC m=+614.482509480" watchObservedRunningTime="2026-09-19 01:22:43.595937113 +0000 UTC m=+614.483606192"
Sep 19 01:22:44.302212 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:22:44.302181    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c9nsz"]
Sep 19 01:22:52.513393 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:22:52.513357    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hjzvw"
Sep 19 01:22:52.514006 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:22:52.513513    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hjzvw"
Sep 19 01:22:52.525573 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:22:52.525553    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hjzvw"
Sep 19 01:22:52.623743 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:22:52.623720    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hjzvw"
Sep 19 01:22:55.276161 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:22:55.276124    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-xsc74"]
Sep 19 01:22:55.279904 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:22:55.279887    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xsc74"
Sep 19 01:22:55.282150 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:22:55.282127    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 19 01:22:55.289804 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:22:55.289778    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-xsc74"]
Sep 19 01:22:55.300238 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:22:55.300217    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/680cad6a-5113-4b67-b0b1-80d04e4697eb-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xsc74\" (UID: \"680cad6a-5113-4b67-b0b1-80d04e4697eb\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xsc74"
Sep 19 01:22:55.300328 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:22:55.300247    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/680cad6a-5113-4b67-b0b1-80d04e4697eb-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xsc74\" (UID: \"680cad6a-5113-4b67-b0b1-80d04e4697eb\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xsc74"
Sep 19 01:22:55.300328 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:22:55.300269    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/680cad6a-5113-4b67-b0b1-80d04e4697eb-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xsc74\" (UID: \"680cad6a-5113-4b67-b0b1-80d04e4697eb\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xsc74"
Sep 19 01:22:55.300328 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:22:55.300292    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/680cad6a-5113-4b67-b0b1-80d04e4697eb-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xsc74\" (UID: \"680cad6a-5113-4b67-b0b1-80d04e4697eb\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xsc74"
Sep 19 01:22:55.300460 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:22:55.300367    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlh8n\" (UniqueName: \"kubernetes.io/projected/680cad6a-5113-4b67-b0b1-80d04e4697eb-kube-api-access-rlh8n\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xsc74\" (UID: \"680cad6a-5113-4b67-b0b1-80d04e4697eb\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xsc74"
Sep 19 01:22:55.300460 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:22:55.300411    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/680cad6a-5113-4b67-b0b1-80d04e4697eb-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xsc74\" (UID: \"680cad6a-5113-4b67-b0b1-80d04e4697eb\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xsc74"
Sep 19 01:22:55.401444 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:22:55.401408    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/680cad6a-5113-4b67-b0b1-80d04e4697eb-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xsc74\" (UID: \"680cad6a-5113-4b67-b0b1-80d04e4697eb\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xsc74"
Sep 19 01:22:55.401608 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:22:55.401449    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/680cad6a-5113-4b67-b0b1-80d04e4697eb-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xsc74\" (UID: \"680cad6a-5113-4b67-b0b1-80d04e4697eb\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xsc74"
Sep 19 01:22:55.401608 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:22:55.401480    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/680cad6a-5113-4b67-b0b1-80d04e4697eb-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xsc74\" (UID: \"680cad6a-5113-4b67-b0b1-80d04e4697eb\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xsc74"
Sep 19 01:22:55.401608 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:22:55.401506    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/680cad6a-5113-4b67-b0b1-80d04e4697eb-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xsc74\" (UID: \"680cad6a-5113-4b67-b0b1-80d04e4697eb\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xsc74"
Sep 19 01:22:55.401608 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:22:55.401550    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlh8n\" (UniqueName: \"kubernetes.io/projected/680cad6a-5113-4b67-b0b1-80d04e4697eb-kube-api-access-rlh8n\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xsc74\" (UID: \"680cad6a-5113-4b67-b0b1-80d04e4697eb\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xsc74"
Sep 19 01:22:55.401608 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:22:55.401597    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/680cad6a-5113-4b67-b0b1-80d04e4697eb-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xsc74\" (UID: \"680cad6a-5113-4b67-b0b1-80d04e4697eb\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xsc74"
Sep 19 01:22:55.401929 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:22:55.401908    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/680cad6a-5113-4b67-b0b1-80d04e4697eb-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xsc74\" (UID: \"680cad6a-5113-4b67-b0b1-80d04e4697eb\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xsc74"
Sep 19 01:22:55.402003 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:22:55.401983    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/680cad6a-5113-4b67-b0b1-80d04e4697eb-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xsc74\" (UID: \"680cad6a-5113-4b67-b0b1-80d04e4697eb\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xsc74"
Sep 19 01:22:55.402058 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:22:55.402027    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/680cad6a-5113-4b67-b0b1-80d04e4697eb-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xsc74\" (UID: \"680cad6a-5113-4b67-b0b1-80d04e4697eb\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xsc74"
Sep 19 01:22:55.403850 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:22:55.403830    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/680cad6a-5113-4b67-b0b1-80d04e4697eb-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xsc74\" (UID: \"680cad6a-5113-4b67-b0b1-80d04e4697eb\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xsc74"
Sep 19 01:22:55.404106 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:22:55.404090    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/680cad6a-5113-4b67-b0b1-80d04e4697eb-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xsc74\" (UID: \"680cad6a-5113-4b67-b0b1-80d04e4697eb\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xsc74"
Sep 19 01:22:55.410018 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:22:55.409996    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlh8n\" (UniqueName: \"kubernetes.io/projected/680cad6a-5113-4b67-b0b1-80d04e4697eb-kube-api-access-rlh8n\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xsc74\" (UID: \"680cad6a-5113-4b67-b0b1-80d04e4697eb\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xsc74"
Sep 19 01:22:55.589413 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:22:55.589347    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xsc74"
Sep 19 01:22:55.859153 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:22:55.859066    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-xsc74"]
Sep 19 01:22:55.861990 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:22:55.861960    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod680cad6a_5113_4b67_b0b1_80d04e4697eb.slice/crio-8063b965a8cb3d0e947bedeb1fff9c304ec6e9a91f440fb47445e7c6eb777ff5 WatchSource:0}: Error finding container 8063b965a8cb3d0e947bedeb1fff9c304ec6e9a91f440fb47445e7c6eb777ff5: Status 404 returned error can't find the container with id 8063b965a8cb3d0e947bedeb1fff9c304ec6e9a91f440fb47445e7c6eb777ff5
Sep 19 01:22:56.630486 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:22:56.630454    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xsc74" event={"ID":"680cad6a-5113-4b67-b0b1-80d04e4697eb","Type":"ContainerStarted","Data":"f864fa9039ad790146e5f140363ddb80031f2ea19e2af572ad5253cb6562b1bc"}
Sep 19 01:22:56.630837 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:22:56.630494    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xsc74" event={"ID":"680cad6a-5113-4b67-b0b1-80d04e4697eb","Type":"ContainerStarted","Data":"8063b965a8cb3d0e947bedeb1fff9c304ec6e9a91f440fb47445e7c6eb777ff5"}
Sep 19 01:22:56.648241 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:22:56.648192    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xsc74" podStartSLOduration=1.470785372 podStartE2EDuration="1.648173525s" totalImagesPullingTime="177.388153ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 01:22:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 01:22:56.647457746 +0000 UTC m=+627.535126822" watchObservedRunningTime="2026-09-19 01:22:56.648173525 +0000 UTC m=+627.535842603"
Sep 19 01:22:57.587982 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:22:57.587948    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c9nsz"]
Sep 19 01:23:05.590360 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:23:05.590320    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xsc74"
Sep 19 01:23:05.590360 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:23:05.590375    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xsc74"
Sep 19 01:23:05.602594 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:23:05.602569    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xsc74"
Sep 19 01:23:05.675350 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:23:05.675322    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xsc74"
Sep 19 01:25:14.951194 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:14.951164    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c9nsz"]
Sep 19 01:25:16.149557 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:16.149528    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-8e708f-75fb8965d6-j4nrt"]
Sep 19 01:25:16.152722 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:16.152705    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-8e708f-75fb8965d6-j4nrt"
Sep 19 01:25:16.155075 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:16.155054    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-8e708f-dockercfg-pn794\""
Sep 19 01:25:16.155210 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:16.155186    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-8e708f\""
Sep 19 01:25:16.172160 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:16.172138    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-8e708f-75fb8965d6-j4nrt"]
Sep 19 01:25:16.248648 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:16.248620    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dkgc\" (UniqueName: \"kubernetes.io/projected/c44fe035-0180-4ec3-b581-4a412e349571-kube-api-access-4dkgc\") pod \"payload-pre-processing-e2e-worker-w0-8e708f-75fb8965d6-j4nrt\" (UID: \"c44fe035-0180-4ec3-b581-4a412e349571\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-8e708f-75fb8965d6-j4nrt"
Sep 19 01:25:16.248762 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:16.248715    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c44fe035-0180-4ec3-b581-4a412e349571-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-8e708f-75fb8965d6-j4nrt\" (UID: \"c44fe035-0180-4ec3-b581-4a412e349571\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-8e708f-75fb8965d6-j4nrt"
Sep 19 01:25:16.349780 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:16.349749    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dkgc\" (UniqueName: \"kubernetes.io/projected/c44fe035-0180-4ec3-b581-4a412e349571-kube-api-access-4dkgc\") pod \"payload-pre-processing-e2e-worker-w0-8e708f-75fb8965d6-j4nrt\" (UID: \"c44fe035-0180-4ec3-b581-4a412e349571\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-8e708f-75fb8965d6-j4nrt"
Sep 19 01:25:16.349926 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:16.349886    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c44fe035-0180-4ec3-b581-4a412e349571-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-8e708f-75fb8965d6-j4nrt\" (UID: \"c44fe035-0180-4ec3-b581-4a412e349571\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-8e708f-75fb8965d6-j4nrt"
Sep 19 01:25:16.350532 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:16.350509    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c44fe035-0180-4ec3-b581-4a412e349571-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-8e708f-75fb8965d6-j4nrt\" (UID: \"c44fe035-0180-4ec3-b581-4a412e349571\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-8e708f-75fb8965d6-j4nrt"
Sep 19 01:25:16.363603 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:16.363580    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dkgc\" (UniqueName: \"kubernetes.io/projected/c44fe035-0180-4ec3-b581-4a412e349571-kube-api-access-4dkgc\") pod \"payload-pre-processing-e2e-worker-w0-8e708f-75fb8965d6-j4nrt\" (UID: \"c44fe035-0180-4ec3-b581-4a412e349571\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-8e708f-75fb8965d6-j4nrt"
Sep 19 01:25:16.462856 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:16.462806    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-8e708f-75fb8965d6-j4nrt"
Sep 19 01:25:16.767042 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:16.766974    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-8e708f-75fb8965d6-j4nrt"]
Sep 19 01:25:16.769992 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:25:16.769966    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc44fe035_0180_4ec3_b581_4a412e349571.slice/crio-a32e02bb55007eb820cb4d91aa3c517581d0c5d6f059ede092df23abd9be9ad9 WatchSource:0}: Error finding container a32e02bb55007eb820cb4d91aa3c517581d0c5d6f059ede092df23abd9be9ad9: Status 404 returned error can't find the container with id a32e02bb55007eb820cb4d91aa3c517581d0c5d6f059ede092df23abd9be9ad9
Sep 19 01:25:17.190799 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:17.190769    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-8e708f-75fb8965d6-j4nrt" event={"ID":"c44fe035-0180-4ec3-b581-4a412e349571","Type":"ContainerStarted","Data":"a32e02bb55007eb820cb4d91aa3c517581d0c5d6f059ede092df23abd9be9ad9"}
Sep 19 01:25:17.466658 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:17.466573    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7515-f9431c"]
Sep 19 01:25:17.469666 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:17.469640    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7515-f9431c"
Sep 19 01:25:17.472357 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:17.472335    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-frkf9\""
Sep 19 01:25:17.478839 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:17.478815    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7515-f9431c"]
Sep 19 01:25:17.558391 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:17.558355    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvt2d\" (UniqueName: \"kubernetes.io/projected/837e92e5-df97-495d-98a1-e12a7bffeae6-kube-api-access-xvt2d\") pod \"test-curl-7515-f9431c\" (UID: \"837e92e5-df97-495d-98a1-e12a7bffeae6\") " pod="opendatahub/test-curl-7515-f9431c"
Sep 19 01:25:17.659299 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:17.659272    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvt2d\" (UniqueName: \"kubernetes.io/projected/837e92e5-df97-495d-98a1-e12a7bffeae6-kube-api-access-xvt2d\") pod \"test-curl-7515-f9431c\" (UID: \"837e92e5-df97-495d-98a1-e12a7bffeae6\") " pod="opendatahub/test-curl-7515-f9431c"
Sep 19 01:25:17.674420 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:17.674394    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvt2d\" (UniqueName: \"kubernetes.io/projected/837e92e5-df97-495d-98a1-e12a7bffeae6-kube-api-access-xvt2d\") pod \"test-curl-7515-f9431c\" (UID: \"837e92e5-df97-495d-98a1-e12a7bffeae6\") " pod="opendatahub/test-curl-7515-f9431c"
Sep 19 01:25:17.782107 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:17.782024    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7515-f9431c"
Sep 19 01:25:18.069096 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:18.069026    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7515-f9431c"]
Sep 19 01:25:18.253549 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:25:18.253519    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod837e92e5_df97_495d_98a1_e12a7bffeae6.slice/crio-84b9e66ec7ca8af92d7d092606b4ee5334e447e685fc9944b30ac3db04295cf3 WatchSource:0}: Error finding container 84b9e66ec7ca8af92d7d092606b4ee5334e447e685fc9944b30ac3db04295cf3: Status 404 returned error can't find the container with id 84b9e66ec7ca8af92d7d092606b4ee5334e447e685fc9944b30ac3db04295cf3
Sep 19 01:25:19.203395 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:19.203352    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-8e708f-75fb8965d6-j4nrt" event={"ID":"c44fe035-0180-4ec3-b581-4a412e349571","Type":"ContainerStarted","Data":"63c20c8888f98b8d6f5102a149aec03c7cb4bae285ff1d349c085927999b8a19"}
Sep 19 01:25:19.203581 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:19.203527    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-8e708f-75fb8965d6-j4nrt"
Sep 19 01:25:19.204601 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:19.204577    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7515-f9431c" event={"ID":"837e92e5-df97-495d-98a1-e12a7bffeae6","Type":"ContainerStarted","Data":"84b9e66ec7ca8af92d7d092606b4ee5334e447e685fc9944b30ac3db04295cf3"}
Sep 19 01:25:19.223495 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:19.223452    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-8e708f-75fb8965d6-j4nrt" podStartSLOduration=1.696716598 podStartE2EDuration="3.223439772s" totalImagesPullingTime="1.526723174s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 01:25:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 01:25:19.221958853 +0000 UTC m=+770.109627950" watchObservedRunningTime="2026-09-19 01:25:19.223439772 +0000 UTC m=+770.111108848"
Sep 19 01:25:25.231275 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:25.231207    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7515-f9431c" event={"ID":"837e92e5-df97-495d-98a1-e12a7bffeae6","Type":"ContainerStarted","Data":"49ead23f1002fdf1826278d804cf78dbc93da236a76e745869e81f43f8bc81d3"}
Sep 19 01:25:26.074059 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:26.074011    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7515-f9431c" podStartSLOduration=2.418774432 podStartE2EDuration="9.073990909s" totalImagesPullingTime="6.655216477s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 01:25:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 01:25:25.289613138 +0000 UTC m=+776.177282214" watchObservedRunningTime="2026-09-19 01:25:26.073990909 +0000 UTC m=+776.961659985"
Sep 19 01:25:26.074943 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:26.074921    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a6519132-5947b7df8c-jfx49"]
Sep 19 01:25:26.078118 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:26.078099    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a6519132-5947b7df8c-jfx49"
Sep 19 01:25:26.080625 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:26.080607    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-a6519132-dockercfg-m8fxd\""
Sep 19 01:25:26.081366 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:26.081350    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-a6519132\""
Sep 19 01:25:26.103740 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:26.103678    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a6519132-5947b7df8c-jfx49"]
Sep 19 01:25:26.104963 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:26.104942    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7515-f9431c"]
Sep 19 01:25:26.113015 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:26.112995    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7515-f9431c"]
Sep 19 01:25:26.128300 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:26.128279    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f1be4d7-6229-4522-a1ac-06c774accf0a-plugins-config-volume\") pod \"payload-processing-e2e-ait-a6519132-5947b7df8c-jfx49\" (UID: \"0f1be4d7-6229-4522-a1ac-06c774accf0a\") " pod="openshift-ingress/payload-processing-e2e-ait-a6519132-5947b7df8c-jfx49"
Sep 19 01:25:26.128422 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:26.128307    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzwbj\" (UniqueName: \"kubernetes.io/projected/0f1be4d7-6229-4522-a1ac-06c774accf0a-kube-api-access-mzwbj\") pod \"payload-processing-e2e-ait-a6519132-5947b7df8c-jfx49\" (UID: \"0f1be4d7-6229-4522-a1ac-06c774accf0a\") " pod="openshift-ingress/payload-processing-e2e-ait-a6519132-5947b7df8c-jfx49"
Sep 19 01:25:26.229096 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:26.229073    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f1be4d7-6229-4522-a1ac-06c774accf0a-plugins-config-volume\") pod \"payload-processing-e2e-ait-a6519132-5947b7df8c-jfx49\" (UID: \"0f1be4d7-6229-4522-a1ac-06c774accf0a\") " pod="openshift-ingress/payload-processing-e2e-ait-a6519132-5947b7df8c-jfx49"
Sep 19 01:25:26.229189 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:26.229101    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzwbj\" (UniqueName: \"kubernetes.io/projected/0f1be4d7-6229-4522-a1ac-06c774accf0a-kube-api-access-mzwbj\") pod \"payload-processing-e2e-ait-a6519132-5947b7df8c-jfx49\" (UID: \"0f1be4d7-6229-4522-a1ac-06c774accf0a\") " pod="openshift-ingress/payload-processing-e2e-ait-a6519132-5947b7df8c-jfx49"
Sep 19 01:25:26.229711 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:26.229692    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f1be4d7-6229-4522-a1ac-06c774accf0a-plugins-config-volume\") pod \"payload-processing-e2e-ait-a6519132-5947b7df8c-jfx49\" (UID: \"0f1be4d7-6229-4522-a1ac-06c774accf0a\") " pod="openshift-ingress/payload-processing-e2e-ait-a6519132-5947b7df8c-jfx49"
Sep 19 01:25:26.235525 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:25:26.235499    2581 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7515-f9431c" container="test-curl-7515-f9431c"
Sep 19 01:25:26.241006 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:26.240987    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzwbj\" (UniqueName: \"kubernetes.io/projected/0f1be4d7-6229-4522-a1ac-06c774accf0a-kube-api-access-mzwbj\") pod \"payload-processing-e2e-ait-a6519132-5947b7df8c-jfx49\" (UID: \"0f1be4d7-6229-4522-a1ac-06c774accf0a\") " pod="openshift-ingress/payload-processing-e2e-ait-a6519132-5947b7df8c-jfx49"
Sep 19 01:25:26.387808 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:26.387741    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a6519132-5947b7df8c-jfx49"
Sep 19 01:25:26.662324 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:26.662297    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a6519132-5947b7df8c-jfx49"]
Sep 19 01:25:26.664781 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:25:26.664756    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f1be4d7_6229_4522_a1ac_06c774accf0a.slice/crio-d166bf57e8111019638382e380168fdb634a87c67056d9c96d9ae20efe0e4ca1 WatchSource:0}: Error finding container d166bf57e8111019638382e380168fdb634a87c67056d9c96d9ae20efe0e4ca1: Status 404 returned error can't find the container with id d166bf57e8111019638382e380168fdb634a87c67056d9c96d9ae20efe0e4ca1
Sep 19 01:25:26.989854 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:26.989787    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-44f561-openshift-default-6f6685d5d-wl7nx"]
Sep 19 01:25:26.990111 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:26.990098    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="837e92e5-df97-495d-98a1-e12a7bffeae6" containerName="test-curl-7515-f9431c"
Sep 19 01:25:26.990152 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:26.990112    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="837e92e5-df97-495d-98a1-e12a7bffeae6" containerName="test-curl-7515-f9431c"
Sep 19 01:25:26.990186 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:26.990169    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="837e92e5-df97-495d-98a1-e12a7bffeae6" containerName="test-curl-7515-f9431c"
Sep 19 01:25:26.993186 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:26.993161    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-44f561-openshift-default-6f6685d5d-wl7nx"
Sep 19 01:25:26.995851 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:26.995828    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-44f561-openshift-default-dockercfg-tjb42\""
Sep 19 01:25:26.995972 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:26.995917    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 19 01:25:27.010013 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:27.009984    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-44f561-openshift-default-6f6685d5d-wl7nx"]
Sep 19 01:25:27.036476 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:27.036453    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01ff1ce5-71ca-4b46-ac86-7274ebd554c5-istio-data\") pod \"e2e-autores-44f561-openshift-default-6f6685d5d-wl7nx\" (UID: \"01ff1ce5-71ca-4b46-ac86-7274ebd554c5\") " pod="openshift-ingress/e2e-autores-44f561-openshift-default-6f6685d5d-wl7nx"
Sep 19 01:25:27.036585 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:27.036486    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01ff1ce5-71ca-4b46-ac86-7274ebd554c5-istio-token\") pod \"e2e-autores-44f561-openshift-default-6f6685d5d-wl7nx\" (UID: \"01ff1ce5-71ca-4b46-ac86-7274ebd554c5\") " pod="openshift-ingress/e2e-autores-44f561-openshift-default-6f6685d5d-wl7nx"
Sep 19 01:25:27.036585 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:27.036518    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01ff1ce5-71ca-4b46-ac86-7274ebd554c5-credential-socket\") pod \"e2e-autores-44f561-openshift-default-6f6685d5d-wl7nx\" (UID: \"01ff1ce5-71ca-4b46-ac86-7274ebd554c5\") " pod="openshift-ingress/e2e-autores-44f561-openshift-default-6f6685d5d-wl7nx"
Sep 19 01:25:27.036585 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:27.036570    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01ff1ce5-71ca-4b46-ac86-7274ebd554c5-workload-certs\") pod \"e2e-autores-44f561-openshift-default-6f6685d5d-wl7nx\" (UID: \"01ff1ce5-71ca-4b46-ac86-7274ebd554c5\") " pod="openshift-ingress/e2e-autores-44f561-openshift-default-6f6685d5d-wl7nx"
Sep 19 01:25:27.036746 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:27.036605    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01ff1ce5-71ca-4b46-ac86-7274ebd554c5-workload-socket\") pod \"e2e-autores-44f561-openshift-default-6f6685d5d-wl7nx\" (UID: \"01ff1ce5-71ca-4b46-ac86-7274ebd554c5\") " pod="openshift-ingress/e2e-autores-44f561-openshift-default-6f6685d5d-wl7nx"
Sep 19 01:25:27.036746 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:27.036626    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01ff1ce5-71ca-4b46-ac86-7274ebd554c5-istiod-ca-cert\") pod \"e2e-autores-44f561-openshift-default-6f6685d5d-wl7nx\" (UID: \"01ff1ce5-71ca-4b46-ac86-7274ebd554c5\") " pod="openshift-ingress/e2e-autores-44f561-openshift-default-6f6685d5d-wl7nx"
Sep 19 01:25:27.036746 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:27.036702    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01ff1ce5-71ca-4b46-ac86-7274ebd554c5-istio-podinfo\") pod \"e2e-autores-44f561-openshift-default-6f6685d5d-wl7nx\" (UID: \"01ff1ce5-71ca-4b46-ac86-7274ebd554c5\") " pod="openshift-ingress/e2e-autores-44f561-openshift-default-6f6685d5d-wl7nx"
Sep 19 01:25:27.036921 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:27.036742    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01ff1ce5-71ca-4b46-ac86-7274ebd554c5-istio-envoy\") pod \"e2e-autores-44f561-openshift-default-6f6685d5d-wl7nx\" (UID: \"01ff1ce5-71ca-4b46-ac86-7274ebd554c5\") " pod="openshift-ingress/e2e-autores-44f561-openshift-default-6f6685d5d-wl7nx"
Sep 19 01:25:27.036921 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:27.036769    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pf99\" (UniqueName: \"kubernetes.io/projected/01ff1ce5-71ca-4b46-ac86-7274ebd554c5-kube-api-access-9pf99\") pod \"e2e-autores-44f561-openshift-default-6f6685d5d-wl7nx\" (UID: \"01ff1ce5-71ca-4b46-ac86-7274ebd554c5\") " pod="openshift-ingress/e2e-autores-44f561-openshift-default-6f6685d5d-wl7nx"
Sep 19 01:25:27.137610 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:27.137586    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01ff1ce5-71ca-4b46-ac86-7274ebd554c5-istio-data\") pod \"e2e-autores-44f561-openshift-default-6f6685d5d-wl7nx\" (UID: \"01ff1ce5-71ca-4b46-ac86-7274ebd554c5\") " pod="openshift-ingress/e2e-autores-44f561-openshift-default-6f6685d5d-wl7nx"
Sep 19 01:25:27.137731 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:27.137614    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01ff1ce5-71ca-4b46-ac86-7274ebd554c5-istio-token\") pod \"e2e-autores-44f561-openshift-default-6f6685d5d-wl7nx\" (UID: \"01ff1ce5-71ca-4b46-ac86-7274ebd554c5\") " pod="openshift-ingress/e2e-autores-44f561-openshift-default-6f6685d5d-wl7nx"
Sep 19 01:25:27.137731 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:27.137648    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01ff1ce5-71ca-4b46-ac86-7274ebd554c5-credential-socket\") pod \"e2e-autores-44f561-openshift-default-6f6685d5d-wl7nx\" (UID: \"01ff1ce5-71ca-4b46-ac86-7274ebd554c5\") " pod="openshift-ingress/e2e-autores-44f561-openshift-default-6f6685d5d-wl7nx"
Sep 19 01:25:27.137731 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:27.137662    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01ff1ce5-71ca-4b46-ac86-7274ebd554c5-workload-certs\") pod \"e2e-autores-44f561-openshift-default-6f6685d5d-wl7nx\" (UID: \"01ff1ce5-71ca-4b46-ac86-7274ebd554c5\") " pod="openshift-ingress/e2e-autores-44f561-openshift-default-6f6685d5d-wl7nx"
Sep 19 01:25:27.137731 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:27.137679    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01ff1ce5-71ca-4b46-ac86-7274ebd554c5-workload-socket\") pod \"e2e-autores-44f561-openshift-default-6f6685d5d-wl7nx\" (UID: \"01ff1ce5-71ca-4b46-ac86-7274ebd554c5\") " pod="openshift-ingress/e2e-autores-44f561-openshift-default-6f6685d5d-wl7nx"
Sep 19 01:25:27.137731 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:27.137698    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01ff1ce5-71ca-4b46-ac86-7274ebd554c5-istiod-ca-cert\") pod \"e2e-autores-44f561-openshift-default-6f6685d5d-wl7nx\" (UID: \"01ff1ce5-71ca-4b46-ac86-7274ebd554c5\") " pod="openshift-ingress/e2e-autores-44f561-openshift-default-6f6685d5d-wl7nx"
Sep 19 01:25:27.138018 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:27.137742    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01ff1ce5-71ca-4b46-ac86-7274ebd554c5-istio-podinfo\") pod \"e2e-autores-44f561-openshift-default-6f6685d5d-wl7nx\" (UID: \"01ff1ce5-71ca-4b46-ac86-7274ebd554c5\") " pod="openshift-ingress/e2e-autores-44f561-openshift-default-6f6685d5d-wl7nx"
Sep 19 01:25:27.138018 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:27.137770    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01ff1ce5-71ca-4b46-ac86-7274ebd554c5-istio-envoy\") pod \"e2e-autores-44f561-openshift-default-6f6685d5d-wl7nx\" (UID: \"01ff1ce5-71ca-4b46-ac86-7274ebd554c5\") " pod="openshift-ingress/e2e-autores-44f561-openshift-default-6f6685d5d-wl7nx"
Sep 19 01:25:27.138018 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:27.137805    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pf99\" (UniqueName: \"kubernetes.io/projected/01ff1ce5-71ca-4b46-ac86-7274ebd554c5-kube-api-access-9pf99\") pod \"e2e-autores-44f561-openshift-default-6f6685d5d-wl7nx\" (UID: \"01ff1ce5-71ca-4b46-ac86-7274ebd554c5\") " pod="openshift-ingress/e2e-autores-44f561-openshift-default-6f6685d5d-wl7nx"
Sep 19 01:25:27.138165 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:27.138037    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01ff1ce5-71ca-4b46-ac86-7274ebd554c5-credential-socket\") pod \"e2e-autores-44f561-openshift-default-6f6685d5d-wl7nx\" (UID: \"01ff1ce5-71ca-4b46-ac86-7274ebd554c5\") " pod="openshift-ingress/e2e-autores-44f561-openshift-default-6f6685d5d-wl7nx"
Sep 19 01:25:27.138165 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:27.138053    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01ff1ce5-71ca-4b46-ac86-7274ebd554c5-istio-data\") pod \"e2e-autores-44f561-openshift-default-6f6685d5d-wl7nx\" (UID: \"01ff1ce5-71ca-4b46-ac86-7274ebd554c5\") " pod="openshift-ingress/e2e-autores-44f561-openshift-default-6f6685d5d-wl7nx"
Sep 19 01:25:27.138165 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:27.138120    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01ff1ce5-71ca-4b46-ac86-7274ebd554c5-workload-socket\") pod \"e2e-autores-44f561-openshift-default-6f6685d5d-wl7nx\" (UID: \"01ff1ce5-71ca-4b46-ac86-7274ebd554c5\") " pod="openshift-ingress/e2e-autores-44f561-openshift-default-6f6685d5d-wl7nx"
Sep 19 01:25:27.138323 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:27.138261    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01ff1ce5-71ca-4b46-ac86-7274ebd554c5-workload-certs\") pod \"e2e-autores-44f561-openshift-default-6f6685d5d-wl7nx\" (UID: \"01ff1ce5-71ca-4b46-ac86-7274ebd554c5\") " pod="openshift-ingress/e2e-autores-44f561-openshift-default-6f6685d5d-wl7nx"
Sep 19 01:25:27.138635 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:27.138611    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01ff1ce5-71ca-4b46-ac86-7274ebd554c5-istiod-ca-cert\") pod \"e2e-autores-44f561-openshift-default-6f6685d5d-wl7nx\" (UID: \"01ff1ce5-71ca-4b46-ac86-7274ebd554c5\") " pod="openshift-ingress/e2e-autores-44f561-openshift-default-6f6685d5d-wl7nx"
Sep 19 01:25:27.140397 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:27.140377    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01ff1ce5-71ca-4b46-ac86-7274ebd554c5-istio-envoy\") pod \"e2e-autores-44f561-openshift-default-6f6685d5d-wl7nx\" (UID: \"01ff1ce5-71ca-4b46-ac86-7274ebd554c5\") " pod="openshift-ingress/e2e-autores-44f561-openshift-default-6f6685d5d-wl7nx"
Sep 19 01:25:27.140502 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:27.140487    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01ff1ce5-71ca-4b46-ac86-7274ebd554c5-istio-podinfo\") pod \"e2e-autores-44f561-openshift-default-6f6685d5d-wl7nx\" (UID: \"01ff1ce5-71ca-4b46-ac86-7274ebd554c5\") " pod="openshift-ingress/e2e-autores-44f561-openshift-default-6f6685d5d-wl7nx"
Sep 19 01:25:27.153774 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:27.153752    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01ff1ce5-71ca-4b46-ac86-7274ebd554c5-istio-token\") pod \"e2e-autores-44f561-openshift-default-6f6685d5d-wl7nx\" (UID: \"01ff1ce5-71ca-4b46-ac86-7274ebd554c5\") " pod="openshift-ingress/e2e-autores-44f561-openshift-default-6f6685d5d-wl7nx"
Sep 19 01:25:27.154175 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:27.154158    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pf99\" (UniqueName: \"kubernetes.io/projected/01ff1ce5-71ca-4b46-ac86-7274ebd554c5-kube-api-access-9pf99\") pod \"e2e-autores-44f561-openshift-default-6f6685d5d-wl7nx\" (UID: \"01ff1ce5-71ca-4b46-ac86-7274ebd554c5\") " pod="openshift-ingress/e2e-autores-44f561-openshift-default-6f6685d5d-wl7nx"
Sep 19 01:25:27.241288 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:27.241198    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a6519132-5947b7df8c-jfx49" event={"ID":"0f1be4d7-6229-4522-a1ac-06c774accf0a","Type":"ContainerStarted","Data":"4eb3c035b6afa71d07daf7eb05ad3a882162a9efd810c59a29ca469b0461e71d"}
Sep 19 01:25:27.241288 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:27.241257    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a6519132-5947b7df8c-jfx49" event={"ID":"0f1be4d7-6229-4522-a1ac-06c774accf0a","Type":"ContainerStarted","Data":"d166bf57e8111019638382e380168fdb634a87c67056d9c96d9ae20efe0e4ca1"}
Sep 19 01:25:27.241720 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:27.241415    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-a6519132-5947b7df8c-jfx49"
Sep 19 01:25:27.241720 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:27.241570    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7515-f9431c" podUID="837e92e5-df97-495d-98a1-e12a7bffeae6" containerName="test-curl-7515-f9431c" containerID="cri-o://49ead23f1002fdf1826278d804cf78dbc93da236a76e745869e81f43f8bc81d3" gracePeriod=2
Sep 19 01:25:27.260560 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:27.260520    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-a6519132-5947b7df8c-jfx49" podStartSLOduration=1.2605025570000001 podStartE2EDuration="1.260502557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 01:25:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 01:25:27.258706191 +0000 UTC m=+778.146375286" watchObservedRunningTime="2026-09-19 01:25:27.260502557 +0000 UTC m=+778.148171659"
Sep 19 01:25:27.260684 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:25:27.260577    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7515-f9431c\" is forbidden: User \"system:node:ip-10-0-129-113.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-113.ec2.internal' and this object" podUID="837e92e5-df97-495d-98a1-e12a7bffeae6" pod="opendatahub/test-curl-7515-f9431c"
Sep 19 01:25:27.305340 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:27.305319    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-44f561-openshift-default-6f6685d5d-wl7nx"
Sep 19 01:25:27.589472 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:27.589397    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-44f561-openshift-default-6f6685d5d-wl7nx"]
Sep 19 01:25:27.592956 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:25:27.592908    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01ff1ce5_71ca_4b46_ac86_7274ebd554c5.slice/crio-aa6d467d1edf717972f467bec144b6567c7ede00e049748608cf6543e43a6e2a WatchSource:0}: Error finding container aa6d467d1edf717972f467bec144b6567c7ede00e049748608cf6543e43a6e2a: Status 404 returned error can't find the container with id aa6d467d1edf717972f467bec144b6567c7ede00e049748608cf6543e43a6e2a
Sep 19 01:25:28.248216 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:28.248172    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-44f561-openshift-default-6f6685d5d-wl7nx" event={"ID":"01ff1ce5-71ca-4b46-ac86-7274ebd554c5","Type":"ContainerStarted","Data":"aa6d467d1edf717972f467bec144b6567c7ede00e049748608cf6543e43a6e2a"}
Sep 19 01:25:29.430708 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:29.430682    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7515-f9431c"
Sep 19 01:25:29.458721 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:29.458692    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xvt2d\" (UniqueName: \"kubernetes.io/projected/837e92e5-df97-495d-98a1-e12a7bffeae6-kube-api-access-xvt2d\") pod \"837e92e5-df97-495d-98a1-e12a7bffeae6\" (UID: \"837e92e5-df97-495d-98a1-e12a7bffeae6\") "
Sep 19 01:25:29.461206 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:29.461173    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/837e92e5-df97-495d-98a1-e12a7bffeae6-kube-api-access-xvt2d" (OuterVolumeSpecName: "kube-api-access-xvt2d") pod "837e92e5-df97-495d-98a1-e12a7bffeae6" (UID: "837e92e5-df97-495d-98a1-e12a7bffeae6"). InnerVolumeSpecName "kube-api-access-xvt2d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 01:25:29.559614 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:29.559537    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvt2d\" (UniqueName: \"kubernetes.io/projected/837e92e5-df97-495d-98a1-e12a7bffeae6-kube-api-access-xvt2d\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:25:30.211234 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:30.211199    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-8e708f-75fb8965d6-j4nrt"
Sep 19 01:25:30.260172 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:30.260137    2581 generic.go:356] "Generic (PLEG): container finished" podID="837e92e5-df97-495d-98a1-e12a7bffeae6" containerID="49ead23f1002fdf1826278d804cf78dbc93da236a76e745869e81f43f8bc81d3" exitCode=137
Sep 19 01:25:30.260340 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:30.260249    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7515-f9431c"
Sep 19 01:25:30.260340 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:30.260263    2581 scope.go:117] "RemoveContainer" containerID="49ead23f1002fdf1826278d804cf78dbc93da236a76e745869e81f43f8bc81d3"
Sep 19 01:25:30.273357 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:25:30.273323    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7515-f9431c\" is forbidden: User \"system:node:ip-10-0-129-113.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-113.ec2.internal' and this object" podUID="837e92e5-df97-495d-98a1-e12a7bffeae6" pod="opendatahub/test-curl-7515-f9431c"
Sep 19 01:25:30.432097 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:30.432078    2581 scope.go:117] "RemoveContainer" containerID="49ead23f1002fdf1826278d804cf78dbc93da236a76e745869e81f43f8bc81d3"
Sep 19 01:25:30.432420 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:25:30.432401    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49ead23f1002fdf1826278d804cf78dbc93da236a76e745869e81f43f8bc81d3\": container with ID starting with 49ead23f1002fdf1826278d804cf78dbc93da236a76e745869e81f43f8bc81d3 not found: ID does not exist" containerID="49ead23f1002fdf1826278d804cf78dbc93da236a76e745869e81f43f8bc81d3"
Sep 19 01:25:30.432468 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:30.432432    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49ead23f1002fdf1826278d804cf78dbc93da236a76e745869e81f43f8bc81d3"} err="failed to get container status \"49ead23f1002fdf1826278d804cf78dbc93da236a76e745869e81f43f8bc81d3\": rpc error: code = NotFound desc = could not find container \"49ead23f1002fdf1826278d804cf78dbc93da236a76e745869e81f43f8bc81d3\": container with ID starting with 49ead23f1002fdf1826278d804cf78dbc93da236a76e745869e81f43f8bc81d3 not found: ID does not exist"
Sep 19 01:25:30.481299 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:30.481235    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c9nsz"]
Sep 19 01:25:30.490109 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:30.490077    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 19 01:25:30.490187 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:30.490139    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 19 01:25:30.490187 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:30.490165    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 19 01:25:31.265536 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:31.265495    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-44f561-openshift-default-6f6685d5d-wl7nx" event={"ID":"01ff1ce5-71ca-4b46-ac86-7274ebd554c5","Type":"ContainerStarted","Data":"88ac8c99cf8de48ac4e1d27b282c98cef42a62e9e59f82ebb941b5c19f383f61"}
Sep 19 01:25:31.267353 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:25:31.267323    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7515-f9431c\" is forbidden: User \"system:node:ip-10-0-129-113.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-113.ec2.internal' and this object" podUID="837e92e5-df97-495d-98a1-e12a7bffeae6" pod="opendatahub/test-curl-7515-f9431c"
Sep 19 01:25:31.271179 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:31.271153    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="837e92e5-df97-495d-98a1-e12a7bffeae6" path="/var/lib/kubelet/pods/837e92e5-df97-495d-98a1-e12a7bffeae6/volumes"
Sep 19 01:25:31.295712 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:31.295660    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-44f561-openshift-default-6f6685d5d-wl7nx" podStartSLOduration=2.400977436 podStartE2EDuration="5.295641822s" totalImagesPullingTime="2.894664386s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 01:25:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 01:25:31.29485613 +0000 UTC m=+782.182525257" watchObservedRunningTime="2026-09-19 01:25:31.295641822 +0000 UTC m=+782.183310900"
Sep 19 01:25:31.305558 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:31.305539    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-44f561-openshift-default-6f6685d5d-wl7nx"
Sep 19 01:25:31.310517 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:31.310490    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-44f561-openshift-default-6f6685d5d-wl7nx"
Sep 19 01:25:32.270552 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:32.270523    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-44f561-openshift-default-6f6685d5d-wl7nx"
Sep 19 01:25:32.271430 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:32.271405    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-44f561-openshift-default-6f6685d5d-wl7nx"
Sep 19 01:25:37.517054 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:37.517020    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-44f561-6c866756fc-kvlmv"]
Sep 19 01:25:37.521602 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:37.521585    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-44f561-6c866756fc-kvlmv"
Sep 19 01:25:37.524028 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:37.524009    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-44f561\""
Sep 19 01:25:37.524121 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:37.524109    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-44f561-dockercfg-85j7d\""
Sep 19 01:25:37.534890 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:37.534845    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-44f561-6c866756fc-kvlmv"]
Sep 19 01:25:37.569100 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:37.569078    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-44f561-768b76957d-975fv"]
Sep 19 01:25:37.572249 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:37.572235    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-44f561-768b76957d-975fv"
Sep 19 01:25:37.584563 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:37.584545    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-44f561-768b76957d-975fv"]
Sep 19 01:25:37.628142 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:37.628112    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/caf3e8d0-a8b8-40c3-bb8f-565d371660ca-plugins-config-volume\") pod \"payload-processing-e2e-autores-44f561-6c866756fc-kvlmv\" (UID: \"caf3e8d0-a8b8-40c3-bb8f-565d371660ca\") " pod="openshift-ingress/payload-processing-e2e-autores-44f561-6c866756fc-kvlmv"
Sep 19 01:25:37.628244 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:37.628157    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/856b0a82-2bcf-43a7-82d9-fef3b1b1769b-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-44f561-768b76957d-975fv\" (UID: \"856b0a82-2bcf-43a7-82d9-fef3b1b1769b\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-44f561-768b76957d-975fv"
Sep 19 01:25:37.628294 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:37.628274    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkn9z\" (UniqueName: \"kubernetes.io/projected/caf3e8d0-a8b8-40c3-bb8f-565d371660ca-kube-api-access-pkn9z\") pod \"payload-processing-e2e-autores-44f561-6c866756fc-kvlmv\" (UID: \"caf3e8d0-a8b8-40c3-bb8f-565d371660ca\") " pod="openshift-ingress/payload-processing-e2e-autores-44f561-6c866756fc-kvlmv"
Sep 19 01:25:37.628340 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:37.628302    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcbxm\" (UniqueName: \"kubernetes.io/projected/856b0a82-2bcf-43a7-82d9-fef3b1b1769b-kube-api-access-kcbxm\") pod \"payload-pre-processing-e2e-autores-44f561-768b76957d-975fv\" (UID: \"856b0a82-2bcf-43a7-82d9-fef3b1b1769b\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-44f561-768b76957d-975fv"
Sep 19 01:25:37.728672 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:37.728641    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkn9z\" (UniqueName: \"kubernetes.io/projected/caf3e8d0-a8b8-40c3-bb8f-565d371660ca-kube-api-access-pkn9z\") pod \"payload-processing-e2e-autores-44f561-6c866756fc-kvlmv\" (UID: \"caf3e8d0-a8b8-40c3-bb8f-565d371660ca\") " pod="openshift-ingress/payload-processing-e2e-autores-44f561-6c866756fc-kvlmv"
Sep 19 01:25:37.728814 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:37.728686    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcbxm\" (UniqueName: \"kubernetes.io/projected/856b0a82-2bcf-43a7-82d9-fef3b1b1769b-kube-api-access-kcbxm\") pod \"payload-pre-processing-e2e-autores-44f561-768b76957d-975fv\" (UID: \"856b0a82-2bcf-43a7-82d9-fef3b1b1769b\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-44f561-768b76957d-975fv"
Sep 19 01:25:37.728814 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:37.728738    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/caf3e8d0-a8b8-40c3-bb8f-565d371660ca-plugins-config-volume\") pod \"payload-processing-e2e-autores-44f561-6c866756fc-kvlmv\" (UID: \"caf3e8d0-a8b8-40c3-bb8f-565d371660ca\") " pod="openshift-ingress/payload-processing-e2e-autores-44f561-6c866756fc-kvlmv"
Sep 19 01:25:37.728814 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:37.728775    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/856b0a82-2bcf-43a7-82d9-fef3b1b1769b-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-44f561-768b76957d-975fv\" (UID: \"856b0a82-2bcf-43a7-82d9-fef3b1b1769b\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-44f561-768b76957d-975fv"
Sep 19 01:25:37.729484 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:37.729461    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/caf3e8d0-a8b8-40c3-bb8f-565d371660ca-plugins-config-volume\") pod \"payload-processing-e2e-autores-44f561-6c866756fc-kvlmv\" (UID: \"caf3e8d0-a8b8-40c3-bb8f-565d371660ca\") " pod="openshift-ingress/payload-processing-e2e-autores-44f561-6c866756fc-kvlmv"
Sep 19 01:25:37.729553 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:37.729480    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/856b0a82-2bcf-43a7-82d9-fef3b1b1769b-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-44f561-768b76957d-975fv\" (UID: \"856b0a82-2bcf-43a7-82d9-fef3b1b1769b\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-44f561-768b76957d-975fv"
Sep 19 01:25:37.739700 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:37.739681    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcbxm\" (UniqueName: \"kubernetes.io/projected/856b0a82-2bcf-43a7-82d9-fef3b1b1769b-kube-api-access-kcbxm\") pod \"payload-pre-processing-e2e-autores-44f561-768b76957d-975fv\" (UID: \"856b0a82-2bcf-43a7-82d9-fef3b1b1769b\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-44f561-768b76957d-975fv"
Sep 19 01:25:37.739782 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:37.739758    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkn9z\" (UniqueName: \"kubernetes.io/projected/caf3e8d0-a8b8-40c3-bb8f-565d371660ca-kube-api-access-pkn9z\") pod \"payload-processing-e2e-autores-44f561-6c866756fc-kvlmv\" (UID: \"caf3e8d0-a8b8-40c3-bb8f-565d371660ca\") " pod="openshift-ingress/payload-processing-e2e-autores-44f561-6c866756fc-kvlmv"
Sep 19 01:25:37.831464 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:37.831410    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-44f561-6c866756fc-kvlmv"
Sep 19 01:25:37.881692 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:37.881662    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-44f561-768b76957d-975fv"
Sep 19 01:25:38.156313 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:38.156287    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-44f561-6c866756fc-kvlmv"]
Sep 19 01:25:38.157913 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:25:38.157883    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaf3e8d0_a8b8_40c3_bb8f_565d371660ca.slice/crio-6d3ecb0548acfc2ed422b625127657f7ed329c77a1257154276e698c91394e6e WatchSource:0}: Error finding container 6d3ecb0548acfc2ed422b625127657f7ed329c77a1257154276e698c91394e6e: Status 404 returned error can't find the container with id 6d3ecb0548acfc2ed422b625127657f7ed329c77a1257154276e698c91394e6e
Sep 19 01:25:38.211411 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:38.211387    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-44f561-768b76957d-975fv"]
Sep 19 01:25:38.213071 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:25:38.213026    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod856b0a82_2bcf_43a7_82d9_fef3b1b1769b.slice/crio-a9e29d31dbf57c41757159904a0d4fcf151cacdb12c8ad328762118d48047f05 WatchSource:0}: Error finding container a9e29d31dbf57c41757159904a0d4fcf151cacdb12c8ad328762118d48047f05: Status 404 returned error can't find the container with id a9e29d31dbf57c41757159904a0d4fcf151cacdb12c8ad328762118d48047f05
Sep 19 01:25:38.250014 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:38.249994    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-a6519132-5947b7df8c-jfx49"
Sep 19 01:25:38.297845 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:38.297812    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-44f561-768b76957d-975fv" event={"ID":"856b0a82-2bcf-43a7-82d9-fef3b1b1769b","Type":"ContainerStarted","Data":"c9e51d69ef35ec1cf53bd737f3ae47a22a1322f0240b13625decb339ef061632"}
Sep 19 01:25:38.298009 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:38.297896    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-44f561-768b76957d-975fv"
Sep 19 01:25:38.298009 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:38.297916    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-44f561-768b76957d-975fv" event={"ID":"856b0a82-2bcf-43a7-82d9-fef3b1b1769b","Type":"ContainerStarted","Data":"a9e29d31dbf57c41757159904a0d4fcf151cacdb12c8ad328762118d48047f05"}
Sep 19 01:25:38.299771 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:38.299748    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-44f561-6c866756fc-kvlmv" event={"ID":"caf3e8d0-a8b8-40c3-bb8f-565d371660ca","Type":"ContainerStarted","Data":"435909db250765e55c86022bb28a3f08c9a5ab59dc986953102c70d9732c584a"}
Sep 19 01:25:38.299909 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:38.299791    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-44f561-6c866756fc-kvlmv" event={"ID":"caf3e8d0-a8b8-40c3-bb8f-565d371660ca","Type":"ContainerStarted","Data":"6d3ecb0548acfc2ed422b625127657f7ed329c77a1257154276e698c91394e6e"}
Sep 19 01:25:38.299909 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:38.299822    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-44f561-6c866756fc-kvlmv"
Sep 19 01:25:38.400781 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:38.400736    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-44f561-768b76957d-975fv" podStartSLOduration=1.400721546 podStartE2EDuration="1.400721546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 01:25:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 01:25:38.372659407 +0000 UTC m=+789.260328484" watchObservedRunningTime="2026-09-19 01:25:38.400721546 +0000 UTC m=+789.288390687"
Sep 19 01:25:38.402269 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:38.402225    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-44f561-6c866756fc-kvlmv" podStartSLOduration=1.402210832 podStartE2EDuration="1.402210832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 01:25:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 01:25:38.399994085 +0000 UTC m=+789.287663185" watchObservedRunningTime="2026-09-19 01:25:38.402210832 +0000 UTC m=+789.289879978"
Sep 19 01:25:38.533299 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:38.533227    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a6519132-rwjvk"]
Sep 19 01:25:38.536493 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:38.536476    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a6519132-rwjvk"
Sep 19 01:25:38.538897 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:38.538877    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s7pvt\""
Sep 19 01:25:38.538978 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:38.538887    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 19 01:25:38.546428 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:38.546403    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a6519132-rwjvk"]
Sep 19 01:25:38.635993 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:38.635968    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-5a005d-openshift-default-cc9788f6b-twz5d"]
Sep 19 01:25:38.636296 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:38.636278    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/236b33a6-056a-420f-8f4b-e87b10c2092f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a6519132-rwjvk\" (UID: \"236b33a6-056a-420f-8f4b-e87b10c2092f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a6519132-rwjvk"
Sep 19 01:25:38.639306 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:38.639293    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5a005d-openshift-default-cc9788f6b-twz5d"
Sep 19 01:25:38.641906 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:38.641884    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-5a005d-openshift-default-dockercfg-ptzqs\""
Sep 19 01:25:38.656514 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:38.656492    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-5a005d-openshift-default-cc9788f6b-twz5d"]
Sep 19 01:25:38.737466 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:38.737439    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2302d85f-6dba-4f71-bc69-6c64172f0f93-workload-certs\") pod \"e2e-inf-b-5a005d-openshift-default-cc9788f6b-twz5d\" (UID: \"2302d85f-6dba-4f71-bc69-6c64172f0f93\") " pod="openshift-ingress/e2e-inf-b-5a005d-openshift-default-cc9788f6b-twz5d"
Sep 19 01:25:38.737585 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:38.737476    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/236b33a6-056a-420f-8f4b-e87b10c2092f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a6519132-rwjvk\" (UID: \"236b33a6-056a-420f-8f4b-e87b10c2092f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a6519132-rwjvk"
Sep 19 01:25:38.737585 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:38.737498    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2302d85f-6dba-4f71-bc69-6c64172f0f93-istiod-ca-cert\") pod \"e2e-inf-b-5a005d-openshift-default-cc9788f6b-twz5d\" (UID: \"2302d85f-6dba-4f71-bc69-6c64172f0f93\") " pod="openshift-ingress/e2e-inf-b-5a005d-openshift-default-cc9788f6b-twz5d"
Sep 19 01:25:38.737585 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:38.737513    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2302d85f-6dba-4f71-bc69-6c64172f0f93-istio-envoy\") pod \"e2e-inf-b-5a005d-openshift-default-cc9788f6b-twz5d\" (UID: \"2302d85f-6dba-4f71-bc69-6c64172f0f93\") " pod="openshift-ingress/e2e-inf-b-5a005d-openshift-default-cc9788f6b-twz5d"
Sep 19 01:25:38.737759 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:38.737582    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2302d85f-6dba-4f71-bc69-6c64172f0f93-credential-socket\") pod \"e2e-inf-b-5a005d-openshift-default-cc9788f6b-twz5d\" (UID: \"2302d85f-6dba-4f71-bc69-6c64172f0f93\") " pod="openshift-ingress/e2e-inf-b-5a005d-openshift-default-cc9788f6b-twz5d"
Sep 19 01:25:38.737759 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:38.737652    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2302d85f-6dba-4f71-bc69-6c64172f0f93-istio-data\") pod \"e2e-inf-b-5a005d-openshift-default-cc9788f6b-twz5d\" (UID: \"2302d85f-6dba-4f71-bc69-6c64172f0f93\") " pod="openshift-ingress/e2e-inf-b-5a005d-openshift-default-cc9788f6b-twz5d"
Sep 19 01:25:38.737759 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:38.737689    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2302d85f-6dba-4f71-bc69-6c64172f0f93-istio-token\") pod \"e2e-inf-b-5a005d-openshift-default-cc9788f6b-twz5d\" (UID: \"2302d85f-6dba-4f71-bc69-6c64172f0f93\") " pod="openshift-ingress/e2e-inf-b-5a005d-openshift-default-cc9788f6b-twz5d"
Sep 19 01:25:38.737759 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:38.737742    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxlh4\" (UniqueName: \"kubernetes.io/projected/2302d85f-6dba-4f71-bc69-6c64172f0f93-kube-api-access-bxlh4\") pod \"e2e-inf-b-5a005d-openshift-default-cc9788f6b-twz5d\" (UID: \"2302d85f-6dba-4f71-bc69-6c64172f0f93\") " pod="openshift-ingress/e2e-inf-b-5a005d-openshift-default-cc9788f6b-twz5d"
Sep 19 01:25:38.737958 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:38.737786    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2302d85f-6dba-4f71-bc69-6c64172f0f93-istio-podinfo\") pod \"e2e-inf-b-5a005d-openshift-default-cc9788f6b-twz5d\" (UID: \"2302d85f-6dba-4f71-bc69-6c64172f0f93\") " pod="openshift-ingress/e2e-inf-b-5a005d-openshift-default-cc9788f6b-twz5d"
Sep 19 01:25:38.737958 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:38.737840    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2302d85f-6dba-4f71-bc69-6c64172f0f93-workload-socket\") pod \"e2e-inf-b-5a005d-openshift-default-cc9788f6b-twz5d\" (UID: \"2302d85f-6dba-4f71-bc69-6c64172f0f93\") " pod="openshift-ingress/e2e-inf-b-5a005d-openshift-default-cc9788f6b-twz5d"
Sep 19 01:25:38.738113 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:38.738096    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/236b33a6-056a-420f-8f4b-e87b10c2092f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a6519132-rwjvk\" (UID: \"236b33a6-056a-420f-8f4b-e87b10c2092f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a6519132-rwjvk"
Sep 19 01:25:38.838995 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:38.838938    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2302d85f-6dba-4f71-bc69-6c64172f0f93-istio-data\") pod \"e2e-inf-b-5a005d-openshift-default-cc9788f6b-twz5d\" (UID: \"2302d85f-6dba-4f71-bc69-6c64172f0f93\") " pod="openshift-ingress/e2e-inf-b-5a005d-openshift-default-cc9788f6b-twz5d"
Sep 19 01:25:38.838995 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:38.838964    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2302d85f-6dba-4f71-bc69-6c64172f0f93-istio-token\") pod \"e2e-inf-b-5a005d-openshift-default-cc9788f6b-twz5d\" (UID: \"2302d85f-6dba-4f71-bc69-6c64172f0f93\") " pod="openshift-ingress/e2e-inf-b-5a005d-openshift-default-cc9788f6b-twz5d"
Sep 19 01:25:38.838995 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:38.838984    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxlh4\" (UniqueName: \"kubernetes.io/projected/2302d85f-6dba-4f71-bc69-6c64172f0f93-kube-api-access-bxlh4\") pod \"e2e-inf-b-5a005d-openshift-default-cc9788f6b-twz5d\" (UID: \"2302d85f-6dba-4f71-bc69-6c64172f0f93\") " pod="openshift-ingress/e2e-inf-b-5a005d-openshift-default-cc9788f6b-twz5d"
Sep 19 01:25:38.839211 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:38.839008    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2302d85f-6dba-4f71-bc69-6c64172f0f93-istio-podinfo\") pod \"e2e-inf-b-5a005d-openshift-default-cc9788f6b-twz5d\" (UID: \"2302d85f-6dba-4f71-bc69-6c64172f0f93\") " pod="openshift-ingress/e2e-inf-b-5a005d-openshift-default-cc9788f6b-twz5d"
Sep 19 01:25:38.839211 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:38.839050    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2302d85f-6dba-4f71-bc69-6c64172f0f93-workload-socket\") pod \"e2e-inf-b-5a005d-openshift-default-cc9788f6b-twz5d\" (UID: \"2302d85f-6dba-4f71-bc69-6c64172f0f93\") " pod="openshift-ingress/e2e-inf-b-5a005d-openshift-default-cc9788f6b-twz5d"
Sep 19 01:25:38.839211 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:38.839099    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2302d85f-6dba-4f71-bc69-6c64172f0f93-workload-certs\") pod \"e2e-inf-b-5a005d-openshift-default-cc9788f6b-twz5d\" (UID: \"2302d85f-6dba-4f71-bc69-6c64172f0f93\") " pod="openshift-ingress/e2e-inf-b-5a005d-openshift-default-cc9788f6b-twz5d"
Sep 19 01:25:38.839211 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:38.839147    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2302d85f-6dba-4f71-bc69-6c64172f0f93-istiod-ca-cert\") pod \"e2e-inf-b-5a005d-openshift-default-cc9788f6b-twz5d\" (UID: \"2302d85f-6dba-4f71-bc69-6c64172f0f93\") " pod="openshift-ingress/e2e-inf-b-5a005d-openshift-default-cc9788f6b-twz5d"
Sep 19 01:25:38.839211 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:38.839176    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2302d85f-6dba-4f71-bc69-6c64172f0f93-istio-envoy\") pod \"e2e-inf-b-5a005d-openshift-default-cc9788f6b-twz5d\" (UID: \"2302d85f-6dba-4f71-bc69-6c64172f0f93\") " pod="openshift-ingress/e2e-inf-b-5a005d-openshift-default-cc9788f6b-twz5d"
Sep 19 01:25:38.839469 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:38.839214    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2302d85f-6dba-4f71-bc69-6c64172f0f93-credential-socket\") pod \"e2e-inf-b-5a005d-openshift-default-cc9788f6b-twz5d\" (UID: \"2302d85f-6dba-4f71-bc69-6c64172f0f93\") " pod="openshift-ingress/e2e-inf-b-5a005d-openshift-default-cc9788f6b-twz5d"
Sep 19 01:25:38.839469 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:38.839332    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2302d85f-6dba-4f71-bc69-6c64172f0f93-istio-data\") pod \"e2e-inf-b-5a005d-openshift-default-cc9788f6b-twz5d\" (UID: \"2302d85f-6dba-4f71-bc69-6c64172f0f93\") " pod="openshift-ingress/e2e-inf-b-5a005d-openshift-default-cc9788f6b-twz5d"
Sep 19 01:25:38.839469 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:38.839406    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2302d85f-6dba-4f71-bc69-6c64172f0f93-workload-socket\") pod \"e2e-inf-b-5a005d-openshift-default-cc9788f6b-twz5d\" (UID: \"2302d85f-6dba-4f71-bc69-6c64172f0f93\") " pod="openshift-ingress/e2e-inf-b-5a005d-openshift-default-cc9788f6b-twz5d"
Sep 19 01:25:38.839653 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:38.839632    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2302d85f-6dba-4f71-bc69-6c64172f0f93-credential-socket\") pod \"e2e-inf-b-5a005d-openshift-default-cc9788f6b-twz5d\" (UID: \"2302d85f-6dba-4f71-bc69-6c64172f0f93\") " pod="openshift-ingress/e2e-inf-b-5a005d-openshift-default-cc9788f6b-twz5d"
Sep 19 01:25:38.839713 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:38.839660    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2302d85f-6dba-4f71-bc69-6c64172f0f93-workload-certs\") pod \"e2e-inf-b-5a005d-openshift-default-cc9788f6b-twz5d\" (UID: \"2302d85f-6dba-4f71-bc69-6c64172f0f93\") " pod="openshift-ingress/e2e-inf-b-5a005d-openshift-default-cc9788f6b-twz5d"
Sep 19 01:25:38.839879 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:38.839847    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2302d85f-6dba-4f71-bc69-6c64172f0f93-istiod-ca-cert\") pod \"e2e-inf-b-5a005d-openshift-default-cc9788f6b-twz5d\" (UID: \"2302d85f-6dba-4f71-bc69-6c64172f0f93\") " pod="openshift-ingress/e2e-inf-b-5a005d-openshift-default-cc9788f6b-twz5d"
Sep 19 01:25:38.841543 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:38.841520    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2302d85f-6dba-4f71-bc69-6c64172f0f93-istio-podinfo\") pod \"e2e-inf-b-5a005d-openshift-default-cc9788f6b-twz5d\" (UID: \"2302d85f-6dba-4f71-bc69-6c64172f0f93\") " pod="openshift-ingress/e2e-inf-b-5a005d-openshift-default-cc9788f6b-twz5d"
Sep 19 01:25:38.841666 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:38.841650    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2302d85f-6dba-4f71-bc69-6c64172f0f93-istio-envoy\") pod \"e2e-inf-b-5a005d-openshift-default-cc9788f6b-twz5d\" (UID: \"2302d85f-6dba-4f71-bc69-6c64172f0f93\") " pod="openshift-ingress/e2e-inf-b-5a005d-openshift-default-cc9788f6b-twz5d"
Sep 19 01:25:38.847444 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:38.847427    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a6519132-rwjvk"
Sep 19 01:25:38.850030 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:38.849997    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxlh4\" (UniqueName: \"kubernetes.io/projected/2302d85f-6dba-4f71-bc69-6c64172f0f93-kube-api-access-bxlh4\") pod \"e2e-inf-b-5a005d-openshift-default-cc9788f6b-twz5d\" (UID: \"2302d85f-6dba-4f71-bc69-6c64172f0f93\") " pod="openshift-ingress/e2e-inf-b-5a005d-openshift-default-cc9788f6b-twz5d"
Sep 19 01:25:38.850223 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:38.850197    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2302d85f-6dba-4f71-bc69-6c64172f0f93-istio-token\") pod \"e2e-inf-b-5a005d-openshift-default-cc9788f6b-twz5d\" (UID: \"2302d85f-6dba-4f71-bc69-6c64172f0f93\") " pod="openshift-ingress/e2e-inf-b-5a005d-openshift-default-cc9788f6b-twz5d"
Sep 19 01:25:38.956448 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:38.956419    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5a005d-openshift-default-cc9788f6b-twz5d"
Sep 19 01:25:39.131611 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:39.131574    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a6519132-rwjvk"]
Sep 19 01:25:39.135322 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:25:39.135299    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod236b33a6_056a_420f_8f4b_e87b10c2092f.slice/crio-d3b31d4919aa7cdb452cf36d4723f933a2269fbabc032fdff2ebd71508f6b140 WatchSource:0}: Error finding container d3b31d4919aa7cdb452cf36d4723f933a2269fbabc032fdff2ebd71508f6b140: Status 404 returned error can't find the container with id d3b31d4919aa7cdb452cf36d4723f933a2269fbabc032fdff2ebd71508f6b140
Sep 19 01:25:39.253739 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:39.253716    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-5a005d-openshift-default-cc9788f6b-twz5d"]
Sep 19 01:25:39.255594 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:25:39.255568    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2302d85f_6dba_4f71_bc69_6c64172f0f93.slice/crio-04db7ad36bc10ea97a77818b18e3f6f7e4420067a099e765c64ae4945c9cbd99 WatchSource:0}: Error finding container 04db7ad36bc10ea97a77818b18e3f6f7e4420067a099e765c64ae4945c9cbd99: Status 404 returned error can't find the container with id 04db7ad36bc10ea97a77818b18e3f6f7e4420067a099e765c64ae4945c9cbd99
Sep 19 01:25:39.257673 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:39.257643    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 19 01:25:39.257735 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:39.257706    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 19 01:25:39.257771 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:39.257732    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 19 01:25:39.306366 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:39.306339    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a6519132-rwjvk" event={"ID":"236b33a6-056a-420f-8f4b-e87b10c2092f","Type":"ContainerStarted","Data":"d3b31d4919aa7cdb452cf36d4723f933a2269fbabc032fdff2ebd71508f6b140"}
Sep 19 01:25:39.307597 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:39.307571    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5a005d-openshift-default-cc9788f6b-twz5d" event={"ID":"2302d85f-6dba-4f71-bc69-6c64172f0f93","Type":"ContainerStarted","Data":"04db7ad36bc10ea97a77818b18e3f6f7e4420067a099e765c64ae4945c9cbd99"}
Sep 19 01:25:40.316419 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:40.316377    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5a005d-openshift-default-cc9788f6b-twz5d" event={"ID":"2302d85f-6dba-4f71-bc69-6c64172f0f93","Type":"ContainerStarted","Data":"c8acbd4b6408ab58b32f26ce0478f8b58e17a2f5a39eb764f0bfaa7d9a1df10f"}
Sep 19 01:25:40.956987 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:40.956898    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-5a005d-openshift-default-cc9788f6b-twz5d"
Sep 19 01:25:40.963223 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:40.963199    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-5a005d-openshift-default-cc9788f6b-twz5d"
Sep 19 01:25:41.009249 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:41.009205    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-5a005d-openshift-default-cc9788f6b-twz5d" podStartSLOduration=3.009189311 podStartE2EDuration="3.009189311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 01:25:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 01:25:40.369273543 +0000 UTC m=+791.256942642" watchObservedRunningTime="2026-09-19 01:25:41.009189311 +0000 UTC m=+791.896858389"
Sep 19 01:25:41.321179 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:41.321105    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-5a005d-openshift-default-cc9788f6b-twz5d"
Sep 19 01:25:41.322277 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:41.322253    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-5a005d-openshift-default-cc9788f6b-twz5d"
Sep 19 01:25:42.326328 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:42.326249    2581 generic.go:356] "Generic (PLEG): container finished" podID="236b33a6-056a-420f-8f4b-e87b10c2092f" containerID="1cc3a09dc21a2928826cbfd30e5700db72143dd1d3290ba908d78cc4d560678d" exitCode=0
Sep 19 01:25:42.326632 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:42.326340    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a6519132-rwjvk" event={"ID":"236b33a6-056a-420f-8f4b-e87b10c2092f","Type":"ContainerDied","Data":"1cc3a09dc21a2928826cbfd30e5700db72143dd1d3290ba908d78cc4d560678d"}
Sep 19 01:25:43.460652 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:43.460627    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a6519132-rwjvk"
Sep 19 01:25:43.584012 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:43.583943    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/236b33a6-056a-420f-8f4b-e87b10c2092f-maas-api-service-ca\") pod \"236b33a6-056a-420f-8f4b-e87b10c2092f\" (UID: \"236b33a6-056a-420f-8f4b-e87b10c2092f\") "
Sep 19 01:25:43.584310 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:43.584292    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/236b33a6-056a-420f-8f4b-e87b10c2092f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "236b33a6-056a-420f-8f4b-e87b10c2092f" (UID: "236b33a6-056a-420f-8f4b-e87b10c2092f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 01:25:43.684850 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:43.684811    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/236b33a6-056a-420f-8f4b-e87b10c2092f-maas-api-service-ca\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:25:44.335771 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:44.335734    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a6519132-rwjvk" event={"ID":"236b33a6-056a-420f-8f4b-e87b10c2092f","Type":"ContainerDied","Data":"d3b31d4919aa7cdb452cf36d4723f933a2269fbabc032fdff2ebd71508f6b140"}
Sep 19 01:25:44.335771 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:44.335754    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a6519132-rwjvk"
Sep 19 01:25:44.335771 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:44.335782    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d3b31d4919aa7cdb452cf36d4723f933a2269fbabc032fdff2ebd71508f6b140"
Sep 19 01:25:48.710431 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:48.710400    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5a005d-756b74fb85-bmnpx"]
Sep 19 01:25:48.710855 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:48.710775    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="236b33a6-056a-420f-8f4b-e87b10c2092f" containerName="revoke-keys"
Sep 19 01:25:48.710855 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:48.710787    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="236b33a6-056a-420f-8f4b-e87b10c2092f" containerName="revoke-keys"
Sep 19 01:25:48.710855 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:48.710853    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="236b33a6-056a-420f-8f4b-e87b10c2092f" containerName="revoke-keys"
Sep 19 01:25:48.714131 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:48.714107    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5a005d-756b74fb85-bmnpx"
Sep 19 01:25:48.716843 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:48.716821    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-5a005d-dockercfg-rp2pt\""
Sep 19 01:25:48.717037 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:48.717022    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-5a005d\""
Sep 19 01:25:48.728316 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:48.728287    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5a005d-756b74fb85-bmnpx"]
Sep 19 01:25:48.832705 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:48.832669    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b03a0fdb-7a5d-44a8-bf5c-88fcfd06b734-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-5a005d-756b74fb85-bmnpx\" (UID: \"b03a0fdb-7a5d-44a8-bf5c-88fcfd06b734\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5a005d-756b74fb85-bmnpx"
Sep 19 01:25:48.832881 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:48.832740    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbcx9\" (UniqueName: \"kubernetes.io/projected/b03a0fdb-7a5d-44a8-bf5c-88fcfd06b734-kube-api-access-hbcx9\") pod \"payload-processing-e2e-inf-b-5a005d-756b74fb85-bmnpx\" (UID: \"b03a0fdb-7a5d-44a8-bf5c-88fcfd06b734\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5a005d-756b74fb85-bmnpx"
Sep 19 01:25:48.934279 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:48.934238    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b03a0fdb-7a5d-44a8-bf5c-88fcfd06b734-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-5a005d-756b74fb85-bmnpx\" (UID: \"b03a0fdb-7a5d-44a8-bf5c-88fcfd06b734\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5a005d-756b74fb85-bmnpx"
Sep 19 01:25:48.934456 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:48.934326    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbcx9\" (UniqueName: \"kubernetes.io/projected/b03a0fdb-7a5d-44a8-bf5c-88fcfd06b734-kube-api-access-hbcx9\") pod \"payload-processing-e2e-inf-b-5a005d-756b74fb85-bmnpx\" (UID: \"b03a0fdb-7a5d-44a8-bf5c-88fcfd06b734\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5a005d-756b74fb85-bmnpx"
Sep 19 01:25:48.934995 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:48.934976    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b03a0fdb-7a5d-44a8-bf5c-88fcfd06b734-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-5a005d-756b74fb85-bmnpx\" (UID: \"b03a0fdb-7a5d-44a8-bf5c-88fcfd06b734\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5a005d-756b74fb85-bmnpx"
Sep 19 01:25:48.944379 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:48.944354    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbcx9\" (UniqueName: \"kubernetes.io/projected/b03a0fdb-7a5d-44a8-bf5c-88fcfd06b734-kube-api-access-hbcx9\") pod \"payload-processing-e2e-inf-b-5a005d-756b74fb85-bmnpx\" (UID: \"b03a0fdb-7a5d-44a8-bf5c-88fcfd06b734\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5a005d-756b74fb85-bmnpx"
Sep 19 01:25:49.024641 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:49.024564    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5a005d-756b74fb85-bmnpx"
Sep 19 01:25:49.309988 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:49.309919    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-44f561-6c866756fc-kvlmv"
Sep 19 01:25:49.310142 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:49.309996    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-44f561-768b76957d-975fv"
Sep 19 01:25:49.313341 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:49.313309    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5a005d-756b74fb85-bmnpx"]
Sep 19 01:25:49.316363 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:25:49.316341    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb03a0fdb_7a5d_44a8_bf5c_88fcfd06b734.slice/crio-68a2fa37a094c55d6edf0fd425d02dd160581951b115646ba6eaacffa557e0b1 WatchSource:0}: Error finding container 68a2fa37a094c55d6edf0fd425d02dd160581951b115646ba6eaacffa557e0b1: Status 404 returned error can't find the container with id 68a2fa37a094c55d6edf0fd425d02dd160581951b115646ba6eaacffa557e0b1
Sep 19 01:25:49.359630 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:49.359604    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5a005d-756b74fb85-bmnpx" event={"ID":"b03a0fdb-7a5d-44a8-bf5c-88fcfd06b734","Type":"ContainerStarted","Data":"68a2fa37a094c55d6edf0fd425d02dd160581951b115646ba6eaacffa557e0b1"}
Sep 19 01:25:50.366424 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:50.366384    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5a005d-756b74fb85-bmnpx" event={"ID":"b03a0fdb-7a5d-44a8-bf5c-88fcfd06b734","Type":"ContainerStarted","Data":"b3cf9ed3612b47fa9996297fcd700e747a52656ca3fcbf70553da46af123ff43"}
Sep 19 01:25:50.366951 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:50.366475    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-5a005d-756b74fb85-bmnpx"
Sep 19 01:25:50.385077 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:50.385023    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-5a005d-756b74fb85-bmnpx" podStartSLOduration=2.385006226 podStartE2EDuration="2.385006226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 01:25:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 01:25:50.383775435 +0000 UTC m=+801.271444540" watchObservedRunningTime="2026-09-19 01:25:50.385006226 +0000 UTC m=+801.272675304"
Sep 19 01:25:52.272110 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:52.272078    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a6519132-5947b7df8c-jfx49"]
Sep 19 01:25:52.272578 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:52.272322    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-a6519132-5947b7df8c-jfx49" podUID="0f1be4d7-6229-4522-a1ac-06c774accf0a" containerName="payload-processing" containerID="cri-o://4eb3c035b6afa71d07daf7eb05ad3a882162a9efd810c59a29ca469b0461e71d" gracePeriod=30
Sep 19 01:25:52.456420 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:52.456389    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a6519132-rwjvk"]
Sep 19 01:25:52.460469 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:52.460443    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a6519132-rwjvk"]
Sep 19 01:25:52.522203 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:52.522143    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a6519132-5947b7df8c-jfx49"
Sep 19 01:25:52.568601 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:52.568572    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mzwbj\" (UniqueName: \"kubernetes.io/projected/0f1be4d7-6229-4522-a1ac-06c774accf0a-kube-api-access-mzwbj\") pod \"0f1be4d7-6229-4522-a1ac-06c774accf0a\" (UID: \"0f1be4d7-6229-4522-a1ac-06c774accf0a\") "
Sep 19 01:25:52.568755 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:52.568684    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f1be4d7-6229-4522-a1ac-06c774accf0a-plugins-config-volume\") pod \"0f1be4d7-6229-4522-a1ac-06c774accf0a\" (UID: \"0f1be4d7-6229-4522-a1ac-06c774accf0a\") "
Sep 19 01:25:52.569099 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:52.569073    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f1be4d7-6229-4522-a1ac-06c774accf0a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0f1be4d7-6229-4522-a1ac-06c774accf0a" (UID: "0f1be4d7-6229-4522-a1ac-06c774accf0a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 01:25:52.571023 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:52.571001    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f1be4d7-6229-4522-a1ac-06c774accf0a-kube-api-access-mzwbj" (OuterVolumeSpecName: "kube-api-access-mzwbj") pod "0f1be4d7-6229-4522-a1ac-06c774accf0a" (UID: "0f1be4d7-6229-4522-a1ac-06c774accf0a"). InnerVolumeSpecName "kube-api-access-mzwbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 01:25:52.669868 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:52.669837    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzwbj\" (UniqueName: \"kubernetes.io/projected/0f1be4d7-6229-4522-a1ac-06c774accf0a-kube-api-access-mzwbj\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:25:52.670012 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:52.669894    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f1be4d7-6229-4522-a1ac-06c774accf0a-plugins-config-volume\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:25:53.272123 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:53.272041    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="236b33a6-056a-420f-8f4b-e87b10c2092f" path="/var/lib/kubelet/pods/236b33a6-056a-420f-8f4b-e87b10c2092f/volumes"
Sep 19 01:25:53.381464 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:53.381434    2581 generic.go:356] "Generic (PLEG): container finished" podID="0f1be4d7-6229-4522-a1ac-06c774accf0a" containerID="4eb3c035b6afa71d07daf7eb05ad3a882162a9efd810c59a29ca469b0461e71d" exitCode=0
Sep 19 01:25:53.381819 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:53.381482    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a6519132-5947b7df8c-jfx49"
Sep 19 01:25:53.381819 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:53.381512    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a6519132-5947b7df8c-jfx49" event={"ID":"0f1be4d7-6229-4522-a1ac-06c774accf0a","Type":"ContainerDied","Data":"4eb3c035b6afa71d07daf7eb05ad3a882162a9efd810c59a29ca469b0461e71d"}
Sep 19 01:25:53.381819 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:53.381553    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a6519132-5947b7df8c-jfx49" event={"ID":"0f1be4d7-6229-4522-a1ac-06c774accf0a","Type":"ContainerDied","Data":"d166bf57e8111019638382e380168fdb634a87c67056d9c96d9ae20efe0e4ca1"}
Sep 19 01:25:53.381819 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:53.381581    2581 scope.go:117] "RemoveContainer" containerID="4eb3c035b6afa71d07daf7eb05ad3a882162a9efd810c59a29ca469b0461e71d"
Sep 19 01:25:53.383336 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:53.383305    2581 generic.go:356] "Generic (PLEG): container finished" podID="01ff1ce5-71ca-4b46-ac86-7274ebd554c5" containerID="88ac8c99cf8de48ac4e1d27b282c98cef42a62e9e59f82ebb941b5c19f383f61" exitCode=0
Sep 19 01:25:53.383464 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:53.383372    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-44f561-openshift-default-6f6685d5d-wl7nx" event={"ID":"01ff1ce5-71ca-4b46-ac86-7274ebd554c5","Type":"ContainerDied","Data":"88ac8c99cf8de48ac4e1d27b282c98cef42a62e9e59f82ebb941b5c19f383f61"}
Sep 19 01:25:53.383769 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:53.383751    2581 scope.go:117] "RemoveContainer" containerID="88ac8c99cf8de48ac4e1d27b282c98cef42a62e9e59f82ebb941b5c19f383f61"
Sep 19 01:25:53.384797 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:53.384772    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 19 01:25:53.384916 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:53.384898    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 19 01:25:53.384989 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:53.384955    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 19 01:25:53.390229 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:53.390211    2581 scope.go:117] "RemoveContainer" containerID="4eb3c035b6afa71d07daf7eb05ad3a882162a9efd810c59a29ca469b0461e71d"
Sep 19 01:25:53.390495 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:25:53.390477    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4eb3c035b6afa71d07daf7eb05ad3a882162a9efd810c59a29ca469b0461e71d\": container with ID starting with 4eb3c035b6afa71d07daf7eb05ad3a882162a9efd810c59a29ca469b0461e71d not found: ID does not exist" containerID="4eb3c035b6afa71d07daf7eb05ad3a882162a9efd810c59a29ca469b0461e71d"
Sep 19 01:25:53.390559 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:53.390502    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4eb3c035b6afa71d07daf7eb05ad3a882162a9efd810c59a29ca469b0461e71d"} err="failed to get container status \"4eb3c035b6afa71d07daf7eb05ad3a882162a9efd810c59a29ca469b0461e71d\": rpc error: code = NotFound desc = could not find container \"4eb3c035b6afa71d07daf7eb05ad3a882162a9efd810c59a29ca469b0461e71d\": container with ID starting with 4eb3c035b6afa71d07daf7eb05ad3a882162a9efd810c59a29ca469b0461e71d not found: ID does not exist"
Sep 19 01:25:53.404194 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:53.404167    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a6519132-5947b7df8c-jfx49"]
Sep 19 01:25:53.408133 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:53.408113    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a6519132-5947b7df8c-jfx49"]
Sep 19 01:25:54.393935 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:54.393076    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-44f561-openshift-default-6f6685d5d-wl7nx" event={"ID":"01ff1ce5-71ca-4b46-ac86-7274ebd554c5","Type":"ContainerStarted","Data":"42be5ff584124e7a16b1c09331e74c07717429bb00aaa8a67b679f50852fa091"}
Sep 19 01:25:54.435745 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:54.435714    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c9nsz"]
Sep 19 01:25:55.272262 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:55.272230    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f1be4d7-6229-4522-a1ac-06c774accf0a" path="/var/lib/kubelet/pods/0f1be4d7-6229-4522-a1ac-06c774accf0a/volumes"
Sep 19 01:25:55.306087 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:55.306061    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-44f561-openshift-default-6f6685d5d-wl7nx"
Sep 19 01:25:55.306249 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:55.306232    2581 patch_prober.go:28] interesting pod/e2e-autores-44f561-openshift-default-6f6685d5d-wl7nx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body=
Sep 19 01:25:55.306297 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:55.306280    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-44f561-openshift-default-6f6685d5d-wl7nx" podUID="01ff1ce5-71ca-4b46-ac86-7274ebd554c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused"
Sep 19 01:25:56.306627 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:56.306585    2581 patch_prober.go:28] interesting pod/e2e-autores-44f561-openshift-default-6f6685d5d-wl7nx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body=
Sep 19 01:25:56.307264 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:56.307232    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-44f561-openshift-default-6f6685d5d-wl7nx" podUID="01ff1ce5-71ca-4b46-ac86-7274ebd554c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused"
Sep 19 01:25:57.305712 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:57.305675    2581 patch_prober.go:28] interesting pod/e2e-autores-44f561-openshift-default-6f6685d5d-wl7nx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body=
Sep 19 01:25:57.306029 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:57.305999    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-44f561-openshift-default-6f6685d5d-wl7nx" podUID="01ff1ce5-71ca-4b46-ac86-7274ebd554c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused"
Sep 19 01:25:58.306311 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:58.306257    2581 patch_prober.go:28] interesting pod/e2e-autores-44f561-openshift-default-6f6685d5d-wl7nx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body=
Sep 19 01:25:58.306840 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:58.306336    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-44f561-openshift-default-6f6685d5d-wl7nx" podUID="01ff1ce5-71ca-4b46-ac86-7274ebd554c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused"
Sep 19 01:25:59.305984 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:59.305937    2581 patch_prober.go:28] interesting pod/e2e-autores-44f561-openshift-default-6f6685d5d-wl7nx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body=
Sep 19 01:25:59.306168 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:25:59.306011    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-44f561-openshift-default-6f6685d5d-wl7nx" podUID="01ff1ce5-71ca-4b46-ac86-7274ebd554c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused"
Sep 19 01:26:00.310826 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:00.310797    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-44f561-openshift-default-6f6685d5d-wl7nx"
Sep 19 01:26:00.311285 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:00.311093    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-44f561-openshift-default-6f6685d5d-wl7nx"
Sep 19 01:26:00.311674 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:00.311655    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-44f561-openshift-default-6f6685d5d-wl7nx"
Sep 19 01:26:00.587905 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:00.587780    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f7939b7a-fffd54866-xlsbg"]
Sep 19 01:26:00.588727 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:00.588700    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f1be4d7-6229-4522-a1ac-06c774accf0a" containerName="payload-processing"
Sep 19 01:26:00.588930 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:00.588917    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f1be4d7-6229-4522-a1ac-06c774accf0a" containerName="payload-processing"
Sep 19 01:26:00.589138 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:00.589125    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f1be4d7-6229-4522-a1ac-06c774accf0a" containerName="payload-processing"
Sep 19 01:26:00.597610 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:00.597578    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f7939b7a-fffd54866-xlsbg"
Sep 19 01:26:00.605648 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:00.604533    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-f7939b7a\""
Sep 19 01:26:00.605648 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:00.604740    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-f7939b7a-dockercfg-th4dq\""
Sep 19 01:26:00.607723 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:00.607698    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f7939b7a-fffd54866-xlsbg"]
Sep 19 01:26:00.618034 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:00.618011    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f7939b7a-58fd768898-cplfg"]
Sep 19 01:26:00.621766 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:00.621743    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-f7939b7a-58fd768898-cplfg"
Sep 19 01:26:00.630724 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:00.630702    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f7939b7a-58fd768898-cplfg"]
Sep 19 01:26:00.650576 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:00.650550    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1758c9b-ec1c-445f-ae3e-3f675ec5daae-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-f7939b7a-fffd54866-xlsbg\" (UID: \"e1758c9b-ec1c-445f-ae3e-3f675ec5daae\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f7939b7a-fffd54866-xlsbg"
Sep 19 01:26:00.650696 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:00.650615    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8wgc\" (UniqueName: \"kubernetes.io/projected/e1758c9b-ec1c-445f-ae3e-3f675ec5daae-kube-api-access-c8wgc\") pod \"payload-pre-processing-e2e-migrate-f7939b7a-fffd54866-xlsbg\" (UID: \"e1758c9b-ec1c-445f-ae3e-3f675ec5daae\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f7939b7a-fffd54866-xlsbg"
Sep 19 01:26:00.752022 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:00.751982    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac07e351-cae8-4583-8e92-380a629eb6aa-plugins-config-volume\") pod \"payload-processing-e2e-migrate-f7939b7a-58fd768898-cplfg\" (UID: \"ac07e351-cae8-4583-8e92-380a629eb6aa\") " pod="openshift-ingress/payload-processing-e2e-migrate-f7939b7a-58fd768898-cplfg"
Sep 19 01:26:00.752022 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:00.752029    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1758c9b-ec1c-445f-ae3e-3f675ec5daae-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-f7939b7a-fffd54866-xlsbg\" (UID: \"e1758c9b-ec1c-445f-ae3e-3f675ec5daae\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f7939b7a-fffd54866-xlsbg"
Sep 19 01:26:00.752279 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:00.752069    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6g57d\" (UniqueName: \"kubernetes.io/projected/ac07e351-cae8-4583-8e92-380a629eb6aa-kube-api-access-6g57d\") pod \"payload-processing-e2e-migrate-f7939b7a-58fd768898-cplfg\" (UID: \"ac07e351-cae8-4583-8e92-380a629eb6aa\") " pod="openshift-ingress/payload-processing-e2e-migrate-f7939b7a-58fd768898-cplfg"
Sep 19 01:26:00.752279 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:00.752112    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8wgc\" (UniqueName: \"kubernetes.io/projected/e1758c9b-ec1c-445f-ae3e-3f675ec5daae-kube-api-access-c8wgc\") pod \"payload-pre-processing-e2e-migrate-f7939b7a-fffd54866-xlsbg\" (UID: \"e1758c9b-ec1c-445f-ae3e-3f675ec5daae\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f7939b7a-fffd54866-xlsbg"
Sep 19 01:26:00.752697 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:00.752674    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1758c9b-ec1c-445f-ae3e-3f675ec5daae-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-f7939b7a-fffd54866-xlsbg\" (UID: \"e1758c9b-ec1c-445f-ae3e-3f675ec5daae\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f7939b7a-fffd54866-xlsbg"
Sep 19 01:26:00.760567 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:00.760546    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8wgc\" (UniqueName: \"kubernetes.io/projected/e1758c9b-ec1c-445f-ae3e-3f675ec5daae-kube-api-access-c8wgc\") pod \"payload-pre-processing-e2e-migrate-f7939b7a-fffd54866-xlsbg\" (UID: \"e1758c9b-ec1c-445f-ae3e-3f675ec5daae\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f7939b7a-fffd54866-xlsbg"
Sep 19 01:26:00.853490 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:00.853458    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6g57d\" (UniqueName: \"kubernetes.io/projected/ac07e351-cae8-4583-8e92-380a629eb6aa-kube-api-access-6g57d\") pod \"payload-processing-e2e-migrate-f7939b7a-58fd768898-cplfg\" (UID: \"ac07e351-cae8-4583-8e92-380a629eb6aa\") " pod="openshift-ingress/payload-processing-e2e-migrate-f7939b7a-58fd768898-cplfg"
Sep 19 01:26:00.853696 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:00.853581    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac07e351-cae8-4583-8e92-380a629eb6aa-plugins-config-volume\") pod \"payload-processing-e2e-migrate-f7939b7a-58fd768898-cplfg\" (UID: \"ac07e351-cae8-4583-8e92-380a629eb6aa\") " pod="openshift-ingress/payload-processing-e2e-migrate-f7939b7a-58fd768898-cplfg"
Sep 19 01:26:00.854300 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:00.854276    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac07e351-cae8-4583-8e92-380a629eb6aa-plugins-config-volume\") pod \"payload-processing-e2e-migrate-f7939b7a-58fd768898-cplfg\" (UID: \"ac07e351-cae8-4583-8e92-380a629eb6aa\") " pod="openshift-ingress/payload-processing-e2e-migrate-f7939b7a-58fd768898-cplfg"
Sep 19 01:26:00.861671 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:00.861644    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6g57d\" (UniqueName: \"kubernetes.io/projected/ac07e351-cae8-4583-8e92-380a629eb6aa-kube-api-access-6g57d\") pod \"payload-processing-e2e-migrate-f7939b7a-58fd768898-cplfg\" (UID: \"ac07e351-cae8-4583-8e92-380a629eb6aa\") " pod="openshift-ingress/payload-processing-e2e-migrate-f7939b7a-58fd768898-cplfg"
Sep 19 01:26:00.915682 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:00.915651    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f7939b7a-fffd54866-xlsbg"
Sep 19 01:26:00.931479 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:00.931452    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-f7939b7a-58fd768898-cplfg"
Sep 19 01:26:01.196179 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:01.196100    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f7939b7a-fffd54866-xlsbg"]
Sep 19 01:26:01.200450 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:26:01.200408    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1758c9b_ec1c_445f_ae3e_3f675ec5daae.slice/crio-f44ed763d45dfa5da2e4ffeab1d74d67eff4ac956b3fa25d13a7eedae8e26a1c WatchSource:0}: Error finding container f44ed763d45dfa5da2e4ffeab1d74d67eff4ac956b3fa25d13a7eedae8e26a1c: Status 404 returned error can't find the container with id f44ed763d45dfa5da2e4ffeab1d74d67eff4ac956b3fa25d13a7eedae8e26a1c
Sep 19 01:26:01.221190 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:01.221166    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f7939b7a-58fd768898-cplfg"]
Sep 19 01:26:01.235777 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:26:01.235751    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac07e351_cae8_4583_8e92_380a629eb6aa.slice/crio-423970c480f237ba0437c3b1f5f09246bda1c36b656b0cad509d37511025aa0b WatchSource:0}: Error finding container 423970c480f237ba0437c3b1f5f09246bda1c36b656b0cad509d37511025aa0b: Status 404 returned error can't find the container with id 423970c480f237ba0437c3b1f5f09246bda1c36b656b0cad509d37511025aa0b
Sep 19 01:26:01.373655 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:01.373624    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-5a005d-756b74fb85-bmnpx"
Sep 19 01:26:01.442445 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:01.442162    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-f7939b7a-58fd768898-cplfg" event={"ID":"ac07e351-cae8-4583-8e92-380a629eb6aa","Type":"ContainerStarted","Data":"c5abdf5cefb1373ebdab9d08ea2f241e8750bd085d8e942d27b68f9e76021342"}
Sep 19 01:26:01.442445 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:01.442228    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-f7939b7a-58fd768898-cplfg" event={"ID":"ac07e351-cae8-4583-8e92-380a629eb6aa","Type":"ContainerStarted","Data":"423970c480f237ba0437c3b1f5f09246bda1c36b656b0cad509d37511025aa0b"}
Sep 19 01:26:01.442445 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:01.442256    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-f7939b7a-58fd768898-cplfg"
Sep 19 01:26:01.444114 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:01.444083    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f7939b7a-fffd54866-xlsbg" event={"ID":"e1758c9b-ec1c-445f-ae3e-3f675ec5daae","Type":"ContainerStarted","Data":"2eed11657cf9e14eb00297fdfa86a7a4eb0e7b4fd99b8e3d3c02e2c921a2d7af"}
Sep 19 01:26:01.444263 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:01.444138    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f7939b7a-fffd54866-xlsbg" event={"ID":"e1758c9b-ec1c-445f-ae3e-3f675ec5daae","Type":"ContainerStarted","Data":"f44ed763d45dfa5da2e4ffeab1d74d67eff4ac956b3fa25d13a7eedae8e26a1c"}
Sep 19 01:26:01.444333 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:01.444291    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f7939b7a-fffd54866-xlsbg"
Sep 19 01:26:01.463248 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:01.463149    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-f7939b7a-58fd768898-cplfg" podStartSLOduration=1.463134056 podStartE2EDuration="1.463134056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 01:26:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 01:26:01.461381885 +0000 UTC m=+812.349050990" watchObservedRunningTime="2026-09-19 01:26:01.463134056 +0000 UTC m=+812.350803133"
Sep 19 01:26:01.479412 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:01.479343    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f7939b7a-fffd54866-xlsbg" podStartSLOduration=1.479323076 podStartE2EDuration="1.479323076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 01:26:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 01:26:01.477191555 +0000 UTC m=+812.364860653" watchObservedRunningTime="2026-09-19 01:26:01.479323076 +0000 UTC m=+812.366992153"
Sep 19 01:26:04.460992 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:04.460930    2581 generic.go:356] "Generic (PLEG): container finished" podID="2302d85f-6dba-4f71-bc69-6c64172f0f93" containerID="c8acbd4b6408ab58b32f26ce0478f8b58e17a2f5a39eb764f0bfaa7d9a1df10f" exitCode=0
Sep 19 01:26:04.460992 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:04.460982    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5a005d-openshift-default-cc9788f6b-twz5d" event={"ID":"2302d85f-6dba-4f71-bc69-6c64172f0f93","Type":"ContainerDied","Data":"c8acbd4b6408ab58b32f26ce0478f8b58e17a2f5a39eb764f0bfaa7d9a1df10f"}
Sep 19 01:26:04.461377 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:04.461363    2581 scope.go:117] "RemoveContainer" containerID="c8acbd4b6408ab58b32f26ce0478f8b58e17a2f5a39eb764f0bfaa7d9a1df10f"
Sep 19 01:26:04.462521 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:04.462494    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 19 01:26:04.462576 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:04.462567    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 19 01:26:04.462610 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:04.462597    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 19 01:26:05.466788 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:05.466754    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5a005d-openshift-default-cc9788f6b-twz5d" event={"ID":"2302d85f-6dba-4f71-bc69-6c64172f0f93","Type":"ContainerStarted","Data":"02374b819abfffc5abb51b208e9cd18dc7d7ee6f29d926f9e3a10ff82eb0bb4b"}
Sep 19 01:26:05.956845 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:05.956808    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-5a005d-openshift-default-cc9788f6b-twz5d"
Sep 19 01:26:05.957204 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:05.957181    2581 patch_prober.go:28] interesting pod/e2e-inf-b-5a005d-openshift-default-cc9788f6b-twz5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Sep 19 01:26:05.957255 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:05.957234    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5a005d-openshift-default-cc9788f6b-twz5d" podUID="2302d85f-6dba-4f71-bc69-6c64172f0f93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Sep 19 01:26:06.540617 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:06.540579    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c9nsz"]
Sep 19 01:26:06.957249 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:06.957218    2581 patch_prober.go:28] interesting pod/e2e-inf-b-5a005d-openshift-default-cc9788f6b-twz5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Sep 19 01:26:06.957401 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:06.957282    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5a005d-openshift-default-cc9788f6b-twz5d" podUID="2302d85f-6dba-4f71-bc69-6c64172f0f93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Sep 19 01:26:07.957749 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:07.957717    2581 patch_prober.go:28] interesting pod/e2e-inf-b-5a005d-openshift-default-cc9788f6b-twz5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Sep 19 01:26:07.958161 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:07.957771    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5a005d-openshift-default-cc9788f6b-twz5d" podUID="2302d85f-6dba-4f71-bc69-6c64172f0f93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Sep 19 01:26:08.957275 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:08.957237    2581 patch_prober.go:28] interesting pod/e2e-inf-b-5a005d-openshift-default-cc9788f6b-twz5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Sep 19 01:26:08.957458 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:08.957303    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5a005d-openshift-default-cc9788f6b-twz5d" podUID="2302d85f-6dba-4f71-bc69-6c64172f0f93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Sep 19 01:26:09.957247 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:09.957210    2581 patch_prober.go:28] interesting pod/e2e-inf-b-5a005d-openshift-default-cc9788f6b-twz5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Sep 19 01:26:09.957656 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:09.957275    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5a005d-openshift-default-cc9788f6b-twz5d" podUID="2302d85f-6dba-4f71-bc69-6c64172f0f93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Sep 19 01:26:10.957275 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:10.957229    2581 patch_prober.go:28] interesting pod/e2e-inf-b-5a005d-openshift-default-cc9788f6b-twz5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Sep 19 01:26:10.957760 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:10.957294    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5a005d-openshift-default-cc9788f6b-twz5d" podUID="2302d85f-6dba-4f71-bc69-6c64172f0f93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Sep 19 01:26:11.290271 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:11.290191    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-8e708f/simulator-w0-8e708f-kserve-5f9cd6679f-6ncxb"]
Sep 19 01:26:11.293218 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:11.293193    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-8e708f/simulator-w0-8e708f-kserve-5f9cd6679f-6ncxb"
Sep 19 01:26:11.296606 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:11.296581    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-8e708f\"/\"simulator-w0-8e708f-kserve-self-signed-certs\""
Sep 19 01:26:11.296757 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:11.296624    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-8e708f\"/\"openshift-service-ca.crt\""
Sep 19 01:26:11.296757 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:11.296631    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-8e708f\"/\"default-dockercfg-slhlm\""
Sep 19 01:26:11.296757 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:11.296631    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-8e708f\"/\"kube-root-ca.crt\""
Sep 19 01:26:11.303686 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:11.303661    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-8e708f/simulator-w0-8e708f-kserve-5f9cd6679f-6ncxb"]
Sep 19 01:26:11.321320 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:11.321292    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-5a005d-openshift-default-cc9788f6b-twz5d"
Sep 19 01:26:11.447546 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:11.447515    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6fdc64e9-7d4e-40a9-a8b4-a243f5ef5df8-home\") pod \"simulator-w0-8e708f-kserve-5f9cd6679f-6ncxb\" (UID: \"6fdc64e9-7d4e-40a9-a8b4-a243f5ef5df8\") " pod="e2e-models-e2e-worker-w0-8e708f/simulator-w0-8e708f-kserve-5f9cd6679f-6ncxb"
Sep 19 01:26:11.447546 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:11.447543    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sb8vx\" (UniqueName: \"kubernetes.io/projected/6fdc64e9-7d4e-40a9-a8b4-a243f5ef5df8-kube-api-access-sb8vx\") pod \"simulator-w0-8e708f-kserve-5f9cd6679f-6ncxb\" (UID: \"6fdc64e9-7d4e-40a9-a8b4-a243f5ef5df8\") " pod="e2e-models-e2e-worker-w0-8e708f/simulator-w0-8e708f-kserve-5f9cd6679f-6ncxb"
Sep 19 01:26:11.447793 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:11.447584    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fdc64e9-7d4e-40a9-a8b4-a243f5ef5df8-tls-certs\") pod \"simulator-w0-8e708f-kserve-5f9cd6679f-6ncxb\" (UID: \"6fdc64e9-7d4e-40a9-a8b4-a243f5ef5df8\") " pod="e2e-models-e2e-worker-w0-8e708f/simulator-w0-8e708f-kserve-5f9cd6679f-6ncxb"
Sep 19 01:26:11.447793 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:11.447737    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6fdc64e9-7d4e-40a9-a8b4-a243f5ef5df8-dshm\") pod \"simulator-w0-8e708f-kserve-5f9cd6679f-6ncxb\" (UID: \"6fdc64e9-7d4e-40a9-a8b4-a243f5ef5df8\") " pod="e2e-models-e2e-worker-w0-8e708f/simulator-w0-8e708f-kserve-5f9cd6679f-6ncxb"
Sep 19 01:26:11.447926 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:11.447811    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6fdc64e9-7d4e-40a9-a8b4-a243f5ef5df8-tmp-dir\") pod \"simulator-w0-8e708f-kserve-5f9cd6679f-6ncxb\" (UID: \"6fdc64e9-7d4e-40a9-a8b4-a243f5ef5df8\") " pod="e2e-models-e2e-worker-w0-8e708f/simulator-w0-8e708f-kserve-5f9cd6679f-6ncxb"
Sep 19 01:26:11.447926 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:11.447887    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6fdc64e9-7d4e-40a9-a8b4-a243f5ef5df8-model-cache\") pod \"simulator-w0-8e708f-kserve-5f9cd6679f-6ncxb\" (UID: \"6fdc64e9-7d4e-40a9-a8b4-a243f5ef5df8\") " pod="e2e-models-e2e-worker-w0-8e708f/simulator-w0-8e708f-kserve-5f9cd6679f-6ncxb"
Sep 19 01:26:11.548643 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:11.548581    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6fdc64e9-7d4e-40a9-a8b4-a243f5ef5df8-home\") pod \"simulator-w0-8e708f-kserve-5f9cd6679f-6ncxb\" (UID: \"6fdc64e9-7d4e-40a9-a8b4-a243f5ef5df8\") " pod="e2e-models-e2e-worker-w0-8e708f/simulator-w0-8e708f-kserve-5f9cd6679f-6ncxb"
Sep 19 01:26:11.548643 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:11.548607    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb8vx\" (UniqueName: \"kubernetes.io/projected/6fdc64e9-7d4e-40a9-a8b4-a243f5ef5df8-kube-api-access-sb8vx\") pod \"simulator-w0-8e708f-kserve-5f9cd6679f-6ncxb\" (UID: \"6fdc64e9-7d4e-40a9-a8b4-a243f5ef5df8\") " pod="e2e-models-e2e-worker-w0-8e708f/simulator-w0-8e708f-kserve-5f9cd6679f-6ncxb"
Sep 19 01:26:11.548643 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:11.548625    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fdc64e9-7d4e-40a9-a8b4-a243f5ef5df8-tls-certs\") pod \"simulator-w0-8e708f-kserve-5f9cd6679f-6ncxb\" (UID: \"6fdc64e9-7d4e-40a9-a8b4-a243f5ef5df8\") " pod="e2e-models-e2e-worker-w0-8e708f/simulator-w0-8e708f-kserve-5f9cd6679f-6ncxb"
Sep 19 01:26:11.548899 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:11.548776    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6fdc64e9-7d4e-40a9-a8b4-a243f5ef5df8-dshm\") pod \"simulator-w0-8e708f-kserve-5f9cd6679f-6ncxb\" (UID: \"6fdc64e9-7d4e-40a9-a8b4-a243f5ef5df8\") " pod="e2e-models-e2e-worker-w0-8e708f/simulator-w0-8e708f-kserve-5f9cd6679f-6ncxb"
Sep 19 01:26:11.548899 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:11.548834    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6fdc64e9-7d4e-40a9-a8b4-a243f5ef5df8-tmp-dir\") pod \"simulator-w0-8e708f-kserve-5f9cd6679f-6ncxb\" (UID: \"6fdc64e9-7d4e-40a9-a8b4-a243f5ef5df8\") " pod="e2e-models-e2e-worker-w0-8e708f/simulator-w0-8e708f-kserve-5f9cd6679f-6ncxb"
Sep 19 01:26:11.549013 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:11.548914    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6fdc64e9-7d4e-40a9-a8b4-a243f5ef5df8-model-cache\") pod \"simulator-w0-8e708f-kserve-5f9cd6679f-6ncxb\" (UID: \"6fdc64e9-7d4e-40a9-a8b4-a243f5ef5df8\") " pod="e2e-models-e2e-worker-w0-8e708f/simulator-w0-8e708f-kserve-5f9cd6679f-6ncxb"
Sep 19 01:26:11.549013 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:11.548990    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6fdc64e9-7d4e-40a9-a8b4-a243f5ef5df8-home\") pod \"simulator-w0-8e708f-kserve-5f9cd6679f-6ncxb\" (UID: \"6fdc64e9-7d4e-40a9-a8b4-a243f5ef5df8\") " pod="e2e-models-e2e-worker-w0-8e708f/simulator-w0-8e708f-kserve-5f9cd6679f-6ncxb"
Sep 19 01:26:11.549208 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:11.549186    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6fdc64e9-7d4e-40a9-a8b4-a243f5ef5df8-tmp-dir\") pod \"simulator-w0-8e708f-kserve-5f9cd6679f-6ncxb\" (UID: \"6fdc64e9-7d4e-40a9-a8b4-a243f5ef5df8\") " pod="e2e-models-e2e-worker-w0-8e708f/simulator-w0-8e708f-kserve-5f9cd6679f-6ncxb"
Sep 19 01:26:11.549208 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:11.549200    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6fdc64e9-7d4e-40a9-a8b4-a243f5ef5df8-model-cache\") pod \"simulator-w0-8e708f-kserve-5f9cd6679f-6ncxb\" (UID: \"6fdc64e9-7d4e-40a9-a8b4-a243f5ef5df8\") " pod="e2e-models-e2e-worker-w0-8e708f/simulator-w0-8e708f-kserve-5f9cd6679f-6ncxb"
Sep 19 01:26:11.551154 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:11.551125    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6fdc64e9-7d4e-40a9-a8b4-a243f5ef5df8-dshm\") pod \"simulator-w0-8e708f-kserve-5f9cd6679f-6ncxb\" (UID: \"6fdc64e9-7d4e-40a9-a8b4-a243f5ef5df8\") " pod="e2e-models-e2e-worker-w0-8e708f/simulator-w0-8e708f-kserve-5f9cd6679f-6ncxb"
Sep 19 01:26:11.551504 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:11.551485    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fdc64e9-7d4e-40a9-a8b4-a243f5ef5df8-tls-certs\") pod \"simulator-w0-8e708f-kserve-5f9cd6679f-6ncxb\" (UID: \"6fdc64e9-7d4e-40a9-a8b4-a243f5ef5df8\") " pod="e2e-models-e2e-worker-w0-8e708f/simulator-w0-8e708f-kserve-5f9cd6679f-6ncxb"
Sep 19 01:26:11.558021 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:11.557996    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb8vx\" (UniqueName: \"kubernetes.io/projected/6fdc64e9-7d4e-40a9-a8b4-a243f5ef5df8-kube-api-access-sb8vx\") pod \"simulator-w0-8e708f-kserve-5f9cd6679f-6ncxb\" (UID: \"6fdc64e9-7d4e-40a9-a8b4-a243f5ef5df8\") " pod="e2e-models-e2e-worker-w0-8e708f/simulator-w0-8e708f-kserve-5f9cd6679f-6ncxb"
Sep 19 01:26:11.605965 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:11.605935    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-8e708f/simulator-w0-8e708f-kserve-5f9cd6679f-6ncxb"
Sep 19 01:26:11.876417 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:11.876388    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-8e708f/simulator-w0-8e708f-kserve-5f9cd6679f-6ncxb"]
Sep 19 01:26:11.879432 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:26:11.879406    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6fdc64e9_7d4e_40a9_a8b4_a243f5ef5df8.slice/crio-02dbf6609d85109b1c9c40d5827ad1b5dd2bcc9d5bda554c6888426fc6963c65 WatchSource:0}: Error finding container 02dbf6609d85109b1c9c40d5827ad1b5dd2bcc9d5bda554c6888426fc6963c65: Status 404 returned error can't find the container with id 02dbf6609d85109b1c9c40d5827ad1b5dd2bcc9d5bda554c6888426fc6963c65
Sep 19 01:26:11.962187 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:11.962157    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-5a005d-openshift-default-cc9788f6b-twz5d"
Sep 19 01:26:11.963262 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:11.963236    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-5a005d-openshift-default-cc9788f6b-twz5d"
Sep 19 01:26:12.452048 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:12.452014    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-f7939b7a-58fd768898-cplfg"
Sep 19 01:26:12.452223 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:12.452163    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f7939b7a-fffd54866-xlsbg"
Sep 19 01:26:12.508937 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:12.508897    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-8e708f/simulator-w0-8e708f-kserve-5f9cd6679f-6ncxb" event={"ID":"6fdc64e9-7d4e-40a9-a8b4-a243f5ef5df8","Type":"ContainerStarted","Data":"549a712dae5e7fb5f0e9e85ab8108a06db6fe0cd53778bfa8711c7337b9f50b3"}
Sep 19 01:26:12.508937 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:12.508947    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-8e708f/simulator-w0-8e708f-kserve-5f9cd6679f-6ncxb" event={"ID":"6fdc64e9-7d4e-40a9-a8b4-a243f5ef5df8","Type":"ContainerStarted","Data":"02dbf6609d85109b1c9c40d5827ad1b5dd2bcc9d5bda554c6888426fc6963c65"}
Sep 19 01:26:12.530935 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:12.530891    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-8e708f/simulator-w0-8e708f-kserve-5f9cd6679f-6ncxb" podStartSLOduration=1.5308731199999999 podStartE2EDuration="1.53087312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 01:26:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 01:26:12.528831932 +0000 UTC m=+823.416501045" watchObservedRunningTime="2026-09-19 01:26:12.53087312 +0000 UTC m=+823.418542188"
Sep 19 01:26:18.593934 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:18.593902    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c9nsz"]
Sep 19 01:26:21.606387 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:21.606353    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-8e708f/simulator-w0-8e708f-kserve-5f9cd6679f-6ncxb"
Sep 19 01:26:21.606965 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:21.606415    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-8e708f/simulator-w0-8e708f-kserve-5f9cd6679f-6ncxb"
Sep 19 01:26:21.618739 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:21.618712    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-8e708f/simulator-w0-8e708f-kserve-5f9cd6679f-6ncxb"
Sep 19 01:26:22.565770 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:22.565741    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-8e708f/simulator-w0-8e708f-kserve-5f9cd6679f-6ncxb"
Sep 19 01:26:31.819475 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:31.819442    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f7939b7a-xc4fk"]
Sep 19 01:26:31.826145 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:31.826124    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f7939b7a-xc4fk"
Sep 19 01:26:31.834211 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:31.834189    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 19 01:26:31.836122 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:31.836105    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s7pvt\""
Sep 19 01:26:31.849138 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:31.849117    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f7939b7a-xc4fk"]
Sep 19 01:26:31.917113 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:31.917082    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89748322-7bbf-4aa1-a554-d4d56ce4d344-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f7939b7a-xc4fk\" (UID: \"89748322-7bbf-4aa1-a554-d4d56ce4d344\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f7939b7a-xc4fk"
Sep 19 01:26:32.018015 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:32.017988    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89748322-7bbf-4aa1-a554-d4d56ce4d344-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f7939b7a-xc4fk\" (UID: \"89748322-7bbf-4aa1-a554-d4d56ce4d344\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f7939b7a-xc4fk"
Sep 19 01:26:32.018586 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:32.018566    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89748322-7bbf-4aa1-a554-d4d56ce4d344-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f7939b7a-xc4fk\" (UID: \"89748322-7bbf-4aa1-a554-d4d56ce4d344\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f7939b7a-xc4fk"
Sep 19 01:26:32.135248 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:32.135227    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f7939b7a-xc4fk"
Sep 19 01:26:32.425738 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:32.425623    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f7939b7a-xc4fk"]
Sep 19 01:26:32.428136 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:26:32.428112    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89748322_7bbf_4aa1_a554_d4d56ce4d344.slice/crio-9aa032e38fe59415ffb3f64bfa08131c46c4b6d809a7b681db6a68dadd8a2d15 WatchSource:0}: Error finding container 9aa032e38fe59415ffb3f64bfa08131c46c4b6d809a7b681db6a68dadd8a2d15: Status 404 returned error can't find the container with id 9aa032e38fe59415ffb3f64bfa08131c46c4b6d809a7b681db6a68dadd8a2d15
Sep 19 01:26:32.600976 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:32.600953    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f7939b7a-xc4fk" event={"ID":"89748322-7bbf-4aa1-a554-d4d56ce4d344","Type":"ContainerStarted","Data":"9aa032e38fe59415ffb3f64bfa08131c46c4b6d809a7b681db6a68dadd8a2d15"}
Sep 19 01:26:33.606609 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:33.606570    2581 generic.go:356] "Generic (PLEG): container finished" podID="89748322-7bbf-4aa1-a554-d4d56ce4d344" containerID="686e541a9465d8245ef26bda49170aea5a79b70c7c8a63946c36d0a4779fdb32" exitCode=0
Sep 19 01:26:33.607060 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:33.606651    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f7939b7a-xc4fk" event={"ID":"89748322-7bbf-4aa1-a554-d4d56ce4d344","Type":"ContainerDied","Data":"686e541a9465d8245ef26bda49170aea5a79b70c7c8a63946c36d0a4779fdb32"}
Sep 19 01:26:34.735025 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:34.735003    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f7939b7a-xc4fk"
Sep 19 01:26:34.840495 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:34.840473    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89748322-7bbf-4aa1-a554-d4d56ce4d344-maas-api-service-ca\") pod \"89748322-7bbf-4aa1-a554-d4d56ce4d344\" (UID: \"89748322-7bbf-4aa1-a554-d4d56ce4d344\") "
Sep 19 01:26:34.840840 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:34.840821    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89748322-7bbf-4aa1-a554-d4d56ce4d344-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "89748322-7bbf-4aa1-a554-d4d56ce4d344" (UID: "89748322-7bbf-4aa1-a554-d4d56ce4d344"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 01:26:34.941666 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:34.941585    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89748322-7bbf-4aa1-a554-d4d56ce4d344-maas-api-service-ca\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:26:35.617085 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:35.617055    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f7939b7a-xc4fk"
Sep 19 01:26:35.617085 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:35.617061    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f7939b7a-xc4fk" event={"ID":"89748322-7bbf-4aa1-a554-d4d56ce4d344","Type":"ContainerDied","Data":"9aa032e38fe59415ffb3f64bfa08131c46c4b6d809a7b681db6a68dadd8a2d15"}
Sep 19 01:26:35.617275 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:35.617109    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9aa032e38fe59415ffb3f64bfa08131c46c4b6d809a7b681db6a68dadd8a2d15"
Sep 19 01:26:42.960981 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:42.960947    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f7939b7a-58fd768898-cplfg"]
Sep 19 01:26:42.961433 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:42.961213    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-f7939b7a-58fd768898-cplfg" podUID="ac07e351-cae8-4583-8e92-380a629eb6aa" containerName="payload-processing" containerID="cri-o://c5abdf5cefb1373ebdab9d08ea2f241e8750bd085d8e942d27b68f9e76021342" gracePeriod=30
Sep 19 01:26:42.973416 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:42.973387    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f7939b7a-fffd54866-xlsbg"]
Sep 19 01:26:42.973633 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:42.973594    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f7939b7a-fffd54866-xlsbg" podUID="e1758c9b-ec1c-445f-ae3e-3f675ec5daae" containerName="payload-pre-processing" containerID="cri-o://2eed11657cf9e14eb00297fdfa86a7a4eb0e7b4fd99b8e3d3c02e2c921a2d7af" gracePeriod=30
Sep 19 01:26:43.224098 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:43.224023    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f7939b7a-xc4fk"]
Sep 19 01:26:43.227704 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:43.227596    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f7939b7a-xc4fk"]
Sep 19 01:26:43.249622 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:43.249601    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f7939b7a-fffd54866-xlsbg"
Sep 19 01:26:43.252408 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:43.252388    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-f7939b7a-58fd768898-cplfg"
Sep 19 01:26:43.274220 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:43.274193    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89748322-7bbf-4aa1-a554-d4d56ce4d344" path="/var/lib/kubelet/pods/89748322-7bbf-4aa1-a554-d4d56ce4d344/volumes"
Sep 19 01:26:43.310305 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:43.310262    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6g57d\" (UniqueName: \"kubernetes.io/projected/ac07e351-cae8-4583-8e92-380a629eb6aa-kube-api-access-6g57d\") pod \"ac07e351-cae8-4583-8e92-380a629eb6aa\" (UID: \"ac07e351-cae8-4583-8e92-380a629eb6aa\") "
Sep 19 01:26:43.310462 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:43.310315    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1758c9b-ec1c-445f-ae3e-3f675ec5daae-plugins-config-volume\") pod \"e1758c9b-ec1c-445f-ae3e-3f675ec5daae\" (UID: \"e1758c9b-ec1c-445f-ae3e-3f675ec5daae\") "
Sep 19 01:26:43.310462 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:43.310362    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac07e351-cae8-4583-8e92-380a629eb6aa-plugins-config-volume\") pod \"ac07e351-cae8-4583-8e92-380a629eb6aa\" (UID: \"ac07e351-cae8-4583-8e92-380a629eb6aa\") "
Sep 19 01:26:43.310462 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:43.310410    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c8wgc\" (UniqueName: \"kubernetes.io/projected/e1758c9b-ec1c-445f-ae3e-3f675ec5daae-kube-api-access-c8wgc\") pod \"e1758c9b-ec1c-445f-ae3e-3f675ec5daae\" (UID: \"e1758c9b-ec1c-445f-ae3e-3f675ec5daae\") "
Sep 19 01:26:43.310776 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:43.310752    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac07e351-cae8-4583-8e92-380a629eb6aa-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ac07e351-cae8-4583-8e92-380a629eb6aa" (UID: "ac07e351-cae8-4583-8e92-380a629eb6aa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 01:26:43.310838 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:43.310752    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1758c9b-ec1c-445f-ae3e-3f675ec5daae-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e1758c9b-ec1c-445f-ae3e-3f675ec5daae" (UID: "e1758c9b-ec1c-445f-ae3e-3f675ec5daae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 01:26:43.312636 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:43.312599    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac07e351-cae8-4583-8e92-380a629eb6aa-kube-api-access-6g57d" (OuterVolumeSpecName: "kube-api-access-6g57d") pod "ac07e351-cae8-4583-8e92-380a629eb6aa" (UID: "ac07e351-cae8-4583-8e92-380a629eb6aa"). InnerVolumeSpecName "kube-api-access-6g57d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 01:26:43.312730 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:43.312601    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1758c9b-ec1c-445f-ae3e-3f675ec5daae-kube-api-access-c8wgc" (OuterVolumeSpecName: "kube-api-access-c8wgc") pod "e1758c9b-ec1c-445f-ae3e-3f675ec5daae" (UID: "e1758c9b-ec1c-445f-ae3e-3f675ec5daae"). InnerVolumeSpecName "kube-api-access-c8wgc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 01:26:43.411256 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:43.411226    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6g57d\" (UniqueName: \"kubernetes.io/projected/ac07e351-cae8-4583-8e92-380a629eb6aa-kube-api-access-6g57d\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:26:43.411256 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:43.411251    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1758c9b-ec1c-445f-ae3e-3f675ec5daae-plugins-config-volume\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:26:43.411256 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:43.411262    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac07e351-cae8-4583-8e92-380a629eb6aa-plugins-config-volume\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:26:43.411458 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:43.411272    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8wgc\" (UniqueName: \"kubernetes.io/projected/e1758c9b-ec1c-445f-ae3e-3f675ec5daae-kube-api-access-c8wgc\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:26:43.656081 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:43.656044    2581 generic.go:356] "Generic (PLEG): container finished" podID="ac07e351-cae8-4583-8e92-380a629eb6aa" containerID="c5abdf5cefb1373ebdab9d08ea2f241e8750bd085d8e942d27b68f9e76021342" exitCode=0
Sep 19 01:26:43.656265 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:43.656091    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-f7939b7a-58fd768898-cplfg"
Sep 19 01:26:43.656265 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:43.656128    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-f7939b7a-58fd768898-cplfg" event={"ID":"ac07e351-cae8-4583-8e92-380a629eb6aa","Type":"ContainerDied","Data":"c5abdf5cefb1373ebdab9d08ea2f241e8750bd085d8e942d27b68f9e76021342"}
Sep 19 01:26:43.656265 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:43.656186    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-f7939b7a-58fd768898-cplfg" event={"ID":"ac07e351-cae8-4583-8e92-380a629eb6aa","Type":"ContainerDied","Data":"423970c480f237ba0437c3b1f5f09246bda1c36b656b0cad509d37511025aa0b"}
Sep 19 01:26:43.656265 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:43.656216    2581 scope.go:117] "RemoveContainer" containerID="c5abdf5cefb1373ebdab9d08ea2f241e8750bd085d8e942d27b68f9e76021342"
Sep 19 01:26:43.657529 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:43.657506    2581 generic.go:356] "Generic (PLEG): container finished" podID="e1758c9b-ec1c-445f-ae3e-3f675ec5daae" containerID="2eed11657cf9e14eb00297fdfa86a7a4eb0e7b4fd99b8e3d3c02e2c921a2d7af" exitCode=0
Sep 19 01:26:43.657653 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:43.657555    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f7939b7a-fffd54866-xlsbg"
Sep 19 01:26:43.657653 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:43.657584    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f7939b7a-fffd54866-xlsbg" event={"ID":"e1758c9b-ec1c-445f-ae3e-3f675ec5daae","Type":"ContainerDied","Data":"2eed11657cf9e14eb00297fdfa86a7a4eb0e7b4fd99b8e3d3c02e2c921a2d7af"}
Sep 19 01:26:43.657653 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:43.657619    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f7939b7a-fffd54866-xlsbg" event={"ID":"e1758c9b-ec1c-445f-ae3e-3f675ec5daae","Type":"ContainerDied","Data":"f44ed763d45dfa5da2e4ffeab1d74d67eff4ac956b3fa25d13a7eedae8e26a1c"}
Sep 19 01:26:43.667617 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:43.667601    2581 scope.go:117] "RemoveContainer" containerID="c5abdf5cefb1373ebdab9d08ea2f241e8750bd085d8e942d27b68f9e76021342"
Sep 19 01:26:43.667916 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:26:43.667893    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5abdf5cefb1373ebdab9d08ea2f241e8750bd085d8e942d27b68f9e76021342\": container with ID starting with c5abdf5cefb1373ebdab9d08ea2f241e8750bd085d8e942d27b68f9e76021342 not found: ID does not exist" containerID="c5abdf5cefb1373ebdab9d08ea2f241e8750bd085d8e942d27b68f9e76021342"
Sep 19 01:26:43.667972 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:43.667926    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5abdf5cefb1373ebdab9d08ea2f241e8750bd085d8e942d27b68f9e76021342"} err="failed to get container status \"c5abdf5cefb1373ebdab9d08ea2f241e8750bd085d8e942d27b68f9e76021342\": rpc error: code = NotFound desc = could not find container \"c5abdf5cefb1373ebdab9d08ea2f241e8750bd085d8e942d27b68f9e76021342\": container with ID starting with c5abdf5cefb1373ebdab9d08ea2f241e8750bd085d8e942d27b68f9e76021342 not found: ID does not exist"
Sep 19 01:26:43.667972 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:43.667943    2581 scope.go:117] "RemoveContainer" containerID="2eed11657cf9e14eb00297fdfa86a7a4eb0e7b4fd99b8e3d3c02e2c921a2d7af"
Sep 19 01:26:43.675501 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:43.675485    2581 scope.go:117] "RemoveContainer" containerID="2eed11657cf9e14eb00297fdfa86a7a4eb0e7b4fd99b8e3d3c02e2c921a2d7af"
Sep 19 01:26:43.675767 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:26:43.675751    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2eed11657cf9e14eb00297fdfa86a7a4eb0e7b4fd99b8e3d3c02e2c921a2d7af\": container with ID starting with 2eed11657cf9e14eb00297fdfa86a7a4eb0e7b4fd99b8e3d3c02e2c921a2d7af not found: ID does not exist" containerID="2eed11657cf9e14eb00297fdfa86a7a4eb0e7b4fd99b8e3d3c02e2c921a2d7af"
Sep 19 01:26:43.675817 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:43.675773    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2eed11657cf9e14eb00297fdfa86a7a4eb0e7b4fd99b8e3d3c02e2c921a2d7af"} err="failed to get container status \"2eed11657cf9e14eb00297fdfa86a7a4eb0e7b4fd99b8e3d3c02e2c921a2d7af\": rpc error: code = NotFound desc = could not find container \"2eed11657cf9e14eb00297fdfa86a7a4eb0e7b4fd99b8e3d3c02e2c921a2d7af\": container with ID starting with 2eed11657cf9e14eb00297fdfa86a7a4eb0e7b4fd99b8e3d3c02e2c921a2d7af not found: ID does not exist"
Sep 19 01:26:43.717059 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:43.717034    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f7939b7a-58fd768898-cplfg"]
Sep 19 01:26:43.723189 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:43.723169    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f7939b7a-58fd768898-cplfg"]
Sep 19 01:26:43.766316 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:43.766291    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f7939b7a-fffd54866-xlsbg"]
Sep 19 01:26:43.768296 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:43.768272    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f7939b7a-fffd54866-xlsbg"]
Sep 19 01:26:45.273638 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:45.273605    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac07e351-cae8-4583-8e92-380a629eb6aa" path="/var/lib/kubelet/pods/ac07e351-cae8-4583-8e92-380a629eb6aa/volumes"
Sep 19 01:26:45.274036 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:45.273955    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1758c9b-ec1c-445f-ae3e-3f675ec5daae" path="/var/lib/kubelet/pods/e1758c9b-ec1c-445f-ae3e-3f675ec5daae/volumes"
Sep 19 01:26:49.663393 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:49.663352    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c9nsz"]
Sep 19 01:26:51.693248 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:51.693205    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f8c7c101-9d765647c-lfxvq"]
Sep 19 01:26:51.694168 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:51.694139    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac07e351-cae8-4583-8e92-380a629eb6aa" containerName="payload-processing"
Sep 19 01:26:51.694168 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:51.694168    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac07e351-cae8-4583-8e92-380a629eb6aa" containerName="payload-processing"
Sep 19 01:26:51.694352 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:51.694187    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1758c9b-ec1c-445f-ae3e-3f675ec5daae" containerName="payload-pre-processing"
Sep 19 01:26:51.694352 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:51.694196    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1758c9b-ec1c-445f-ae3e-3f675ec5daae" containerName="payload-pre-processing"
Sep 19 01:26:51.694352 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:51.694218    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89748322-7bbf-4aa1-a554-d4d56ce4d344" containerName="revoke-keys"
Sep 19 01:26:51.694352 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:51.694227    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="89748322-7bbf-4aa1-a554-d4d56ce4d344" containerName="revoke-keys"
Sep 19 01:26:51.694530 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:51.694380    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac07e351-cae8-4583-8e92-380a629eb6aa" containerName="payload-processing"
Sep 19 01:26:51.694530 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:51.694401    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1758c9b-ec1c-445f-ae3e-3f675ec5daae" containerName="payload-pre-processing"
Sep 19 01:26:51.694530 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:51.694414    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="89748322-7bbf-4aa1-a554-d4d56ce4d344" containerName="revoke-keys"
Sep 19 01:26:51.700780 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:51.699330    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f8c7c101-9d765647c-lfxvq"
Sep 19 01:26:51.703746 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:51.703719    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-f8c7c101\""
Sep 19 01:26:51.704425 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:51.704398    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-f8c7c101-dockercfg-zdtd8\""
Sep 19 01:26:51.714521 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:51.714494    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f8c7c101-9d765647c-lfxvq"]
Sep 19 01:26:51.782416 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:51.782385    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8d9f\" (UniqueName: \"kubernetes.io/projected/625b7cd5-db6c-4dc6-88a7-2a000d2cc0b9-kube-api-access-f8d9f\") pod \"payload-processing-e2e-ait-f8c7c101-9d765647c-lfxvq\" (UID: \"625b7cd5-db6c-4dc6-88a7-2a000d2cc0b9\") " pod="openshift-ingress/payload-processing-e2e-ait-f8c7c101-9d765647c-lfxvq"
Sep 19 01:26:51.782536 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:51.782425    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/625b7cd5-db6c-4dc6-88a7-2a000d2cc0b9-plugins-config-volume\") pod \"payload-processing-e2e-ait-f8c7c101-9d765647c-lfxvq\" (UID: \"625b7cd5-db6c-4dc6-88a7-2a000d2cc0b9\") " pod="openshift-ingress/payload-processing-e2e-ait-f8c7c101-9d765647c-lfxvq"
Sep 19 01:26:51.883199 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:51.883174    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8d9f\" (UniqueName: \"kubernetes.io/projected/625b7cd5-db6c-4dc6-88a7-2a000d2cc0b9-kube-api-access-f8d9f\") pod \"payload-processing-e2e-ait-f8c7c101-9d765647c-lfxvq\" (UID: \"625b7cd5-db6c-4dc6-88a7-2a000d2cc0b9\") " pod="openshift-ingress/payload-processing-e2e-ait-f8c7c101-9d765647c-lfxvq"
Sep 19 01:26:51.883302 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:51.883210    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/625b7cd5-db6c-4dc6-88a7-2a000d2cc0b9-plugins-config-volume\") pod \"payload-processing-e2e-ait-f8c7c101-9d765647c-lfxvq\" (UID: \"625b7cd5-db6c-4dc6-88a7-2a000d2cc0b9\") " pod="openshift-ingress/payload-processing-e2e-ait-f8c7c101-9d765647c-lfxvq"
Sep 19 01:26:51.883913 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:51.883897    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/625b7cd5-db6c-4dc6-88a7-2a000d2cc0b9-plugins-config-volume\") pod \"payload-processing-e2e-ait-f8c7c101-9d765647c-lfxvq\" (UID: \"625b7cd5-db6c-4dc6-88a7-2a000d2cc0b9\") " pod="openshift-ingress/payload-processing-e2e-ait-f8c7c101-9d765647c-lfxvq"
Sep 19 01:26:51.895996 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:51.895974    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8d9f\" (UniqueName: \"kubernetes.io/projected/625b7cd5-db6c-4dc6-88a7-2a000d2cc0b9-kube-api-access-f8d9f\") pod \"payload-processing-e2e-ait-f8c7c101-9d765647c-lfxvq\" (UID: \"625b7cd5-db6c-4dc6-88a7-2a000d2cc0b9\") " pod="openshift-ingress/payload-processing-e2e-ait-f8c7c101-9d765647c-lfxvq"
Sep 19 01:26:52.015778 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:52.015718    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f8c7c101-9d765647c-lfxvq"
Sep 19 01:26:52.315535 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:52.315479    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f8c7c101-9d765647c-lfxvq"]
Sep 19 01:26:52.318068 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:26:52.318040    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod625b7cd5_db6c_4dc6_88a7_2a000d2cc0b9.slice/crio-b2af07d4b2e9b1b39d12cb64f8f7443d81f6441f75dc5e9a13f73af4da33e444 WatchSource:0}: Error finding container b2af07d4b2e9b1b39d12cb64f8f7443d81f6441f75dc5e9a13f73af4da33e444: Status 404 returned error can't find the container with id b2af07d4b2e9b1b39d12cb64f8f7443d81f6441f75dc5e9a13f73af4da33e444
Sep 19 01:26:52.712205 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:52.712168    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f8c7c101-9d765647c-lfxvq" event={"ID":"625b7cd5-db6c-4dc6-88a7-2a000d2cc0b9","Type":"ContainerStarted","Data":"fffabb773b03f14d29786b526e330ddeee7ed15830da05a1ba961f84023aa668"}
Sep 19 01:26:52.712678 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:52.712229    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f8c7c101-9d765647c-lfxvq" event={"ID":"625b7cd5-db6c-4dc6-88a7-2a000d2cc0b9","Type":"ContainerStarted","Data":"b2af07d4b2e9b1b39d12cb64f8f7443d81f6441f75dc5e9a13f73af4da33e444"}
Sep 19 01:26:52.712678 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:52.712300    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-f8c7c101-9d765647c-lfxvq"
Sep 19 01:26:52.741230 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:26:52.741181    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-f8c7c101-9d765647c-lfxvq" podStartSLOduration=1.741167414 podStartE2EDuration="1.741167414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 01:26:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 01:26:52.739637441 +0000 UTC m=+863.627306548" watchObservedRunningTime="2026-09-19 01:26:52.741167414 +0000 UTC m=+863.628836487"
Sep 19 01:27:03.719050 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:03.719015    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-f8c7c101-9d765647c-lfxvq"
Sep 19 01:27:08.587257 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:08.587220    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c9nsz"]
Sep 19 01:27:11.989824 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:11.989743    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-b3b22c/simulator-w4-b3b22c-kserve-56bdc45497-rrcv7"]
Sep 19 01:27:11.994689 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:11.994667    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b3b22c/simulator-w4-b3b22c-kserve-56bdc45497-rrcv7"
Sep 19 01:27:11.998110 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:11.998083    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-b3b22c\"/\"kube-root-ca.crt\""
Sep 19 01:27:11.998110 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:11.998104    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-b3b22c\"/\"openshift-service-ca.crt\""
Sep 19 01:27:11.998282 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:11.998083    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-b3b22c\"/\"simulator-w4-b3b22c-kserve-self-signed-certs\""
Sep 19 01:27:11.998282 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:11.998104    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-b3b22c\"/\"default-dockercfg-jjh6d\""
Sep 19 01:27:12.002874 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:12.002835    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b3b22c/simulator-w4-b3b22c-kserve-56bdc45497-rrcv7"]
Sep 19 01:27:12.145080 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:12.145044    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33e953a5-9fa2-47ce-b518-8c2752d02da2-tls-certs\") pod \"simulator-w4-b3b22c-kserve-56bdc45497-rrcv7\" (UID: \"33e953a5-9fa2-47ce-b518-8c2752d02da2\") " pod="e2e-models-e2e-worker-w4-b3b22c/simulator-w4-b3b22c-kserve-56bdc45497-rrcv7"
Sep 19 01:27:12.145080 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:12.145081    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qc4bz\" (UniqueName: \"kubernetes.io/projected/33e953a5-9fa2-47ce-b518-8c2752d02da2-kube-api-access-qc4bz\") pod \"simulator-w4-b3b22c-kserve-56bdc45497-rrcv7\" (UID: \"33e953a5-9fa2-47ce-b518-8c2752d02da2\") " pod="e2e-models-e2e-worker-w4-b3b22c/simulator-w4-b3b22c-kserve-56bdc45497-rrcv7"
Sep 19 01:27:12.145328 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:12.145106    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33e953a5-9fa2-47ce-b518-8c2752d02da2-model-cache\") pod \"simulator-w4-b3b22c-kserve-56bdc45497-rrcv7\" (UID: \"33e953a5-9fa2-47ce-b518-8c2752d02da2\") " pod="e2e-models-e2e-worker-w4-b3b22c/simulator-w4-b3b22c-kserve-56bdc45497-rrcv7"
Sep 19 01:27:12.145328 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:12.145208    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33e953a5-9fa2-47ce-b518-8c2752d02da2-home\") pod \"simulator-w4-b3b22c-kserve-56bdc45497-rrcv7\" (UID: \"33e953a5-9fa2-47ce-b518-8c2752d02da2\") " pod="e2e-models-e2e-worker-w4-b3b22c/simulator-w4-b3b22c-kserve-56bdc45497-rrcv7"
Sep 19 01:27:12.145328 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:12.145253    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33e953a5-9fa2-47ce-b518-8c2752d02da2-dshm\") pod \"simulator-w4-b3b22c-kserve-56bdc45497-rrcv7\" (UID: \"33e953a5-9fa2-47ce-b518-8c2752d02da2\") " pod="e2e-models-e2e-worker-w4-b3b22c/simulator-w4-b3b22c-kserve-56bdc45497-rrcv7"
Sep 19 01:27:12.145465 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:12.145344    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33e953a5-9fa2-47ce-b518-8c2752d02da2-tmp-dir\") pod \"simulator-w4-b3b22c-kserve-56bdc45497-rrcv7\" (UID: \"33e953a5-9fa2-47ce-b518-8c2752d02da2\") " pod="e2e-models-e2e-worker-w4-b3b22c/simulator-w4-b3b22c-kserve-56bdc45497-rrcv7"
Sep 19 01:27:12.246423 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:12.246331    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33e953a5-9fa2-47ce-b518-8c2752d02da2-tmp-dir\") pod \"simulator-w4-b3b22c-kserve-56bdc45497-rrcv7\" (UID: \"33e953a5-9fa2-47ce-b518-8c2752d02da2\") " pod="e2e-models-e2e-worker-w4-b3b22c/simulator-w4-b3b22c-kserve-56bdc45497-rrcv7"
Sep 19 01:27:12.246423 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:12.246399    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33e953a5-9fa2-47ce-b518-8c2752d02da2-tls-certs\") pod \"simulator-w4-b3b22c-kserve-56bdc45497-rrcv7\" (UID: \"33e953a5-9fa2-47ce-b518-8c2752d02da2\") " pod="e2e-models-e2e-worker-w4-b3b22c/simulator-w4-b3b22c-kserve-56bdc45497-rrcv7"
Sep 19 01:27:12.246714 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:12.246429    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc4bz\" (UniqueName: \"kubernetes.io/projected/33e953a5-9fa2-47ce-b518-8c2752d02da2-kube-api-access-qc4bz\") pod \"simulator-w4-b3b22c-kserve-56bdc45497-rrcv7\" (UID: \"33e953a5-9fa2-47ce-b518-8c2752d02da2\") " pod="e2e-models-e2e-worker-w4-b3b22c/simulator-w4-b3b22c-kserve-56bdc45497-rrcv7"
Sep 19 01:27:12.246714 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:12.246460    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33e953a5-9fa2-47ce-b518-8c2752d02da2-model-cache\") pod \"simulator-w4-b3b22c-kserve-56bdc45497-rrcv7\" (UID: \"33e953a5-9fa2-47ce-b518-8c2752d02da2\") " pod="e2e-models-e2e-worker-w4-b3b22c/simulator-w4-b3b22c-kserve-56bdc45497-rrcv7"
Sep 19 01:27:12.246714 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:12.246505    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33e953a5-9fa2-47ce-b518-8c2752d02da2-home\") pod \"simulator-w4-b3b22c-kserve-56bdc45497-rrcv7\" (UID: \"33e953a5-9fa2-47ce-b518-8c2752d02da2\") " pod="e2e-models-e2e-worker-w4-b3b22c/simulator-w4-b3b22c-kserve-56bdc45497-rrcv7"
Sep 19 01:27:12.246714 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:12.246538    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33e953a5-9fa2-47ce-b518-8c2752d02da2-dshm\") pod \"simulator-w4-b3b22c-kserve-56bdc45497-rrcv7\" (UID: \"33e953a5-9fa2-47ce-b518-8c2752d02da2\") " pod="e2e-models-e2e-worker-w4-b3b22c/simulator-w4-b3b22c-kserve-56bdc45497-rrcv7"
Sep 19 01:27:12.246953 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:12.246760    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33e953a5-9fa2-47ce-b518-8c2752d02da2-tmp-dir\") pod \"simulator-w4-b3b22c-kserve-56bdc45497-rrcv7\" (UID: \"33e953a5-9fa2-47ce-b518-8c2752d02da2\") " pod="e2e-models-e2e-worker-w4-b3b22c/simulator-w4-b3b22c-kserve-56bdc45497-rrcv7"
Sep 19 01:27:12.246953 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:12.246803    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33e953a5-9fa2-47ce-b518-8c2752d02da2-model-cache\") pod \"simulator-w4-b3b22c-kserve-56bdc45497-rrcv7\" (UID: \"33e953a5-9fa2-47ce-b518-8c2752d02da2\") " pod="e2e-models-e2e-worker-w4-b3b22c/simulator-w4-b3b22c-kserve-56bdc45497-rrcv7"
Sep 19 01:27:12.246953 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:12.246936    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33e953a5-9fa2-47ce-b518-8c2752d02da2-home\") pod \"simulator-w4-b3b22c-kserve-56bdc45497-rrcv7\" (UID: \"33e953a5-9fa2-47ce-b518-8c2752d02da2\") " pod="e2e-models-e2e-worker-w4-b3b22c/simulator-w4-b3b22c-kserve-56bdc45497-rrcv7"
Sep 19 01:27:12.248929 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:12.248909    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33e953a5-9fa2-47ce-b518-8c2752d02da2-dshm\") pod \"simulator-w4-b3b22c-kserve-56bdc45497-rrcv7\" (UID: \"33e953a5-9fa2-47ce-b518-8c2752d02da2\") " pod="e2e-models-e2e-worker-w4-b3b22c/simulator-w4-b3b22c-kserve-56bdc45497-rrcv7"
Sep 19 01:27:12.249044 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:12.248955    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33e953a5-9fa2-47ce-b518-8c2752d02da2-tls-certs\") pod \"simulator-w4-b3b22c-kserve-56bdc45497-rrcv7\" (UID: \"33e953a5-9fa2-47ce-b518-8c2752d02da2\") " pod="e2e-models-e2e-worker-w4-b3b22c/simulator-w4-b3b22c-kserve-56bdc45497-rrcv7"
Sep 19 01:27:12.255141 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:12.255119    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qc4bz\" (UniqueName: \"kubernetes.io/projected/33e953a5-9fa2-47ce-b518-8c2752d02da2-kube-api-access-qc4bz\") pod \"simulator-w4-b3b22c-kserve-56bdc45497-rrcv7\" (UID: \"33e953a5-9fa2-47ce-b518-8c2752d02da2\") " pod="e2e-models-e2e-worker-w4-b3b22c/simulator-w4-b3b22c-kserve-56bdc45497-rrcv7"
Sep 19 01:27:12.321864 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:12.321845    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b3b22c/simulator-w4-b3b22c-kserve-56bdc45497-rrcv7"
Sep 19 01:27:12.592533 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:12.592512    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b3b22c/simulator-w4-b3b22c-kserve-56bdc45497-rrcv7"]
Sep 19 01:27:12.594577 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:27:12.594547    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod33e953a5_9fa2_47ce_b518_8c2752d02da2.slice/crio-5e0d6ceed7f75a6f628ca9f5a65f318ee3bdff7aeede45f5a9df9517b221dac8 WatchSource:0}: Error finding container 5e0d6ceed7f75a6f628ca9f5a65f318ee3bdff7aeede45f5a9df9517b221dac8: Status 404 returned error can't find the container with id 5e0d6ceed7f75a6f628ca9f5a65f318ee3bdff7aeede45f5a9df9517b221dac8
Sep 19 01:27:12.807347 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:12.807288    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b3b22c/simulator-w4-b3b22c-kserve-56bdc45497-rrcv7" event={"ID":"33e953a5-9fa2-47ce-b518-8c2752d02da2","Type":"ContainerStarted","Data":"1b4c63ae45bf05e93f63a774b79dcdbb4080e6feba00e7a5bd9fd7399bcd82a8"}
Sep 19 01:27:12.807502 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:12.807379    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b3b22c/simulator-w4-b3b22c-kserve-56bdc45497-rrcv7" event={"ID":"33e953a5-9fa2-47ce-b518-8c2752d02da2","Type":"ContainerStarted","Data":"5e0d6ceed7f75a6f628ca9f5a65f318ee3bdff7aeede45f5a9df9517b221dac8"}
Sep 19 01:27:12.825643 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:12.825593    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-b3b22c/simulator-w4-b3b22c-kserve-56bdc45497-rrcv7" podStartSLOduration=1.8255794349999999 podStartE2EDuration="1.825579435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 01:27:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 01:27:12.824531695 +0000 UTC m=+883.712200785" watchObservedRunningTime="2026-09-19 01:27:12.825579435 +0000 UTC m=+883.713248571"
Sep 19 01:27:17.617344 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:17.617310    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f8c7c101-bx5rk"]
Sep 19 01:27:17.620559 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:17.620542    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f8c7c101-bx5rk"
Sep 19 01:27:17.622909 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:17.622891    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 19 01:27:17.622991 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:17.622898    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s7pvt\""
Sep 19 01:27:17.631140 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:17.631116    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f8c7c101-bx5rk"]
Sep 19 01:27:17.787689 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:17.787656    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc1e818d-16b6-4376-9629-5cf1773d087d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f8c7c101-bx5rk\" (UID: \"bc1e818d-16b6-4376-9629-5cf1773d087d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f8c7c101-bx5rk"
Sep 19 01:27:17.888808 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:17.888717    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc1e818d-16b6-4376-9629-5cf1773d087d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f8c7c101-bx5rk\" (UID: \"bc1e818d-16b6-4376-9629-5cf1773d087d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f8c7c101-bx5rk"
Sep 19 01:27:17.889336 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:17.889317    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc1e818d-16b6-4376-9629-5cf1773d087d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f8c7c101-bx5rk\" (UID: \"bc1e818d-16b6-4376-9629-5cf1773d087d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f8c7c101-bx5rk"
Sep 19 01:27:17.931745 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:17.931727    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f8c7c101-bx5rk"
Sep 19 01:27:18.224176 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:18.224152    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f8c7c101-bx5rk"]
Sep 19 01:27:18.226213 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:27:18.226187    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc1e818d_16b6_4376_9629_5cf1773d087d.slice/crio-48e1becc9adbb35723fecda7008df8ebfb52f13f03bee6ede99d1ee3f5cd3a4d WatchSource:0}: Error finding container 48e1becc9adbb35723fecda7008df8ebfb52f13f03bee6ede99d1ee3f5cd3a4d: Status 404 returned error can't find the container with id 48e1becc9adbb35723fecda7008df8ebfb52f13f03bee6ede99d1ee3f5cd3a4d
Sep 19 01:27:18.835850 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:18.835820    2581 generic.go:356] "Generic (PLEG): container finished" podID="bc1e818d-16b6-4376-9629-5cf1773d087d" containerID="f85ef3e5a3717f6d76bbb9b2a0f146cba82e66b6cdcdba81612a604e0cd70fc1" exitCode=0
Sep 19 01:27:18.836179 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:18.835905    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f8c7c101-bx5rk" event={"ID":"bc1e818d-16b6-4376-9629-5cf1773d087d","Type":"ContainerDied","Data":"f85ef3e5a3717f6d76bbb9b2a0f146cba82e66b6cdcdba81612a604e0cd70fc1"}
Sep 19 01:27:18.836179 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:18.835950    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f8c7c101-bx5rk" event={"ID":"bc1e818d-16b6-4376-9629-5cf1773d087d","Type":"ContainerStarted","Data":"48e1becc9adbb35723fecda7008df8ebfb52f13f03bee6ede99d1ee3f5cd3a4d"}
Sep 19 01:27:19.698308 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:19.698273    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c9nsz"]
Sep 19 01:27:19.975758 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:19.975732    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f8c7c101-bx5rk"
Sep 19 01:27:20.105179 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:20.105153    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc1e818d-16b6-4376-9629-5cf1773d087d-maas-api-service-ca\") pod \"bc1e818d-16b6-4376-9629-5cf1773d087d\" (UID: \"bc1e818d-16b6-4376-9629-5cf1773d087d\") "
Sep 19 01:27:20.105484 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:20.105462    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc1e818d-16b6-4376-9629-5cf1773d087d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bc1e818d-16b6-4376-9629-5cf1773d087d" (UID: "bc1e818d-16b6-4376-9629-5cf1773d087d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 01:27:20.206270 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:20.206185    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc1e818d-16b6-4376-9629-5cf1773d087d-maas-api-service-ca\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:27:20.846519 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:20.846487    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f8c7c101-bx5rk"
Sep 19 01:27:20.846668 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:20.846516    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f8c7c101-bx5rk" event={"ID":"bc1e818d-16b6-4376-9629-5cf1773d087d","Type":"ContainerDied","Data":"48e1becc9adbb35723fecda7008df8ebfb52f13f03bee6ede99d1ee3f5cd3a4d"}
Sep 19 01:27:20.846668 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:20.846564    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="48e1becc9adbb35723fecda7008df8ebfb52f13f03bee6ede99d1ee3f5cd3a4d"
Sep 19 01:27:22.322813 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:22.322775    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b3b22c/simulator-w4-b3b22c-kserve-56bdc45497-rrcv7"
Sep 19 01:27:22.323228 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:22.322828    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-b3b22c/simulator-w4-b3b22c-kserve-56bdc45497-rrcv7"
Sep 19 01:27:22.335309 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:22.335285    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-b3b22c/simulator-w4-b3b22c-kserve-56bdc45497-rrcv7"
Sep 19 01:27:22.867354 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:22.867327    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b3b22c/simulator-w4-b3b22c-kserve-56bdc45497-rrcv7"
Sep 19 01:27:29.048670 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:29.048639    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f8c7c101-9d765647c-lfxvq"]
Sep 19 01:27:29.049140 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:29.048974    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-f8c7c101-9d765647c-lfxvq" podUID="625b7cd5-db6c-4dc6-88a7-2a000d2cc0b9" containerName="payload-processing" containerID="cri-o://fffabb773b03f14d29786b526e330ddeee7ed15830da05a1ba961f84023aa668" gracePeriod=30
Sep 19 01:27:29.229176 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:29.229148    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f8c7c101-bx5rk"]
Sep 19 01:27:29.232536 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:29.232505    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f8c7c101-bx5rk"]
Sep 19 01:27:29.274062 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:29.274026    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc1e818d-16b6-4376-9629-5cf1773d087d" path="/var/lib/kubelet/pods/bc1e818d-16b6-4376-9629-5cf1773d087d/volumes"
Sep 19 01:27:29.307513 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:29.307459    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f8c7c101-9d765647c-lfxvq"
Sep 19 01:27:29.377106 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:29.377086    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/625b7cd5-db6c-4dc6-88a7-2a000d2cc0b9-plugins-config-volume\") pod \"625b7cd5-db6c-4dc6-88a7-2a000d2cc0b9\" (UID: \"625b7cd5-db6c-4dc6-88a7-2a000d2cc0b9\") "
Sep 19 01:27:29.377212 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:29.377113    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f8d9f\" (UniqueName: \"kubernetes.io/projected/625b7cd5-db6c-4dc6-88a7-2a000d2cc0b9-kube-api-access-f8d9f\") pod \"625b7cd5-db6c-4dc6-88a7-2a000d2cc0b9\" (UID: \"625b7cd5-db6c-4dc6-88a7-2a000d2cc0b9\") "
Sep 19 01:27:29.377484 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:29.377452    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/625b7cd5-db6c-4dc6-88a7-2a000d2cc0b9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "625b7cd5-db6c-4dc6-88a7-2a000d2cc0b9" (UID: "625b7cd5-db6c-4dc6-88a7-2a000d2cc0b9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 01:27:29.379449 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:29.379422    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/625b7cd5-db6c-4dc6-88a7-2a000d2cc0b9-kube-api-access-f8d9f" (OuterVolumeSpecName: "kube-api-access-f8d9f") pod "625b7cd5-db6c-4dc6-88a7-2a000d2cc0b9" (UID: "625b7cd5-db6c-4dc6-88a7-2a000d2cc0b9"). InnerVolumeSpecName "kube-api-access-f8d9f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 01:27:29.477707 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:29.477675    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8d9f\" (UniqueName: \"kubernetes.io/projected/625b7cd5-db6c-4dc6-88a7-2a000d2cc0b9-kube-api-access-f8d9f\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:27:29.477707 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:29.477702    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/625b7cd5-db6c-4dc6-88a7-2a000d2cc0b9-plugins-config-volume\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:27:29.888442 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:29.888413    2581 generic.go:356] "Generic (PLEG): container finished" podID="625b7cd5-db6c-4dc6-88a7-2a000d2cc0b9" containerID="fffabb773b03f14d29786b526e330ddeee7ed15830da05a1ba961f84023aa668" exitCode=0
Sep 19 01:27:29.888607 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:29.888461    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f8c7c101-9d765647c-lfxvq"
Sep 19 01:27:29.888607 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:29.888498    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f8c7c101-9d765647c-lfxvq" event={"ID":"625b7cd5-db6c-4dc6-88a7-2a000d2cc0b9","Type":"ContainerDied","Data":"fffabb773b03f14d29786b526e330ddeee7ed15830da05a1ba961f84023aa668"}
Sep 19 01:27:29.888607 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:29.888552    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f8c7c101-9d765647c-lfxvq" event={"ID":"625b7cd5-db6c-4dc6-88a7-2a000d2cc0b9","Type":"ContainerDied","Data":"b2af07d4b2e9b1b39d12cb64f8f7443d81f6441f75dc5e9a13f73af4da33e444"}
Sep 19 01:27:29.888607 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:29.888575    2581 scope.go:117] "RemoveContainer" containerID="fffabb773b03f14d29786b526e330ddeee7ed15830da05a1ba961f84023aa668"
Sep 19 01:27:29.897721 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:29.897703    2581 scope.go:117] "RemoveContainer" containerID="fffabb773b03f14d29786b526e330ddeee7ed15830da05a1ba961f84023aa668"
Sep 19 01:27:29.898036 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:27:29.898018    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fffabb773b03f14d29786b526e330ddeee7ed15830da05a1ba961f84023aa668\": container with ID starting with fffabb773b03f14d29786b526e330ddeee7ed15830da05a1ba961f84023aa668 not found: ID does not exist" containerID="fffabb773b03f14d29786b526e330ddeee7ed15830da05a1ba961f84023aa668"
Sep 19 01:27:29.898089 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:29.898045    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fffabb773b03f14d29786b526e330ddeee7ed15830da05a1ba961f84023aa668"} err="failed to get container status \"fffabb773b03f14d29786b526e330ddeee7ed15830da05a1ba961f84023aa668\": rpc error: code = NotFound desc = could not find container \"fffabb773b03f14d29786b526e330ddeee7ed15830da05a1ba961f84023aa668\": container with ID starting with fffabb773b03f14d29786b526e330ddeee7ed15830da05a1ba961f84023aa668 not found: ID does not exist"
Sep 19 01:27:29.909758 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:29.909732    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f8c7c101-9d765647c-lfxvq"]
Sep 19 01:27:29.916580 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:29.916561    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f8c7c101-9d765647c-lfxvq"]
Sep 19 01:27:31.202918 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:31.202894    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x785d_f759a12f-5b6f-4924-96f2-dd5e00fc1348/ovn-acl-logging/0.log"
Sep 19 01:27:31.203347 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:31.203010    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x785d_f759a12f-5b6f-4924-96f2-dd5e00fc1348/ovn-acl-logging/0.log"
Sep 19 01:27:31.272764 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:31.272738    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="625b7cd5-db6c-4dc6-88a7-2a000d2cc0b9" path="/var/lib/kubelet/pods/625b7cd5-db6c-4dc6-88a7-2a000d2cc0b9/volumes"
Sep 19 01:27:36.393370 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:36.393337    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c9nsz"]
Sep 19 01:27:37.672396 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:37.672361    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-44f561/autores-model-44f561-kserve-7648d7f5ff-sprv7"]
Sep 19 01:27:37.672809 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:37.672684    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc1e818d-16b6-4376-9629-5cf1773d087d" containerName="revoke-keys"
Sep 19 01:27:37.672809 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:37.672695    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc1e818d-16b6-4376-9629-5cf1773d087d" containerName="revoke-keys"
Sep 19 01:27:37.672809 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:37.672705    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="625b7cd5-db6c-4dc6-88a7-2a000d2cc0b9" containerName="payload-processing"
Sep 19 01:27:37.672809 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:37.672710    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="625b7cd5-db6c-4dc6-88a7-2a000d2cc0b9" containerName="payload-processing"
Sep 19 01:27:37.672809 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:37.672765    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc1e818d-16b6-4376-9629-5cf1773d087d" containerName="revoke-keys"
Sep 19 01:27:37.672809 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:37.672772    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="625b7cd5-db6c-4dc6-88a7-2a000d2cc0b9" containerName="payload-processing"
Sep 19 01:27:37.676986 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:37.676969    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-44f561/autores-model-44f561-kserve-7648d7f5ff-sprv7"
Sep 19 01:27:37.679583 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:37.679559    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-44f561\"/\"kube-root-ca.crt\""
Sep 19 01:27:37.680468 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:37.680446    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-44f561\"/\"default-dockercfg-8dc8m\""
Sep 19 01:27:37.680568 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:37.680446    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-44f561\"/\"openshift-service-ca.crt\""
Sep 19 01:27:37.680680 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:37.680656    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-44f561\"/\"autores-model-44f561-kserve-self-signed-certs\""
Sep 19 01:27:37.686201 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:37.686180    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-44f561/autores-model-44f561-kserve-7648d7f5ff-sprv7"]
Sep 19 01:27:37.744186 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:37.744160    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95a241ff-482e-43c3-9535-f027fa120475-model-cache\") pod \"autores-model-44f561-kserve-7648d7f5ff-sprv7\" (UID: \"95a241ff-482e-43c3-9535-f027fa120475\") " pod="ai-tenant-e2e-autores-44f561/autores-model-44f561-kserve-7648d7f5ff-sprv7"
Sep 19 01:27:37.744280 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:37.744202    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95a241ff-482e-43c3-9535-f027fa120475-dshm\") pod \"autores-model-44f561-kserve-7648d7f5ff-sprv7\" (UID: \"95a241ff-482e-43c3-9535-f027fa120475\") " pod="ai-tenant-e2e-autores-44f561/autores-model-44f561-kserve-7648d7f5ff-sprv7"
Sep 19 01:27:37.744280 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:37.744220    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95a241ff-482e-43c3-9535-f027fa120475-tls-certs\") pod \"autores-model-44f561-kserve-7648d7f5ff-sprv7\" (UID: \"95a241ff-482e-43c3-9535-f027fa120475\") " pod="ai-tenant-e2e-autores-44f561/autores-model-44f561-kserve-7648d7f5ff-sprv7"
Sep 19 01:27:37.744280 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:37.744243    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95a241ff-482e-43c3-9535-f027fa120475-tmp-dir\") pod \"autores-model-44f561-kserve-7648d7f5ff-sprv7\" (UID: \"95a241ff-482e-43c3-9535-f027fa120475\") " pod="ai-tenant-e2e-autores-44f561/autores-model-44f561-kserve-7648d7f5ff-sprv7"
Sep 19 01:27:37.744280 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:37.744274    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95a241ff-482e-43c3-9535-f027fa120475-home\") pod \"autores-model-44f561-kserve-7648d7f5ff-sprv7\" (UID: \"95a241ff-482e-43c3-9535-f027fa120475\") " pod="ai-tenant-e2e-autores-44f561/autores-model-44f561-kserve-7648d7f5ff-sprv7"
Sep 19 01:27:37.744416 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:37.744297    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcq94\" (UniqueName: \"kubernetes.io/projected/95a241ff-482e-43c3-9535-f027fa120475-kube-api-access-rcq94\") pod \"autores-model-44f561-kserve-7648d7f5ff-sprv7\" (UID: \"95a241ff-482e-43c3-9535-f027fa120475\") " pod="ai-tenant-e2e-autores-44f561/autores-model-44f561-kserve-7648d7f5ff-sprv7"
Sep 19 01:27:37.845218 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:37.845190    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95a241ff-482e-43c3-9535-f027fa120475-model-cache\") pod \"autores-model-44f561-kserve-7648d7f5ff-sprv7\" (UID: \"95a241ff-482e-43c3-9535-f027fa120475\") " pod="ai-tenant-e2e-autores-44f561/autores-model-44f561-kserve-7648d7f5ff-sprv7"
Sep 19 01:27:37.845326 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:37.845269    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95a241ff-482e-43c3-9535-f027fa120475-dshm\") pod \"autores-model-44f561-kserve-7648d7f5ff-sprv7\" (UID: \"95a241ff-482e-43c3-9535-f027fa120475\") " pod="ai-tenant-e2e-autores-44f561/autores-model-44f561-kserve-7648d7f5ff-sprv7"
Sep 19 01:27:37.845326 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:37.845302    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95a241ff-482e-43c3-9535-f027fa120475-tls-certs\") pod \"autores-model-44f561-kserve-7648d7f5ff-sprv7\" (UID: \"95a241ff-482e-43c3-9535-f027fa120475\") " pod="ai-tenant-e2e-autores-44f561/autores-model-44f561-kserve-7648d7f5ff-sprv7"
Sep 19 01:27:37.845466 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:37.845329    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95a241ff-482e-43c3-9535-f027fa120475-tmp-dir\") pod \"autores-model-44f561-kserve-7648d7f5ff-sprv7\" (UID: \"95a241ff-482e-43c3-9535-f027fa120475\") " pod="ai-tenant-e2e-autores-44f561/autores-model-44f561-kserve-7648d7f5ff-sprv7"
Sep 19 01:27:37.845466 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:37.845363    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95a241ff-482e-43c3-9535-f027fa120475-home\") pod \"autores-model-44f561-kserve-7648d7f5ff-sprv7\" (UID: \"95a241ff-482e-43c3-9535-f027fa120475\") " pod="ai-tenant-e2e-autores-44f561/autores-model-44f561-kserve-7648d7f5ff-sprv7"
Sep 19 01:27:37.845466 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:37.845404    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcq94\" (UniqueName: \"kubernetes.io/projected/95a241ff-482e-43c3-9535-f027fa120475-kube-api-access-rcq94\") pod \"autores-model-44f561-kserve-7648d7f5ff-sprv7\" (UID: \"95a241ff-482e-43c3-9535-f027fa120475\") " pod="ai-tenant-e2e-autores-44f561/autores-model-44f561-kserve-7648d7f5ff-sprv7"
Sep 19 01:27:37.845664 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:37.845640    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95a241ff-482e-43c3-9535-f027fa120475-model-cache\") pod \"autores-model-44f561-kserve-7648d7f5ff-sprv7\" (UID: \"95a241ff-482e-43c3-9535-f027fa120475\") " pod="ai-tenant-e2e-autores-44f561/autores-model-44f561-kserve-7648d7f5ff-sprv7"
Sep 19 01:27:37.845919 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:37.845897    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95a241ff-482e-43c3-9535-f027fa120475-home\") pod \"autores-model-44f561-kserve-7648d7f5ff-sprv7\" (UID: \"95a241ff-482e-43c3-9535-f027fa120475\") " pod="ai-tenant-e2e-autores-44f561/autores-model-44f561-kserve-7648d7f5ff-sprv7"
Sep 19 01:27:37.846175 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:37.846157    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95a241ff-482e-43c3-9535-f027fa120475-tmp-dir\") pod \"autores-model-44f561-kserve-7648d7f5ff-sprv7\" (UID: \"95a241ff-482e-43c3-9535-f027fa120475\") " pod="ai-tenant-e2e-autores-44f561/autores-model-44f561-kserve-7648d7f5ff-sprv7"
Sep 19 01:27:37.848459 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:37.848435    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95a241ff-482e-43c3-9535-f027fa120475-dshm\") pod \"autores-model-44f561-kserve-7648d7f5ff-sprv7\" (UID: \"95a241ff-482e-43c3-9535-f027fa120475\") " pod="ai-tenant-e2e-autores-44f561/autores-model-44f561-kserve-7648d7f5ff-sprv7"
Sep 19 01:27:37.848569 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:37.848439    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95a241ff-482e-43c3-9535-f027fa120475-tls-certs\") pod \"autores-model-44f561-kserve-7648d7f5ff-sprv7\" (UID: \"95a241ff-482e-43c3-9535-f027fa120475\") " pod="ai-tenant-e2e-autores-44f561/autores-model-44f561-kserve-7648d7f5ff-sprv7"
Sep 19 01:27:37.854634 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:37.854611    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcq94\" (UniqueName: \"kubernetes.io/projected/95a241ff-482e-43c3-9535-f027fa120475-kube-api-access-rcq94\") pod \"autores-model-44f561-kserve-7648d7f5ff-sprv7\" (UID: \"95a241ff-482e-43c3-9535-f027fa120475\") " pod="ai-tenant-e2e-autores-44f561/autores-model-44f561-kserve-7648d7f5ff-sprv7"
Sep 19 01:27:37.986965 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:37.986885    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-44f561/autores-model-44f561-kserve-7648d7f5ff-sprv7"
Sep 19 01:27:38.258800 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:38.258777    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-44f561/autores-model-44f561-kserve-7648d7f5ff-sprv7"]
Sep 19 01:27:38.260881 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:27:38.260832    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod95a241ff_482e_43c3_9535_f027fa120475.slice/crio-869d50485e4853566b912bcae97dbec94d9d800e8ed01651b6eaa097398414d5 WatchSource:0}: Error finding container 869d50485e4853566b912bcae97dbec94d9d800e8ed01651b6eaa097398414d5: Status 404 returned error can't find the container with id 869d50485e4853566b912bcae97dbec94d9d800e8ed01651b6eaa097398414d5
Sep 19 01:27:38.933444 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:38.933408    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-44f561/autores-model-44f561-kserve-7648d7f5ff-sprv7" event={"ID":"95a241ff-482e-43c3-9535-f027fa120475","Type":"ContainerStarted","Data":"e0b7f37aa772093d12233d58dbc3c6d3c660cf149b748afe19f9bbdbb681017a"}
Sep 19 01:27:38.933821 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:38.933454    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-44f561/autores-model-44f561-kserve-7648d7f5ff-sprv7" event={"ID":"95a241ff-482e-43c3-9535-f027fa120475","Type":"ContainerStarted","Data":"869d50485e4853566b912bcae97dbec94d9d800e8ed01651b6eaa097398414d5"}
Sep 19 01:27:38.953264 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:38.953214    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-44f561/autores-model-44f561-kserve-7648d7f5ff-sprv7" podStartSLOduration=1.953198042 podStartE2EDuration="1.953198042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 01:27:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 01:27:38.951298016 +0000 UTC m=+909.838967092" watchObservedRunningTime="2026-09-19 01:27:38.953198042 +0000 UTC m=+909.840867173"
Sep 19 01:27:47.988060 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:47.988021    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-44f561/autores-model-44f561-kserve-7648d7f5ff-sprv7"
Sep 19 01:27:47.988569 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:47.988082    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-44f561/autores-model-44f561-kserve-7648d7f5ff-sprv7"
Sep 19 01:27:48.001230 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:48.001207    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-44f561/autores-model-44f561-kserve-7648d7f5ff-sprv7"
Sep 19 01:27:48.991920 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:48.991893    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-44f561/autores-model-44f561-kserve-7648d7f5ff-sprv7"
Sep 19 01:27:55.483991 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:55.483951    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c9nsz"]
Sep 19 01:27:59.376088 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:59.376049    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-c4ce82/test-model-c4ce82-kserve-7b9cfc5c47-tqrkm"]
Sep 19 01:27:59.379847 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:59.379817    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c4ce82/test-model-c4ce82-kserve-7b9cfc5c47-tqrkm"
Sep 19 01:27:59.383027 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:59.382971    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-c4ce82\"/\"openshift-service-ca.crt\""
Sep 19 01:27:59.383178 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:59.383052    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-c4ce82\"/\"test-model-c4ce82-kserve-self-signed-certs\""
Sep 19 01:27:59.383178 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:59.383056    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-c4ce82\"/\"default-dockercfg-w9lsd\""
Sep 19 01:27:59.383178 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:59.383052    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-c4ce82\"/\"kube-root-ca.crt\""
Sep 19 01:27:59.389588 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:59.389538    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-c4ce82/test-model-c4ce82-kserve-7b9cfc5c47-tqrkm"]
Sep 19 01:27:59.518124 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:59.518097    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0d58b7b-f528-444c-96fa-e6bea067bb5b-dshm\") pod \"test-model-c4ce82-kserve-7b9cfc5c47-tqrkm\" (UID: \"b0d58b7b-f528-444c-96fa-e6bea067bb5b\") " pod="ai-tenant-e2e-inf-a-c4ce82/test-model-c4ce82-kserve-7b9cfc5c47-tqrkm"
Sep 19 01:27:59.518252 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:59.518146    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0d58b7b-f528-444c-96fa-e6bea067bb5b-home\") pod \"test-model-c4ce82-kserve-7b9cfc5c47-tqrkm\" (UID: \"b0d58b7b-f528-444c-96fa-e6bea067bb5b\") " pod="ai-tenant-e2e-inf-a-c4ce82/test-model-c4ce82-kserve-7b9cfc5c47-tqrkm"
Sep 19 01:27:59.518252 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:59.518223    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0d58b7b-f528-444c-96fa-e6bea067bb5b-tmp-dir\") pod \"test-model-c4ce82-kserve-7b9cfc5c47-tqrkm\" (UID: \"b0d58b7b-f528-444c-96fa-e6bea067bb5b\") " pod="ai-tenant-e2e-inf-a-c4ce82/test-model-c4ce82-kserve-7b9cfc5c47-tqrkm"
Sep 19 01:27:59.518252 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:59.518248    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0d58b7b-f528-444c-96fa-e6bea067bb5b-tls-certs\") pod \"test-model-c4ce82-kserve-7b9cfc5c47-tqrkm\" (UID: \"b0d58b7b-f528-444c-96fa-e6bea067bb5b\") " pod="ai-tenant-e2e-inf-a-c4ce82/test-model-c4ce82-kserve-7b9cfc5c47-tqrkm"
Sep 19 01:27:59.518354 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:59.518269    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0d58b7b-f528-444c-96fa-e6bea067bb5b-model-cache\") pod \"test-model-c4ce82-kserve-7b9cfc5c47-tqrkm\" (UID: \"b0d58b7b-f528-444c-96fa-e6bea067bb5b\") " pod="ai-tenant-e2e-inf-a-c4ce82/test-model-c4ce82-kserve-7b9cfc5c47-tqrkm"
Sep 19 01:27:59.518354 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:59.518286    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w9f2\" (UniqueName: \"kubernetes.io/projected/b0d58b7b-f528-444c-96fa-e6bea067bb5b-kube-api-access-7w9f2\") pod \"test-model-c4ce82-kserve-7b9cfc5c47-tqrkm\" (UID: \"b0d58b7b-f528-444c-96fa-e6bea067bb5b\") " pod="ai-tenant-e2e-inf-a-c4ce82/test-model-c4ce82-kserve-7b9cfc5c47-tqrkm"
Sep 19 01:27:59.619372 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:59.619343    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0d58b7b-f528-444c-96fa-e6bea067bb5b-home\") pod \"test-model-c4ce82-kserve-7b9cfc5c47-tqrkm\" (UID: \"b0d58b7b-f528-444c-96fa-e6bea067bb5b\") " pod="ai-tenant-e2e-inf-a-c4ce82/test-model-c4ce82-kserve-7b9cfc5c47-tqrkm"
Sep 19 01:27:59.619569 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:59.619416    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0d58b7b-f528-444c-96fa-e6bea067bb5b-tmp-dir\") pod \"test-model-c4ce82-kserve-7b9cfc5c47-tqrkm\" (UID: \"b0d58b7b-f528-444c-96fa-e6bea067bb5b\") " pod="ai-tenant-e2e-inf-a-c4ce82/test-model-c4ce82-kserve-7b9cfc5c47-tqrkm"
Sep 19 01:27:59.619569 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:59.619535    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0d58b7b-f528-444c-96fa-e6bea067bb5b-tls-certs\") pod \"test-model-c4ce82-kserve-7b9cfc5c47-tqrkm\" (UID: \"b0d58b7b-f528-444c-96fa-e6bea067bb5b\") " pod="ai-tenant-e2e-inf-a-c4ce82/test-model-c4ce82-kserve-7b9cfc5c47-tqrkm"
Sep 19 01:27:59.619692 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:59.619587    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0d58b7b-f528-444c-96fa-e6bea067bb5b-model-cache\") pod \"test-model-c4ce82-kserve-7b9cfc5c47-tqrkm\" (UID: \"b0d58b7b-f528-444c-96fa-e6bea067bb5b\") " pod="ai-tenant-e2e-inf-a-c4ce82/test-model-c4ce82-kserve-7b9cfc5c47-tqrkm"
Sep 19 01:27:59.619692 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:59.619619    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w9f2\" (UniqueName: \"kubernetes.io/projected/b0d58b7b-f528-444c-96fa-e6bea067bb5b-kube-api-access-7w9f2\") pod \"test-model-c4ce82-kserve-7b9cfc5c47-tqrkm\" (UID: \"b0d58b7b-f528-444c-96fa-e6bea067bb5b\") " pod="ai-tenant-e2e-inf-a-c4ce82/test-model-c4ce82-kserve-7b9cfc5c47-tqrkm"
Sep 19 01:27:59.619692 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:59.619668    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0d58b7b-f528-444c-96fa-e6bea067bb5b-dshm\") pod \"test-model-c4ce82-kserve-7b9cfc5c47-tqrkm\" (UID: \"b0d58b7b-f528-444c-96fa-e6bea067bb5b\") " pod="ai-tenant-e2e-inf-a-c4ce82/test-model-c4ce82-kserve-7b9cfc5c47-tqrkm"
Sep 19 01:27:59.619845 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:59.619740    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0d58b7b-f528-444c-96fa-e6bea067bb5b-tmp-dir\") pod \"test-model-c4ce82-kserve-7b9cfc5c47-tqrkm\" (UID: \"b0d58b7b-f528-444c-96fa-e6bea067bb5b\") " pod="ai-tenant-e2e-inf-a-c4ce82/test-model-c4ce82-kserve-7b9cfc5c47-tqrkm"
Sep 19 01:27:59.619845 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:59.619774    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0d58b7b-f528-444c-96fa-e6bea067bb5b-home\") pod \"test-model-c4ce82-kserve-7b9cfc5c47-tqrkm\" (UID: \"b0d58b7b-f528-444c-96fa-e6bea067bb5b\") " pod="ai-tenant-e2e-inf-a-c4ce82/test-model-c4ce82-kserve-7b9cfc5c47-tqrkm"
Sep 19 01:27:59.619991 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:59.619920    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0d58b7b-f528-444c-96fa-e6bea067bb5b-model-cache\") pod \"test-model-c4ce82-kserve-7b9cfc5c47-tqrkm\" (UID: \"b0d58b7b-f528-444c-96fa-e6bea067bb5b\") " pod="ai-tenant-e2e-inf-a-c4ce82/test-model-c4ce82-kserve-7b9cfc5c47-tqrkm"
Sep 19 01:27:59.621952 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:59.621932    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0d58b7b-f528-444c-96fa-e6bea067bb5b-dshm\") pod \"test-model-c4ce82-kserve-7b9cfc5c47-tqrkm\" (UID: \"b0d58b7b-f528-444c-96fa-e6bea067bb5b\") " pod="ai-tenant-e2e-inf-a-c4ce82/test-model-c4ce82-kserve-7b9cfc5c47-tqrkm"
Sep 19 01:27:59.622248 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:59.622231    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0d58b7b-f528-444c-96fa-e6bea067bb5b-tls-certs\") pod \"test-model-c4ce82-kserve-7b9cfc5c47-tqrkm\" (UID: \"b0d58b7b-f528-444c-96fa-e6bea067bb5b\") " pod="ai-tenant-e2e-inf-a-c4ce82/test-model-c4ce82-kserve-7b9cfc5c47-tqrkm"
Sep 19 01:27:59.629479 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:59.629422    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w9f2\" (UniqueName: \"kubernetes.io/projected/b0d58b7b-f528-444c-96fa-e6bea067bb5b-kube-api-access-7w9f2\") pod \"test-model-c4ce82-kserve-7b9cfc5c47-tqrkm\" (UID: \"b0d58b7b-f528-444c-96fa-e6bea067bb5b\") " pod="ai-tenant-e2e-inf-a-c4ce82/test-model-c4ce82-kserve-7b9cfc5c47-tqrkm"
Sep 19 01:27:59.690097 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:59.690072    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c4ce82/test-model-c4ce82-kserve-7b9cfc5c47-tqrkm"
Sep 19 01:27:59.962239 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:27:59.962211    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-c4ce82/test-model-c4ce82-kserve-7b9cfc5c47-tqrkm"]
Sep 19 01:27:59.964604 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:27:59.964568    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb0d58b7b_f528_444c_96fa_e6bea067bb5b.slice/crio-8ed1900f0c0c622d4ade74305fb9bb4213aa9950d5778034baf3236ef042ffe8 WatchSource:0}: Error finding container 8ed1900f0c0c622d4ade74305fb9bb4213aa9950d5778034baf3236ef042ffe8: Status 404 returned error can't find the container with id 8ed1900f0c0c622d4ade74305fb9bb4213aa9950d5778034baf3236ef042ffe8
Sep 19 01:28:00.031291 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:00.031262    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c4ce82/test-model-c4ce82-kserve-7b9cfc5c47-tqrkm" event={"ID":"b0d58b7b-f528-444c-96fa-e6bea067bb5b","Type":"ContainerStarted","Data":"2972f21973002caa3c8221a4f619b3b3b96b661e951e3583aec397a6e7a5eaf7"}
Sep 19 01:28:00.031422 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:00.031307    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c4ce82/test-model-c4ce82-kserve-7b9cfc5c47-tqrkm" event={"ID":"b0d58b7b-f528-444c-96fa-e6bea067bb5b","Type":"ContainerStarted","Data":"8ed1900f0c0c622d4ade74305fb9bb4213aa9950d5778034baf3236ef042ffe8"}
Sep 19 01:28:00.047519 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:00.047472    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-c4ce82/test-model-c4ce82-kserve-7b9cfc5c47-tqrkm" podStartSLOduration=1.047455612 podStartE2EDuration="1.047455612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 01:27:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 01:28:00.047047114 +0000 UTC m=+930.934716217" watchObservedRunningTime="2026-09-19 01:28:00.047455612 +0000 UTC m=+930.935124690"
Sep 19 01:28:03.804655 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:03.804609    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55340d68-ghcmt"]
Sep 19 01:28:03.807794 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:03.807777    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55340d68-ghcmt"
Sep 19 01:28:03.810062 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:03.810039    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s7pvt\""
Sep 19 01:28:03.810062 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:03.810046    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 19 01:28:03.815592 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:03.815454    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55340d68-ghcmt"]
Sep 19 01:28:03.959636 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:03.959611    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a158c53-ce7b-4cf2-8aa0-fbcacd426ab4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-55340d68-ghcmt\" (UID: \"1a158c53-ce7b-4cf2-8aa0-fbcacd426ab4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55340d68-ghcmt"
Sep 19 01:28:04.060453 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:04.060391    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a158c53-ce7b-4cf2-8aa0-fbcacd426ab4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-55340d68-ghcmt\" (UID: \"1a158c53-ce7b-4cf2-8aa0-fbcacd426ab4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55340d68-ghcmt"
Sep 19 01:28:04.060989 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:04.060972    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a158c53-ce7b-4cf2-8aa0-fbcacd426ab4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-55340d68-ghcmt\" (UID: \"1a158c53-ce7b-4cf2-8aa0-fbcacd426ab4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55340d68-ghcmt"
Sep 19 01:28:04.119798 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:04.119767    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55340d68-ghcmt"
Sep 19 01:28:04.402649 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:04.402623    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55340d68-ghcmt"]
Sep 19 01:28:04.404724 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:28:04.404691    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a158c53_ce7b_4cf2_8aa0_fbcacd426ab4.slice/crio-d9b8760b9032eba50e089cfdb13417a3b7b38d5edad47c0c05f4e477a99f1c36 WatchSource:0}: Error finding container d9b8760b9032eba50e089cfdb13417a3b7b38d5edad47c0c05f4e477a99f1c36: Status 404 returned error can't find the container with id d9b8760b9032eba50e089cfdb13417a3b7b38d5edad47c0c05f4e477a99f1c36
Sep 19 01:28:05.055802 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:05.055768    2581 generic.go:356] "Generic (PLEG): container finished" podID="1a158c53-ce7b-4cf2-8aa0-fbcacd426ab4" containerID="18d8479030ec325675d61010119209028edc2df02040a304effc386aae6e1142" exitCode=0
Sep 19 01:28:05.056140 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:05.055823    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55340d68-ghcmt" event={"ID":"1a158c53-ce7b-4cf2-8aa0-fbcacd426ab4","Type":"ContainerDied","Data":"18d8479030ec325675d61010119209028edc2df02040a304effc386aae6e1142"}
Sep 19 01:28:05.056140 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:05.055916    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55340d68-ghcmt" event={"ID":"1a158c53-ce7b-4cf2-8aa0-fbcacd426ab4","Type":"ContainerStarted","Data":"d9b8760b9032eba50e089cfdb13417a3b7b38d5edad47c0c05f4e477a99f1c36"}
Sep 19 01:28:06.187885 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:06.187849    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55340d68-ghcmt"
Sep 19 01:28:06.282329 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:06.282244    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a158c53-ce7b-4cf2-8aa0-fbcacd426ab4-maas-api-service-ca\") pod \"1a158c53-ce7b-4cf2-8aa0-fbcacd426ab4\" (UID: \"1a158c53-ce7b-4cf2-8aa0-fbcacd426ab4\") "
Sep 19 01:28:06.282536 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:06.282515    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a158c53-ce7b-4cf2-8aa0-fbcacd426ab4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1a158c53-ce7b-4cf2-8aa0-fbcacd426ab4" (UID: "1a158c53-ce7b-4cf2-8aa0-fbcacd426ab4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 01:28:06.282702 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:06.282685    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a158c53-ce7b-4cf2-8aa0-fbcacd426ab4-maas-api-service-ca\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:28:07.066350 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:07.066322    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55340d68-ghcmt"
Sep 19 01:28:07.066562 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:07.066355    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55340d68-ghcmt" event={"ID":"1a158c53-ce7b-4cf2-8aa0-fbcacd426ab4","Type":"ContainerDied","Data":"d9b8760b9032eba50e089cfdb13417a3b7b38d5edad47c0c05f4e477a99f1c36"}
Sep 19 01:28:07.066562 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:07.066406    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d9b8760b9032eba50e089cfdb13417a3b7b38d5edad47c0c05f4e477a99f1c36"
Sep 19 01:28:09.690512 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:09.690478    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-c4ce82/test-model-c4ce82-kserve-7b9cfc5c47-tqrkm"
Sep 19 01:28:09.690512 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:09.690523    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-c4ce82/test-model-c4ce82-kserve-7b9cfc5c47-tqrkm"
Sep 19 01:28:09.703111 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:09.703088    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-c4ce82/test-model-c4ce82-kserve-7b9cfc5c47-tqrkm"
Sep 19 01:28:10.090998 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:10.090922    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-c4ce82/test-model-c4ce82-kserve-7b9cfc5c47-tqrkm"
Sep 19 01:28:12.814796 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:12.814761    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-44f561-v2xdz"]
Sep 19 01:28:12.815164 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:12.815087    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a158c53-ce7b-4cf2-8aa0-fbcacd426ab4" containerName="revoke-keys"
Sep 19 01:28:12.815164 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:12.815098    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a158c53-ce7b-4cf2-8aa0-fbcacd426ab4" containerName="revoke-keys"
Sep 19 01:28:12.815241 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:12.815167    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a158c53-ce7b-4cf2-8aa0-fbcacd426ab4" containerName="revoke-keys"
Sep 19 01:28:12.818145 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:12.818129    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-44f561-v2xdz"
Sep 19 01:28:12.820545 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:12.820524    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 19 01:28:12.820659 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:12.820562    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s7pvt\""
Sep 19 01:28:12.835449 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:12.835426    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-44f561-v2xdz"]
Sep 19 01:28:12.936004 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:12.935973    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc113fbb-bc1f-4571-92a5-9257b20f27af-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-44f561-v2xdz\" (UID: \"dc113fbb-bc1f-4571-92a5-9257b20f27af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-44f561-v2xdz"
Sep 19 01:28:13.036887 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:13.036832    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc113fbb-bc1f-4571-92a5-9257b20f27af-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-44f561-v2xdz\" (UID: \"dc113fbb-bc1f-4571-92a5-9257b20f27af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-44f561-v2xdz"
Sep 19 01:28:13.037422 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:13.037402    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc113fbb-bc1f-4571-92a5-9257b20f27af-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-44f561-v2xdz\" (UID: \"dc113fbb-bc1f-4571-92a5-9257b20f27af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-44f561-v2xdz"
Sep 19 01:28:13.128485 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:13.128458    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-44f561-v2xdz"
Sep 19 01:28:13.404901 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:13.404874    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-44f561-v2xdz"]
Sep 19 01:28:13.406429 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:28:13.406402    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc113fbb_bc1f_4571_92a5_9257b20f27af.slice/crio-acabbf9c00940f070c10fb1abee293b6ef9ef7037031d7211dcccde0f64cf7f5 WatchSource:0}: Error finding container acabbf9c00940f070c10fb1abee293b6ef9ef7037031d7211dcccde0f64cf7f5: Status 404 returned error can't find the container with id acabbf9c00940f070c10fb1abee293b6ef9ef7037031d7211dcccde0f64cf7f5
Sep 19 01:28:14.104927 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:14.104898    2581 generic.go:356] "Generic (PLEG): container finished" podID="dc113fbb-bc1f-4571-92a5-9257b20f27af" containerID="4dab58502dc990e25489c186901be18f0d00dadce0d01b65ef98517f873aec2a" exitCode=0
Sep 19 01:28:14.105208 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:14.104961    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-44f561-v2xdz" event={"ID":"dc113fbb-bc1f-4571-92a5-9257b20f27af","Type":"ContainerDied","Data":"4dab58502dc990e25489c186901be18f0d00dadce0d01b65ef98517f873aec2a"}
Sep 19 01:28:14.105208 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:14.105002    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-44f561-v2xdz" event={"ID":"dc113fbb-bc1f-4571-92a5-9257b20f27af","Type":"ContainerStarted","Data":"acabbf9c00940f070c10fb1abee293b6ef9ef7037031d7211dcccde0f64cf7f5"}
Sep 19 01:28:15.250509 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:15.250485    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-44f561-v2xdz"
Sep 19 01:28:15.352264 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:15.352223    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc113fbb-bc1f-4571-92a5-9257b20f27af-maas-api-service-ca\") pod \"dc113fbb-bc1f-4571-92a5-9257b20f27af\" (UID: \"dc113fbb-bc1f-4571-92a5-9257b20f27af\") "
Sep 19 01:28:15.352576 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:15.352552    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc113fbb-bc1f-4571-92a5-9257b20f27af-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "dc113fbb-bc1f-4571-92a5-9257b20f27af" (UID: "dc113fbb-bc1f-4571-92a5-9257b20f27af"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 01:28:15.453759 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:15.453691    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc113fbb-bc1f-4571-92a5-9257b20f27af-maas-api-service-ca\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:28:16.122832 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:16.122793    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-44f561-v2xdz" event={"ID":"dc113fbb-bc1f-4571-92a5-9257b20f27af","Type":"ContainerDied","Data":"acabbf9c00940f070c10fb1abee293b6ef9ef7037031d7211dcccde0f64cf7f5"}
Sep 19 01:28:16.122832 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:16.122819    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-44f561-v2xdz"
Sep 19 01:28:16.123063 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:16.122843    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="acabbf9c00940f070c10fb1abee293b6ef9ef7037031d7211dcccde0f64cf7f5"
Sep 19 01:28:19.526009 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:19.525973    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55340d68-ghcmt"]
Sep 19 01:28:19.531731 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:19.531701    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55340d68-ghcmt"]
Sep 19 01:28:20.988589 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:20.988550    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c9nsz"]
Sep 19 01:28:21.272199 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:21.272125    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a158c53-ce7b-4cf2-8aa0-fbcacd426ab4" path="/var/lib/kubelet/pods/1a158c53-ce7b-4cf2-8aa0-fbcacd426ab4/volumes"
Sep 19 01:28:25.094285 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:25.094248    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-44f561-6c866756fc-kvlmv"]
Sep 19 01:28:25.094732 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:25.094503    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-44f561-6c866756fc-kvlmv" podUID="caf3e8d0-a8b8-40c3-bb8f-565d371660ca" containerName="payload-processing" containerID="cri-o://435909db250765e55c86022bb28a3f08c9a5ab59dc986953102c70d9732c584a" gracePeriod=30
Sep 19 01:28:25.104211 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:25.104185    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-44f561-768b76957d-975fv"]
Sep 19 01:28:25.104421 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:25.104384    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-44f561-768b76957d-975fv" podUID="856b0a82-2bcf-43a7-82d9-fef3b1b1769b" containerName="payload-pre-processing" containerID="cri-o://c9e51d69ef35ec1cf53bd737f3ae47a22a1322f0240b13625decb339ef061632" gracePeriod=30
Sep 19 01:28:25.380137 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:25.380112    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-44f561-6c866756fc-kvlmv"
Sep 19 01:28:25.382738 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:25.382713    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-44f561-768b76957d-975fv"
Sep 19 01:28:25.436980 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:25.436945    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/caf3e8d0-a8b8-40c3-bb8f-565d371660ca-plugins-config-volume\") pod \"caf3e8d0-a8b8-40c3-bb8f-565d371660ca\" (UID: \"caf3e8d0-a8b8-40c3-bb8f-565d371660ca\") "
Sep 19 01:28:25.437127 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:25.437015    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kcbxm\" (UniqueName: \"kubernetes.io/projected/856b0a82-2bcf-43a7-82d9-fef3b1b1769b-kube-api-access-kcbxm\") pod \"856b0a82-2bcf-43a7-82d9-fef3b1b1769b\" (UID: \"856b0a82-2bcf-43a7-82d9-fef3b1b1769b\") "
Sep 19 01:28:25.437127 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:25.437056    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pkn9z\" (UniqueName: \"kubernetes.io/projected/caf3e8d0-a8b8-40c3-bb8f-565d371660ca-kube-api-access-pkn9z\") pod \"caf3e8d0-a8b8-40c3-bb8f-565d371660ca\" (UID: \"caf3e8d0-a8b8-40c3-bb8f-565d371660ca\") "
Sep 19 01:28:25.437127 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:25.437099    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/856b0a82-2bcf-43a7-82d9-fef3b1b1769b-plugins-config-volume\") pod \"856b0a82-2bcf-43a7-82d9-fef3b1b1769b\" (UID: \"856b0a82-2bcf-43a7-82d9-fef3b1b1769b\") "
Sep 19 01:28:25.437476 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:25.437445    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/caf3e8d0-a8b8-40c3-bb8f-565d371660ca-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "caf3e8d0-a8b8-40c3-bb8f-565d371660ca" (UID: "caf3e8d0-a8b8-40c3-bb8f-565d371660ca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 01:28:25.437597 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:25.437523    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/856b0a82-2bcf-43a7-82d9-fef3b1b1769b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "856b0a82-2bcf-43a7-82d9-fef3b1b1769b" (UID: "856b0a82-2bcf-43a7-82d9-fef3b1b1769b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 01:28:25.439348 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:25.439323    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/856b0a82-2bcf-43a7-82d9-fef3b1b1769b-kube-api-access-kcbxm" (OuterVolumeSpecName: "kube-api-access-kcbxm") pod "856b0a82-2bcf-43a7-82d9-fef3b1b1769b" (UID: "856b0a82-2bcf-43a7-82d9-fef3b1b1769b"). InnerVolumeSpecName "kube-api-access-kcbxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 01:28:25.439452 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:25.439432    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caf3e8d0-a8b8-40c3-bb8f-565d371660ca-kube-api-access-pkn9z" (OuterVolumeSpecName: "kube-api-access-pkn9z") pod "caf3e8d0-a8b8-40c3-bb8f-565d371660ca" (UID: "caf3e8d0-a8b8-40c3-bb8f-565d371660ca"). InnerVolumeSpecName "kube-api-access-pkn9z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 01:28:25.459843 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:25.459814    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-44f561-v2xdz"]
Sep 19 01:28:25.466195 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:25.466169    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-44f561-v2xdz"]
Sep 19 01:28:25.537745 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:25.537717    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcbxm\" (UniqueName: \"kubernetes.io/projected/856b0a82-2bcf-43a7-82d9-fef3b1b1769b-kube-api-access-kcbxm\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:28:25.537745 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:25.537743    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkn9z\" (UniqueName: \"kubernetes.io/projected/caf3e8d0-a8b8-40c3-bb8f-565d371660ca-kube-api-access-pkn9z\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:28:25.537923 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:25.537756    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/856b0a82-2bcf-43a7-82d9-fef3b1b1769b-plugins-config-volume\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:28:25.537923 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:25.537770    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/caf3e8d0-a8b8-40c3-bb8f-565d371660ca-plugins-config-volume\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:28:26.170666 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:26.170629    2581 generic.go:356] "Generic (PLEG): container finished" podID="caf3e8d0-a8b8-40c3-bb8f-565d371660ca" containerID="435909db250765e55c86022bb28a3f08c9a5ab59dc986953102c70d9732c584a" exitCode=0
Sep 19 01:28:26.171119 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:26.170685    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-44f561-6c866756fc-kvlmv"
Sep 19 01:28:26.171119 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:26.170721    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-44f561-6c866756fc-kvlmv" event={"ID":"caf3e8d0-a8b8-40c3-bb8f-565d371660ca","Type":"ContainerDied","Data":"435909db250765e55c86022bb28a3f08c9a5ab59dc986953102c70d9732c584a"}
Sep 19 01:28:26.171119 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:26.170814    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-44f561-6c866756fc-kvlmv" event={"ID":"caf3e8d0-a8b8-40c3-bb8f-565d371660ca","Type":"ContainerDied","Data":"6d3ecb0548acfc2ed422b625127657f7ed329c77a1257154276e698c91394e6e"}
Sep 19 01:28:26.171119 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:26.170848    2581 scope.go:117] "RemoveContainer" containerID="435909db250765e55c86022bb28a3f08c9a5ab59dc986953102c70d9732c584a"
Sep 19 01:28:26.172341 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:26.172242    2581 generic.go:356] "Generic (PLEG): container finished" podID="856b0a82-2bcf-43a7-82d9-fef3b1b1769b" containerID="c9e51d69ef35ec1cf53bd737f3ae47a22a1322f0240b13625decb339ef061632" exitCode=0
Sep 19 01:28:26.172341 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:26.172315    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-44f561-768b76957d-975fv"
Sep 19 01:28:26.172341 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:26.172322    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-44f561-768b76957d-975fv" event={"ID":"856b0a82-2bcf-43a7-82d9-fef3b1b1769b","Type":"ContainerDied","Data":"c9e51d69ef35ec1cf53bd737f3ae47a22a1322f0240b13625decb339ef061632"}
Sep 19 01:28:26.172466 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:26.172362    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-44f561-768b76957d-975fv" event={"ID":"856b0a82-2bcf-43a7-82d9-fef3b1b1769b","Type":"ContainerDied","Data":"a9e29d31dbf57c41757159904a0d4fcf151cacdb12c8ad328762118d48047f05"}
Sep 19 01:28:26.182616 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:26.182590    2581 scope.go:117] "RemoveContainer" containerID="435909db250765e55c86022bb28a3f08c9a5ab59dc986953102c70d9732c584a"
Sep 19 01:28:26.182888 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:28:26.182847    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"435909db250765e55c86022bb28a3f08c9a5ab59dc986953102c70d9732c584a\": container with ID starting with 435909db250765e55c86022bb28a3f08c9a5ab59dc986953102c70d9732c584a not found: ID does not exist" containerID="435909db250765e55c86022bb28a3f08c9a5ab59dc986953102c70d9732c584a"
Sep 19 01:28:26.182986 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:26.182895    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"435909db250765e55c86022bb28a3f08c9a5ab59dc986953102c70d9732c584a"} err="failed to get container status \"435909db250765e55c86022bb28a3f08c9a5ab59dc986953102c70d9732c584a\": rpc error: code = NotFound desc = could not find container \"435909db250765e55c86022bb28a3f08c9a5ab59dc986953102c70d9732c584a\": container with ID starting with 435909db250765e55c86022bb28a3f08c9a5ab59dc986953102c70d9732c584a not found: ID does not exist"
Sep 19 01:28:26.182986 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:26.182919    2581 scope.go:117] "RemoveContainer" containerID="c9e51d69ef35ec1cf53bd737f3ae47a22a1322f0240b13625decb339ef061632"
Sep 19 01:28:26.191035 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:26.191017    2581 scope.go:117] "RemoveContainer" containerID="c9e51d69ef35ec1cf53bd737f3ae47a22a1322f0240b13625decb339ef061632"
Sep 19 01:28:26.191299 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:28:26.191283    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9e51d69ef35ec1cf53bd737f3ae47a22a1322f0240b13625decb339ef061632\": container with ID starting with c9e51d69ef35ec1cf53bd737f3ae47a22a1322f0240b13625decb339ef061632 not found: ID does not exist" containerID="c9e51d69ef35ec1cf53bd737f3ae47a22a1322f0240b13625decb339ef061632"
Sep 19 01:28:26.191342 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:26.191306    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9e51d69ef35ec1cf53bd737f3ae47a22a1322f0240b13625decb339ef061632"} err="failed to get container status \"c9e51d69ef35ec1cf53bd737f3ae47a22a1322f0240b13625decb339ef061632\": rpc error: code = NotFound desc = could not find container \"c9e51d69ef35ec1cf53bd737f3ae47a22a1322f0240b13625decb339ef061632\": container with ID starting with c9e51d69ef35ec1cf53bd737f3ae47a22a1322f0240b13625decb339ef061632 not found: ID does not exist"
Sep 19 01:28:26.202117 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:26.202092    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-44f561-768b76957d-975fv"]
Sep 19 01:28:26.216061 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:26.216037    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-44f561-768b76957d-975fv"]
Sep 19 01:28:26.243603 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:26.243579    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-44f561-6c866756fc-kvlmv"]
Sep 19 01:28:26.248088 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:26.248067    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-44f561-6c866756fc-kvlmv"]
Sep 19 01:28:27.272446 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:27.272404    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="856b0a82-2bcf-43a7-82d9-fef3b1b1769b" path="/var/lib/kubelet/pods/856b0a82-2bcf-43a7-82d9-fef3b1b1769b/volumes"
Sep 19 01:28:27.272954 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:27.272935    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="caf3e8d0-a8b8-40c3-bb8f-565d371660ca" path="/var/lib/kubelet/pods/caf3e8d0-a8b8-40c3-bb8f-565d371660ca/volumes"
Sep 19 01:28:27.273286 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:27.273269    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc113fbb-bc1f-4571-92a5-9257b20f27af" path="/var/lib/kubelet/pods/dc113fbb-bc1f-4571-92a5-9257b20f27af/volumes"
Sep 19 01:28:30.846691 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:30.846660    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-44f561/autores-model-44f561-kserve-7648d7f5ff-sprv7"]
Sep 19 01:28:40.873075 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:40.872987    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b3b22c-ptdh8"]
Sep 19 01:28:40.873597 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:40.873425    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="856b0a82-2bcf-43a7-82d9-fef3b1b1769b" containerName="payload-pre-processing"
Sep 19 01:28:40.873597 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:40.873443    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="856b0a82-2bcf-43a7-82d9-fef3b1b1769b" containerName="payload-pre-processing"
Sep 19 01:28:40.873597 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:40.873473    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="caf3e8d0-a8b8-40c3-bb8f-565d371660ca" containerName="payload-processing"
Sep 19 01:28:40.873597 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:40.873482    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="caf3e8d0-a8b8-40c3-bb8f-565d371660ca" containerName="payload-processing"
Sep 19 01:28:40.873597 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:40.873496    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc113fbb-bc1f-4571-92a5-9257b20f27af" containerName="revoke-keys"
Sep 19 01:28:40.873597 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:40.873506    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc113fbb-bc1f-4571-92a5-9257b20f27af" containerName="revoke-keys"
Sep 19 01:28:40.873597 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:40.873595    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="856b0a82-2bcf-43a7-82d9-fef3b1b1769b" containerName="payload-pre-processing"
Sep 19 01:28:40.874019 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:40.873608    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="caf3e8d0-a8b8-40c3-bb8f-565d371660ca" containerName="payload-processing"
Sep 19 01:28:40.874019 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:40.873619    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc113fbb-bc1f-4571-92a5-9257b20f27af" containerName="revoke-keys"
Sep 19 01:28:40.878170 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:40.878151    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b3b22c-ptdh8"
Sep 19 01:28:40.881245 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:40.881225    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 19 01:28:40.881313 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:40.881227    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s7pvt\""
Sep 19 01:28:40.887931 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:40.887907    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b3b22c-ptdh8"]
Sep 19 01:28:40.961465 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:40.961428    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40852f53-78ef-4f71-9332-8d4c1ca4fab3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-b3b22c-ptdh8\" (UID: \"40852f53-78ef-4f71-9332-8d4c1ca4fab3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b3b22c-ptdh8"
Sep 19 01:28:41.062292 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:41.062257    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40852f53-78ef-4f71-9332-8d4c1ca4fab3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-b3b22c-ptdh8\" (UID: \"40852f53-78ef-4f71-9332-8d4c1ca4fab3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b3b22c-ptdh8"
Sep 19 01:28:41.062901 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:41.062856    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40852f53-78ef-4f71-9332-8d4c1ca4fab3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-b3b22c-ptdh8\" (UID: \"40852f53-78ef-4f71-9332-8d4c1ca4fab3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b3b22c-ptdh8"
Sep 19 01:28:41.188558 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:41.188484    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b3b22c-ptdh8"
Sep 19 01:28:41.475705 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:41.475683    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b3b22c-ptdh8"]
Sep 19 01:28:41.477327 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:28:41.477302    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40852f53_78ef_4f71_9332_8d4c1ca4fab3.slice/crio-068169f4f6a95b44f02567cdfa4fb7306832fabc30d54eba351e586fbb9f93c4 WatchSource:0}: Error finding container 068169f4f6a95b44f02567cdfa4fb7306832fabc30d54eba351e586fbb9f93c4: Status 404 returned error can't find the container with id 068169f4f6a95b44f02567cdfa4fb7306832fabc30d54eba351e586fbb9f93c4
Sep 19 01:28:42.254900 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:42.254829    2581 generic.go:356] "Generic (PLEG): container finished" podID="40852f53-78ef-4f71-9332-8d4c1ca4fab3" containerID="e6df035147b35c8c248c3ecb2dfcd70e1691e4df378fcd5d0700a5e01eab7db5" exitCode=0
Sep 19 01:28:42.255275 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:42.254899    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b3b22c-ptdh8" event={"ID":"40852f53-78ef-4f71-9332-8d4c1ca4fab3","Type":"ContainerDied","Data":"e6df035147b35c8c248c3ecb2dfcd70e1691e4df378fcd5d0700a5e01eab7db5"}
Sep 19 01:28:42.255275 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:42.254996    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b3b22c-ptdh8" event={"ID":"40852f53-78ef-4f71-9332-8d4c1ca4fab3","Type":"ContainerStarted","Data":"068169f4f6a95b44f02567cdfa4fb7306832fabc30d54eba351e586fbb9f93c4"}
Sep 19 01:28:43.390812 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:43.390790    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b3b22c-ptdh8"
Sep 19 01:28:43.483412 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:43.483367    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c9nsz"]
Sep 19 01:28:43.483731 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:43.483709    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40852f53-78ef-4f71-9332-8d4c1ca4fab3-maas-api-service-ca\") pod \"40852f53-78ef-4f71-9332-8d4c1ca4fab3\" (UID: \"40852f53-78ef-4f71-9332-8d4c1ca4fab3\") "
Sep 19 01:28:43.484070 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:43.484048    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40852f53-78ef-4f71-9332-8d4c1ca4fab3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "40852f53-78ef-4f71-9332-8d4c1ca4fab3" (UID: "40852f53-78ef-4f71-9332-8d4c1ca4fab3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 01:28:43.585453 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:43.585377    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40852f53-78ef-4f71-9332-8d4c1ca4fab3-maas-api-service-ca\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:28:44.267969 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:44.267941    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b3b22c-ptdh8"
Sep 19 01:28:44.268147 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:44.267994    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b3b22c-ptdh8" event={"ID":"40852f53-78ef-4f71-9332-8d4c1ca4fab3","Type":"ContainerDied","Data":"068169f4f6a95b44f02567cdfa4fb7306832fabc30d54eba351e586fbb9f93c4"}
Sep 19 01:28:44.268147 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:44.268036    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="068169f4f6a95b44f02567cdfa4fb7306832fabc30d54eba351e586fbb9f93c4"
Sep 19 01:28:45.699921 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:45.699887    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b3b22c/simulator-w4-b3b22c-kserve-56bdc45497-rrcv7"]
Sep 19 01:28:45.866339 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:45.866297    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-44f561/autores-model-44f561-kserve-7648d7f5ff-sprv7" podUID="95a241ff-482e-43c3-9535-f027fa120475" containerName="main" containerID="cri-o://e0b7f37aa772093d12233d58dbc3c6d3c660cf149b748afe19f9bbdbb681017a" gracePeriod=45
Sep 19 01:28:46.112525 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:46.112503    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-44f561/autores-model-44f561-kserve-7648d7f5ff-sprv7"
Sep 19 01:28:46.208799 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:46.208768    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95a241ff-482e-43c3-9535-f027fa120475-tmp-dir\") pod \"95a241ff-482e-43c3-9535-f027fa120475\" (UID: \"95a241ff-482e-43c3-9535-f027fa120475\") "
Sep 19 01:28:46.208964 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:46.208824    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95a241ff-482e-43c3-9535-f027fa120475-dshm\") pod \"95a241ff-482e-43c3-9535-f027fa120475\" (UID: \"95a241ff-482e-43c3-9535-f027fa120475\") "
Sep 19 01:28:46.208964 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:46.208848    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95a241ff-482e-43c3-9535-f027fa120475-home\") pod \"95a241ff-482e-43c3-9535-f027fa120475\" (UID: \"95a241ff-482e-43c3-9535-f027fa120475\") "
Sep 19 01:28:46.208964 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:46.208946    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95a241ff-482e-43c3-9535-f027fa120475-model-cache\") pod \"95a241ff-482e-43c3-9535-f027fa120475\" (UID: \"95a241ff-482e-43c3-9535-f027fa120475\") "
Sep 19 01:28:46.209123 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:46.208967    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95a241ff-482e-43c3-9535-f027fa120475-tls-certs\") pod \"95a241ff-482e-43c3-9535-f027fa120475\" (UID: \"95a241ff-482e-43c3-9535-f027fa120475\") "
Sep 19 01:28:46.209123 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:46.208983    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rcq94\" (UniqueName: \"kubernetes.io/projected/95a241ff-482e-43c3-9535-f027fa120475-kube-api-access-rcq94\") pod \"95a241ff-482e-43c3-9535-f027fa120475\" (UID: \"95a241ff-482e-43c3-9535-f027fa120475\") "
Sep 19 01:28:46.209123 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:46.209041    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95a241ff-482e-43c3-9535-f027fa120475-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "95a241ff-482e-43c3-9535-f027fa120475" (UID: "95a241ff-482e-43c3-9535-f027fa120475"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 01:28:46.209299 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:46.209226    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95a241ff-482e-43c3-9535-f027fa120475-home" (OuterVolumeSpecName: "home") pod "95a241ff-482e-43c3-9535-f027fa120475" (UID: "95a241ff-482e-43c3-9535-f027fa120475"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 01:28:46.209299 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:46.209236    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95a241ff-482e-43c3-9535-f027fa120475-model-cache" (OuterVolumeSpecName: "model-cache") pod "95a241ff-482e-43c3-9535-f027fa120475" (UID: "95a241ff-482e-43c3-9535-f027fa120475"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 01:28:46.209389 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:46.209376    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95a241ff-482e-43c3-9535-f027fa120475-home\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:28:46.209426 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:46.209398    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95a241ff-482e-43c3-9535-f027fa120475-model-cache\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:28:46.209426 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:46.209414    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95a241ff-482e-43c3-9535-f027fa120475-tmp-dir\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:28:46.211177 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:46.211155    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95a241ff-482e-43c3-9535-f027fa120475-dshm" (OuterVolumeSpecName: "dshm") pod "95a241ff-482e-43c3-9535-f027fa120475" (UID: "95a241ff-482e-43c3-9535-f027fa120475"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 01:28:46.211266 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:46.211223    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95a241ff-482e-43c3-9535-f027fa120475-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "95a241ff-482e-43c3-9535-f027fa120475" (UID: "95a241ff-482e-43c3-9535-f027fa120475"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 01:28:46.211266 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:46.211248    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95a241ff-482e-43c3-9535-f027fa120475-kube-api-access-rcq94" (OuterVolumeSpecName: "kube-api-access-rcq94") pod "95a241ff-482e-43c3-9535-f027fa120475" (UID: "95a241ff-482e-43c3-9535-f027fa120475"). InnerVolumeSpecName "kube-api-access-rcq94". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 01:28:46.284038 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:46.283952    2581 generic.go:356] "Generic (PLEG): container finished" podID="95a241ff-482e-43c3-9535-f027fa120475" containerID="e0b7f37aa772093d12233d58dbc3c6d3c660cf149b748afe19f9bbdbb681017a" exitCode=0
Sep 19 01:28:46.284038 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:46.283997    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-44f561/autores-model-44f561-kserve-7648d7f5ff-sprv7"
Sep 19 01:28:46.284038 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:46.284012    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-44f561/autores-model-44f561-kserve-7648d7f5ff-sprv7" event={"ID":"95a241ff-482e-43c3-9535-f027fa120475","Type":"ContainerDied","Data":"e0b7f37aa772093d12233d58dbc3c6d3c660cf149b748afe19f9bbdbb681017a"}
Sep 19 01:28:46.284038 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:46.284039    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-44f561/autores-model-44f561-kserve-7648d7f5ff-sprv7" event={"ID":"95a241ff-482e-43c3-9535-f027fa120475","Type":"ContainerDied","Data":"869d50485e4853566b912bcae97dbec94d9d800e8ed01651b6eaa097398414d5"}
Sep 19 01:28:46.284350 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:46.284055    2581 scope.go:117] "RemoveContainer" containerID="e0b7f37aa772093d12233d58dbc3c6d3c660cf149b748afe19f9bbdbb681017a"
Sep 19 01:28:46.293366 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:46.293349    2581 scope.go:117] "RemoveContainer" containerID="e0b7f37aa772093d12233d58dbc3c6d3c660cf149b748afe19f9bbdbb681017a"
Sep 19 01:28:46.293624 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:28:46.293606    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0b7f37aa772093d12233d58dbc3c6d3c660cf149b748afe19f9bbdbb681017a\": container with ID starting with e0b7f37aa772093d12233d58dbc3c6d3c660cf149b748afe19f9bbdbb681017a not found: ID does not exist" containerID="e0b7f37aa772093d12233d58dbc3c6d3c660cf149b748afe19f9bbdbb681017a"
Sep 19 01:28:46.293666 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:46.293633    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0b7f37aa772093d12233d58dbc3c6d3c660cf149b748afe19f9bbdbb681017a"} err="failed to get container status \"e0b7f37aa772093d12233d58dbc3c6d3c660cf149b748afe19f9bbdbb681017a\": rpc error: code = NotFound desc = could not find container \"e0b7f37aa772093d12233d58dbc3c6d3c660cf149b748afe19f9bbdbb681017a\": container with ID starting with e0b7f37aa772093d12233d58dbc3c6d3c660cf149b748afe19f9bbdbb681017a not found: ID does not exist"
Sep 19 01:28:46.304340 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:46.304318    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-44f561/autores-model-44f561-kserve-7648d7f5ff-sprv7"]
Sep 19 01:28:46.308628 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:46.308605    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-44f561/autores-model-44f561-kserve-7648d7f5ff-sprv7"]
Sep 19 01:28:46.310106 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:46.310088    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95a241ff-482e-43c3-9535-f027fa120475-dshm\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:28:46.310163 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:46.310113    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95a241ff-482e-43c3-9535-f027fa120475-tls-certs\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:28:46.310163 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:46.310127    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcq94\" (UniqueName: \"kubernetes.io/projected/95a241ff-482e-43c3-9535-f027fa120475-kube-api-access-rcq94\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:28:47.272178 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:47.272138    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95a241ff-482e-43c3-9535-f027fa120475" path="/var/lib/kubelet/pods/95a241ff-482e-43c3-9535-f027fa120475/volumes"
Sep 19 01:28:51.596936 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:51.596902    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b3b22c-ptdh8"]
Sep 19 01:28:51.601875 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:51.601833    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b3b22c-ptdh8"]
Sep 19 01:28:53.272475 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:53.272442    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40852f53-78ef-4f71-9332-8d4c1ca4fab3" path="/var/lib/kubelet/pods/40852f53-78ef-4f71-9332-8d4c1ca4fab3/volumes"
Sep 19 01:28:59.395322 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:28:59.395279    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c9nsz"]
Sep 19 01:29:00.718947 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:00.718910    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-b3b22c/simulator-w4-b3b22c-kserve-56bdc45497-rrcv7" podUID="33e953a5-9fa2-47ce-b518-8c2752d02da2" containerName="main" containerID="cri-o://1b4c63ae45bf05e93f63a774b79dcdbb4080e6feba00e7a5bd9fd7399bcd82a8" gracePeriod=45
Sep 19 01:29:00.957088 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:00.957065    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b3b22c/simulator-w4-b3b22c-kserve-56bdc45497-rrcv7"
Sep 19 01:29:01.030980 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:01.030896    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qc4bz\" (UniqueName: \"kubernetes.io/projected/33e953a5-9fa2-47ce-b518-8c2752d02da2-kube-api-access-qc4bz\") pod \"33e953a5-9fa2-47ce-b518-8c2752d02da2\" (UID: \"33e953a5-9fa2-47ce-b518-8c2752d02da2\") "
Sep 19 01:29:01.030980 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:01.030958    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33e953a5-9fa2-47ce-b518-8c2752d02da2-model-cache\") pod \"33e953a5-9fa2-47ce-b518-8c2752d02da2\" (UID: \"33e953a5-9fa2-47ce-b518-8c2752d02da2\") "
Sep 19 01:29:01.031174 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:01.031036    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33e953a5-9fa2-47ce-b518-8c2752d02da2-tls-certs\") pod \"33e953a5-9fa2-47ce-b518-8c2752d02da2\" (UID: \"33e953a5-9fa2-47ce-b518-8c2752d02da2\") "
Sep 19 01:29:01.031174 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:01.031059    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33e953a5-9fa2-47ce-b518-8c2752d02da2-home\") pod \"33e953a5-9fa2-47ce-b518-8c2752d02da2\" (UID: \"33e953a5-9fa2-47ce-b518-8c2752d02da2\") "
Sep 19 01:29:01.031174 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:01.031107    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33e953a5-9fa2-47ce-b518-8c2752d02da2-tmp-dir\") pod \"33e953a5-9fa2-47ce-b518-8c2752d02da2\" (UID: \"33e953a5-9fa2-47ce-b518-8c2752d02da2\") "
Sep 19 01:29:01.031174 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:01.031161    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33e953a5-9fa2-47ce-b518-8c2752d02da2-dshm\") pod \"33e953a5-9fa2-47ce-b518-8c2752d02da2\" (UID: \"33e953a5-9fa2-47ce-b518-8c2752d02da2\") "
Sep 19 01:29:01.031437 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:01.031244    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33e953a5-9fa2-47ce-b518-8c2752d02da2-model-cache" (OuterVolumeSpecName: "model-cache") pod "33e953a5-9fa2-47ce-b518-8c2752d02da2" (UID: "33e953a5-9fa2-47ce-b518-8c2752d02da2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 01:29:01.031437 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:01.031296    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33e953a5-9fa2-47ce-b518-8c2752d02da2-home" (OuterVolumeSpecName: "home") pod "33e953a5-9fa2-47ce-b518-8c2752d02da2" (UID: "33e953a5-9fa2-47ce-b518-8c2752d02da2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 01:29:01.031544 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:01.031470    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33e953a5-9fa2-47ce-b518-8c2752d02da2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "33e953a5-9fa2-47ce-b518-8c2752d02da2" (UID: "33e953a5-9fa2-47ce-b518-8c2752d02da2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 01:29:01.031604 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:01.031543    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33e953a5-9fa2-47ce-b518-8c2752d02da2-model-cache\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:29:01.031604 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:01.031565    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33e953a5-9fa2-47ce-b518-8c2752d02da2-home\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:29:01.033295 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:01.033263    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33e953a5-9fa2-47ce-b518-8c2752d02da2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "33e953a5-9fa2-47ce-b518-8c2752d02da2" (UID: "33e953a5-9fa2-47ce-b518-8c2752d02da2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 01:29:01.033396 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:01.033361    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33e953a5-9fa2-47ce-b518-8c2752d02da2-kube-api-access-qc4bz" (OuterVolumeSpecName: "kube-api-access-qc4bz") pod "33e953a5-9fa2-47ce-b518-8c2752d02da2" (UID: "33e953a5-9fa2-47ce-b518-8c2752d02da2"). InnerVolumeSpecName "kube-api-access-qc4bz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 01:29:01.033502 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:01.033482    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33e953a5-9fa2-47ce-b518-8c2752d02da2-dshm" (OuterVolumeSpecName: "dshm") pod "33e953a5-9fa2-47ce-b518-8c2752d02da2" (UID: "33e953a5-9fa2-47ce-b518-8c2752d02da2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 01:29:01.132641 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:01.132606    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33e953a5-9fa2-47ce-b518-8c2752d02da2-tmp-dir\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:29:01.132641 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:01.132636    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33e953a5-9fa2-47ce-b518-8c2752d02da2-dshm\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:29:01.132641 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:01.132647    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qc4bz\" (UniqueName: \"kubernetes.io/projected/33e953a5-9fa2-47ce-b518-8c2752d02da2-kube-api-access-qc4bz\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:29:01.132853 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:01.132655    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33e953a5-9fa2-47ce-b518-8c2752d02da2-tls-certs\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:29:01.352833 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:01.352804    2581 generic.go:356] "Generic (PLEG): container finished" podID="33e953a5-9fa2-47ce-b518-8c2752d02da2" containerID="1b4c63ae45bf05e93f63a774b79dcdbb4080e6feba00e7a5bd9fd7399bcd82a8" exitCode=0
Sep 19 01:29:01.352981 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:01.352846    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b3b22c/simulator-w4-b3b22c-kserve-56bdc45497-rrcv7"
Sep 19 01:29:01.352981 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:01.352851    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b3b22c/simulator-w4-b3b22c-kserve-56bdc45497-rrcv7" event={"ID":"33e953a5-9fa2-47ce-b518-8c2752d02da2","Type":"ContainerDied","Data":"1b4c63ae45bf05e93f63a774b79dcdbb4080e6feba00e7a5bd9fd7399bcd82a8"}
Sep 19 01:29:01.352981 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:01.352898    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b3b22c/simulator-w4-b3b22c-kserve-56bdc45497-rrcv7" event={"ID":"33e953a5-9fa2-47ce-b518-8c2752d02da2","Type":"ContainerDied","Data":"5e0d6ceed7f75a6f628ca9f5a65f318ee3bdff7aeede45f5a9df9517b221dac8"}
Sep 19 01:29:01.352981 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:01.352914    2581 scope.go:117] "RemoveContainer" containerID="1b4c63ae45bf05e93f63a774b79dcdbb4080e6feba00e7a5bd9fd7399bcd82a8"
Sep 19 01:29:01.361343 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:01.361326    2581 scope.go:117] "RemoveContainer" containerID="1b4c63ae45bf05e93f63a774b79dcdbb4080e6feba00e7a5bd9fd7399bcd82a8"
Sep 19 01:29:01.361607 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:29:01.361588    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b4c63ae45bf05e93f63a774b79dcdbb4080e6feba00e7a5bd9fd7399bcd82a8\": container with ID starting with 1b4c63ae45bf05e93f63a774b79dcdbb4080e6feba00e7a5bd9fd7399bcd82a8 not found: ID does not exist" containerID="1b4c63ae45bf05e93f63a774b79dcdbb4080e6feba00e7a5bd9fd7399bcd82a8"
Sep 19 01:29:01.361651 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:01.361616    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b4c63ae45bf05e93f63a774b79dcdbb4080e6feba00e7a5bd9fd7399bcd82a8"} err="failed to get container status \"1b4c63ae45bf05e93f63a774b79dcdbb4080e6feba00e7a5bd9fd7399bcd82a8\": rpc error: code = NotFound desc = could not find container \"1b4c63ae45bf05e93f63a774b79dcdbb4080e6feba00e7a5bd9fd7399bcd82a8\": container with ID starting with 1b4c63ae45bf05e93f63a774b79dcdbb4080e6feba00e7a5bd9fd7399bcd82a8 not found: ID does not exist"
Sep 19 01:29:01.368688 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:01.368665    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b3b22c/simulator-w4-b3b22c-kserve-56bdc45497-rrcv7"]
Sep 19 01:29:01.371826 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:01.371804    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-b3b22c/simulator-w4-b3b22c-kserve-56bdc45497-rrcv7"]
Sep 19 01:29:03.272475 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:03.272440    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33e953a5-9fa2-47ce-b518-8c2752d02da2" path="/var/lib/kubelet/pods/33e953a5-9fa2-47ce-b518-8c2752d02da2/volumes"
Sep 19 01:29:18.645382 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:18.645351    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-44f561-openshift-default-6f6685d5d-wl7nx"]
Sep 19 01:29:18.645842 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:18.645567    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-44f561-openshift-default-6f6685d5d-wl7nx" podUID="01ff1ce5-71ca-4b46-ac86-7274ebd554c5" containerName="istio-proxy" containerID="cri-o://42be5ff584124e7a16b1c09331e74c07717429bb00aaa8a67b679f50852fa091" gracePeriod=30
Sep 19 01:29:23.894784 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:23.894748    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c9nsz"]
Sep 19 01:29:23.896024 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:23.896005    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-44f561-openshift-default-6f6685d5d-wl7nx"
Sep 19 01:29:24.008046 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:24.007980    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01ff1ce5-71ca-4b46-ac86-7274ebd554c5-workload-certs\") pod \"01ff1ce5-71ca-4b46-ac86-7274ebd554c5\" (UID: \"01ff1ce5-71ca-4b46-ac86-7274ebd554c5\") "
Sep 19 01:29:24.008046 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:24.008028    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01ff1ce5-71ca-4b46-ac86-7274ebd554c5-istiod-ca-cert\") pod \"01ff1ce5-71ca-4b46-ac86-7274ebd554c5\" (UID: \"01ff1ce5-71ca-4b46-ac86-7274ebd554c5\") "
Sep 19 01:29:24.008220 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:24.008061    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01ff1ce5-71ca-4b46-ac86-7274ebd554c5-istio-envoy\") pod \"01ff1ce5-71ca-4b46-ac86-7274ebd554c5\" (UID: \"01ff1ce5-71ca-4b46-ac86-7274ebd554c5\") "
Sep 19 01:29:24.008220 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:24.008088    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01ff1ce5-71ca-4b46-ac86-7274ebd554c5-credential-socket\") pod \"01ff1ce5-71ca-4b46-ac86-7274ebd554c5\" (UID: \"01ff1ce5-71ca-4b46-ac86-7274ebd554c5\") "
Sep 19 01:29:24.008220 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:24.008119    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9pf99\" (UniqueName: \"kubernetes.io/projected/01ff1ce5-71ca-4b46-ac86-7274ebd554c5-kube-api-access-9pf99\") pod \"01ff1ce5-71ca-4b46-ac86-7274ebd554c5\" (UID: \"01ff1ce5-71ca-4b46-ac86-7274ebd554c5\") "
Sep 19 01:29:24.008220 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:24.008086    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01ff1ce5-71ca-4b46-ac86-7274ebd554c5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "01ff1ce5-71ca-4b46-ac86-7274ebd554c5" (UID: "01ff1ce5-71ca-4b46-ac86-7274ebd554c5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 01:29:24.008220 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:24.008165    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01ff1ce5-71ca-4b46-ac86-7274ebd554c5-istio-token\") pod \"01ff1ce5-71ca-4b46-ac86-7274ebd554c5\" (UID: \"01ff1ce5-71ca-4b46-ac86-7274ebd554c5\") "
Sep 19 01:29:24.008220 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:24.008192    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01ff1ce5-71ca-4b46-ac86-7274ebd554c5-istio-data\") pod \"01ff1ce5-71ca-4b46-ac86-7274ebd554c5\" (UID: \"01ff1ce5-71ca-4b46-ac86-7274ebd554c5\") "
Sep 19 01:29:24.008511 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:24.008309    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01ff1ce5-71ca-4b46-ac86-7274ebd554c5-workload-socket\") pod \"01ff1ce5-71ca-4b46-ac86-7274ebd554c5\" (UID: \"01ff1ce5-71ca-4b46-ac86-7274ebd554c5\") "
Sep 19 01:29:24.008511 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:24.008353    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01ff1ce5-71ca-4b46-ac86-7274ebd554c5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "01ff1ce5-71ca-4b46-ac86-7274ebd554c5" (UID: "01ff1ce5-71ca-4b46-ac86-7274ebd554c5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 01:29:24.008511 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:24.008447    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01ff1ce5-71ca-4b46-ac86-7274ebd554c5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "01ff1ce5-71ca-4b46-ac86-7274ebd554c5" (UID: "01ff1ce5-71ca-4b46-ac86-7274ebd554c5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 01:29:24.008511 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:24.008457    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01ff1ce5-71ca-4b46-ac86-7274ebd554c5-istio-data" (OuterVolumeSpecName: "istio-data") pod "01ff1ce5-71ca-4b46-ac86-7274ebd554c5" (UID: "01ff1ce5-71ca-4b46-ac86-7274ebd554c5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 01:29:24.008744 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:24.008556    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01ff1ce5-71ca-4b46-ac86-7274ebd554c5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "01ff1ce5-71ca-4b46-ac86-7274ebd554c5" (UID: "01ff1ce5-71ca-4b46-ac86-7274ebd554c5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 01:29:24.008744 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:24.008611    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01ff1ce5-71ca-4b46-ac86-7274ebd554c5-istio-podinfo\") pod \"01ff1ce5-71ca-4b46-ac86-7274ebd554c5\" (UID: \"01ff1ce5-71ca-4b46-ac86-7274ebd554c5\") "
Sep 19 01:29:24.009172 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:24.009147    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01ff1ce5-71ca-4b46-ac86-7274ebd554c5-istio-data\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:29:24.009302 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:24.009177    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01ff1ce5-71ca-4b46-ac86-7274ebd554c5-workload-socket\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:29:24.009302 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:24.009194    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01ff1ce5-71ca-4b46-ac86-7274ebd554c5-workload-certs\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:29:24.009302 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:24.009208    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01ff1ce5-71ca-4b46-ac86-7274ebd554c5-istiod-ca-cert\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:29:24.009302 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:24.009223    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01ff1ce5-71ca-4b46-ac86-7274ebd554c5-credential-socket\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:29:24.010449 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:24.010425    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01ff1ce5-71ca-4b46-ac86-7274ebd554c5-istio-token" (OuterVolumeSpecName: "istio-token") pod "01ff1ce5-71ca-4b46-ac86-7274ebd554c5" (UID: "01ff1ce5-71ca-4b46-ac86-7274ebd554c5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 01:29:24.010636 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:24.010611    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01ff1ce5-71ca-4b46-ac86-7274ebd554c5-kube-api-access-9pf99" (OuterVolumeSpecName: "kube-api-access-9pf99") pod "01ff1ce5-71ca-4b46-ac86-7274ebd554c5" (UID: "01ff1ce5-71ca-4b46-ac86-7274ebd554c5"). InnerVolumeSpecName "kube-api-access-9pf99". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 01:29:24.010729 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:24.010694    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01ff1ce5-71ca-4b46-ac86-7274ebd554c5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "01ff1ce5-71ca-4b46-ac86-7274ebd554c5" (UID: "01ff1ce5-71ca-4b46-ac86-7274ebd554c5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 01:29:24.011124 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:24.011102    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/01ff1ce5-71ca-4b46-ac86-7274ebd554c5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "01ff1ce5-71ca-4b46-ac86-7274ebd554c5" (UID: "01ff1ce5-71ca-4b46-ac86-7274ebd554c5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 19 01:29:24.109806 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:24.109780    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01ff1ce5-71ca-4b46-ac86-7274ebd554c5-istio-envoy\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:29:24.109922 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:24.109810    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pf99\" (UniqueName: \"kubernetes.io/projected/01ff1ce5-71ca-4b46-ac86-7274ebd554c5-kube-api-access-9pf99\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:29:24.109922 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:24.109823    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01ff1ce5-71ca-4b46-ac86-7274ebd554c5-istio-token\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:29:24.109922 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:24.109832    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01ff1ce5-71ca-4b46-ac86-7274ebd554c5-istio-podinfo\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:29:24.464505 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:24.464480    2581 generic.go:356] "Generic (PLEG): container finished" podID="01ff1ce5-71ca-4b46-ac86-7274ebd554c5" containerID="42be5ff584124e7a16b1c09331e74c07717429bb00aaa8a67b679f50852fa091" exitCode=0
Sep 19 01:29:24.464658 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:24.464524    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-44f561-openshift-default-6f6685d5d-wl7nx"
Sep 19 01:29:24.464658 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:24.464568    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-44f561-openshift-default-6f6685d5d-wl7nx" event={"ID":"01ff1ce5-71ca-4b46-ac86-7274ebd554c5","Type":"ContainerDied","Data":"42be5ff584124e7a16b1c09331e74c07717429bb00aaa8a67b679f50852fa091"}
Sep 19 01:29:24.464658 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:24.464618    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-44f561-openshift-default-6f6685d5d-wl7nx" event={"ID":"01ff1ce5-71ca-4b46-ac86-7274ebd554c5","Type":"ContainerDied","Data":"aa6d467d1edf717972f467bec144b6567c7ede00e049748608cf6543e43a6e2a"}
Sep 19 01:29:24.464658 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:24.464639    2581 scope.go:117] "RemoveContainer" containerID="42be5ff584124e7a16b1c09331e74c07717429bb00aaa8a67b679f50852fa091"
Sep 19 01:29:24.473388 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:24.473371    2581 scope.go:117] "RemoveContainer" containerID="88ac8c99cf8de48ac4e1d27b282c98cef42a62e9e59f82ebb941b5c19f383f61"
Sep 19 01:29:24.481272 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:24.481256    2581 scope.go:117] "RemoveContainer" containerID="42be5ff584124e7a16b1c09331e74c07717429bb00aaa8a67b679f50852fa091"
Sep 19 01:29:24.481547 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:29:24.481529    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42be5ff584124e7a16b1c09331e74c07717429bb00aaa8a67b679f50852fa091\": container with ID starting with 42be5ff584124e7a16b1c09331e74c07717429bb00aaa8a67b679f50852fa091 not found: ID does not exist" containerID="42be5ff584124e7a16b1c09331e74c07717429bb00aaa8a67b679f50852fa091"
Sep 19 01:29:24.481600 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:24.481559    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42be5ff584124e7a16b1c09331e74c07717429bb00aaa8a67b679f50852fa091"} err="failed to get container status \"42be5ff584124e7a16b1c09331e74c07717429bb00aaa8a67b679f50852fa091\": rpc error: code = NotFound desc = could not find container \"42be5ff584124e7a16b1c09331e74c07717429bb00aaa8a67b679f50852fa091\": container with ID starting with 42be5ff584124e7a16b1c09331e74c07717429bb00aaa8a67b679f50852fa091 not found: ID does not exist"
Sep 19 01:29:24.481600 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:24.481577    2581 scope.go:117] "RemoveContainer" containerID="88ac8c99cf8de48ac4e1d27b282c98cef42a62e9e59f82ebb941b5c19f383f61"
Sep 19 01:29:24.481854 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:29:24.481837    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88ac8c99cf8de48ac4e1d27b282c98cef42a62e9e59f82ebb941b5c19f383f61\": container with ID starting with 88ac8c99cf8de48ac4e1d27b282c98cef42a62e9e59f82ebb941b5c19f383f61 not found: ID does not exist" containerID="88ac8c99cf8de48ac4e1d27b282c98cef42a62e9e59f82ebb941b5c19f383f61"
Sep 19 01:29:24.481928 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:24.481885    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88ac8c99cf8de48ac4e1d27b282c98cef42a62e9e59f82ebb941b5c19f383f61"} err="failed to get container status \"88ac8c99cf8de48ac4e1d27b282c98cef42a62e9e59f82ebb941b5c19f383f61\": rpc error: code = NotFound desc = could not find container \"88ac8c99cf8de48ac4e1d27b282c98cef42a62e9e59f82ebb941b5c19f383f61\": container with ID starting with 88ac8c99cf8de48ac4e1d27b282c98cef42a62e9e59f82ebb941b5c19f383f61 not found: ID does not exist"
Sep 19 01:29:24.486161 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:24.486141    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-44f561-openshift-default-6f6685d5d-wl7nx"]
Sep 19 01:29:24.494597 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:24.489827    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-44f561-openshift-default-6f6685d5d-wl7nx"]
Sep 19 01:29:25.272469 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:25.272438    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01ff1ce5-71ca-4b46-ac86-7274ebd554c5" path="/var/lib/kubelet/pods/01ff1ce5-71ca-4b46-ac86-7274ebd554c5/volumes"
Sep 19 01:29:27.136310 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:27.136274    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-cf0ace-69dc9d55d6-p6wfl"]
Sep 19 01:29:27.136654 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:27.136598    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95a241ff-482e-43c3-9535-f027fa120475" containerName="main"
Sep 19 01:29:27.136654 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:27.136610    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="95a241ff-482e-43c3-9535-f027fa120475" containerName="main"
Sep 19 01:29:27.136654 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:27.136623    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01ff1ce5-71ca-4b46-ac86-7274ebd554c5" containerName="istio-proxy"
Sep 19 01:29:27.136654 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:27.136628    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="01ff1ce5-71ca-4b46-ac86-7274ebd554c5" containerName="istio-proxy"
Sep 19 01:29:27.136654 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:27.136634    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01ff1ce5-71ca-4b46-ac86-7274ebd554c5" containerName="istio-proxy"
Sep 19 01:29:27.136654 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:27.136640    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="01ff1ce5-71ca-4b46-ac86-7274ebd554c5" containerName="istio-proxy"
Sep 19 01:29:27.136654 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:27.136650    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40852f53-78ef-4f71-9332-8d4c1ca4fab3" containerName="revoke-keys"
Sep 19 01:29:27.136654 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:27.136656    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="40852f53-78ef-4f71-9332-8d4c1ca4fab3" containerName="revoke-keys"
Sep 19 01:29:27.137093 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:27.136664    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33e953a5-9fa2-47ce-b518-8c2752d02da2" containerName="main"
Sep 19 01:29:27.137093 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:27.136669    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="33e953a5-9fa2-47ce-b518-8c2752d02da2" containerName="main"
Sep 19 01:29:27.137093 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:27.136719    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="95a241ff-482e-43c3-9535-f027fa120475" containerName="main"
Sep 19 01:29:27.137093 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:27.136726    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="40852f53-78ef-4f71-9332-8d4c1ca4fab3" containerName="revoke-keys"
Sep 19 01:29:27.137093 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:27.136732    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="33e953a5-9fa2-47ce-b518-8c2752d02da2" containerName="main"
Sep 19 01:29:27.137093 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:27.136738    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="01ff1ce5-71ca-4b46-ac86-7274ebd554c5" containerName="istio-proxy"
Sep 19 01:29:27.141034 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:27.141018    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-cf0ace-69dc9d55d6-p6wfl"
Sep 19 01:29:27.143456 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:27.143437    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-cf0ace-dockercfg-vh2xg\""
Sep 19 01:29:27.143674 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:27.143662    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-cf0ace\""
Sep 19 01:29:27.154150 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:27.154129    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-cf0ace-69dc9d55d6-p6wfl"]
Sep 19 01:29:27.235306 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:27.235275    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hv7dc\" (UniqueName: \"kubernetes.io/projected/318478b1-28c2-4c07-a3bf-83b875f6e1e1-kube-api-access-hv7dc\") pod \"payload-pre-processing-e2e-explicit-cf0ace-69dc9d55d6-p6wfl\" (UID: \"318478b1-28c2-4c07-a3bf-83b875f6e1e1\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-cf0ace-69dc9d55d6-p6wfl"
Sep 19 01:29:27.235446 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:27.235323    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/318478b1-28c2-4c07-a3bf-83b875f6e1e1-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-cf0ace-69dc9d55d6-p6wfl\" (UID: \"318478b1-28c2-4c07-a3bf-83b875f6e1e1\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-cf0ace-69dc9d55d6-p6wfl"
Sep 19 01:29:27.336353 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:27.336325    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hv7dc\" (UniqueName: \"kubernetes.io/projected/318478b1-28c2-4c07-a3bf-83b875f6e1e1-kube-api-access-hv7dc\") pod \"payload-pre-processing-e2e-explicit-cf0ace-69dc9d55d6-p6wfl\" (UID: \"318478b1-28c2-4c07-a3bf-83b875f6e1e1\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-cf0ace-69dc9d55d6-p6wfl"
Sep 19 01:29:27.336469 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:27.336369    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/318478b1-28c2-4c07-a3bf-83b875f6e1e1-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-cf0ace-69dc9d55d6-p6wfl\" (UID: \"318478b1-28c2-4c07-a3bf-83b875f6e1e1\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-cf0ace-69dc9d55d6-p6wfl"
Sep 19 01:29:27.337112 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:27.337095    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/318478b1-28c2-4c07-a3bf-83b875f6e1e1-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-cf0ace-69dc9d55d6-p6wfl\" (UID: \"318478b1-28c2-4c07-a3bf-83b875f6e1e1\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-cf0ace-69dc9d55d6-p6wfl"
Sep 19 01:29:27.345260 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:27.345240    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hv7dc\" (UniqueName: \"kubernetes.io/projected/318478b1-28c2-4c07-a3bf-83b875f6e1e1-kube-api-access-hv7dc\") pod \"payload-pre-processing-e2e-explicit-cf0ace-69dc9d55d6-p6wfl\" (UID: \"318478b1-28c2-4c07-a3bf-83b875f6e1e1\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-cf0ace-69dc9d55d6-p6wfl"
Sep 19 01:29:27.450472 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:27.450413    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-cf0ace-69dc9d55d6-p6wfl"
Sep 19 01:29:27.721063 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:27.721026    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-cf0ace-69dc9d55d6-p6wfl"]
Sep 19 01:29:27.723350 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:29:27.723307    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod318478b1_28c2_4c07_a3bf_83b875f6e1e1.slice/crio-c6473ab26a5ab8e5c39787ac2d56e6bcb69e9f424977415b181a9f94240fd959 WatchSource:0}: Error finding container c6473ab26a5ab8e5c39787ac2d56e6bcb69e9f424977415b181a9f94240fd959: Status 404 returned error can't find the container with id c6473ab26a5ab8e5c39787ac2d56e6bcb69e9f424977415b181a9f94240fd959
Sep 19 01:29:28.486641 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:28.486612    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-cf0ace-69dc9d55d6-p6wfl" event={"ID":"318478b1-28c2-4c07-a3bf-83b875f6e1e1","Type":"ContainerStarted","Data":"24ebeee39e2abdcf5d91a0924e5bcd3931b969716f8a777851da316113d35bb0"}
Sep 19 01:29:28.487093 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:28.486668    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-cf0ace-69dc9d55d6-p6wfl" event={"ID":"318478b1-28c2-4c07-a3bf-83b875f6e1e1","Type":"ContainerStarted","Data":"c6473ab26a5ab8e5c39787ac2d56e6bcb69e9f424977415b181a9f94240fd959"}
Sep 19 01:29:28.487093 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:28.486697    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-cf0ace-69dc9d55d6-p6wfl"
Sep 19 01:29:28.503226 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:28.503177    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-cf0ace-69dc9d55d6-p6wfl" podStartSLOduration=1.503160845 podStartE2EDuration="1.503160845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 01:29:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 01:29:28.502325004 +0000 UTC m=+1019.389994100" watchObservedRunningTime="2026-09-19 01:29:28.503160845 +0000 UTC m=+1019.390829922"
Sep 19 01:29:39.493734 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:39.493705    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-cf0ace-69dc9d55d6-p6wfl"
Sep 19 01:29:41.396711 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:41.396672    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c9nsz"]
Sep 19 01:29:52.899760 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:52.899727    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c9nsz"]
Sep 19 01:29:57.487088 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:57.487050    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-5a005d/test-model-5a005d-kserve-844f5dc4c9-f72bf"]
Sep 19 01:29:57.487482 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:57.487454    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="01ff1ce5-71ca-4b46-ac86-7274ebd554c5" containerName="istio-proxy"
Sep 19 01:29:57.490563 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:57.490535    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5a005d/test-model-5a005d-kserve-844f5dc4c9-f72bf"
Sep 19 01:29:57.492933 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:57.492912    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-5a005d\"/\"openshift-service-ca.crt\""
Sep 19 01:29:57.493721 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:57.493700    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-5a005d\"/\"test-model-5a005d-kserve-self-signed-certs\""
Sep 19 01:29:57.493837 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:57.493730    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-5a005d\"/\"kube-root-ca.crt\""
Sep 19 01:29:57.493837 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:57.493823    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-5a005d\"/\"default-dockercfg-qc94g\""
Sep 19 01:29:57.500083 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:57.500061    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-5a005d/test-model-5a005d-kserve-844f5dc4c9-f72bf"]
Sep 19 01:29:57.566770 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:57.566737    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29rsb\" (UniqueName: \"kubernetes.io/projected/6f17cb1a-739d-4613-998a-ad43d62e6367-kube-api-access-29rsb\") pod \"test-model-5a005d-kserve-844f5dc4c9-f72bf\" (UID: \"6f17cb1a-739d-4613-998a-ad43d62e6367\") " pod="ai-tenant-e2e-inf-b-5a005d/test-model-5a005d-kserve-844f5dc4c9-f72bf"
Sep 19 01:29:57.566962 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:57.566788    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f17cb1a-739d-4613-998a-ad43d62e6367-dshm\") pod \"test-model-5a005d-kserve-844f5dc4c9-f72bf\" (UID: \"6f17cb1a-739d-4613-998a-ad43d62e6367\") " pod="ai-tenant-e2e-inf-b-5a005d/test-model-5a005d-kserve-844f5dc4c9-f72bf"
Sep 19 01:29:57.566962 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:57.566811    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f17cb1a-739d-4613-998a-ad43d62e6367-model-cache\") pod \"test-model-5a005d-kserve-844f5dc4c9-f72bf\" (UID: \"6f17cb1a-739d-4613-998a-ad43d62e6367\") " pod="ai-tenant-e2e-inf-b-5a005d/test-model-5a005d-kserve-844f5dc4c9-f72bf"
Sep 19 01:29:57.566962 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:57.566920    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f17cb1a-739d-4613-998a-ad43d62e6367-tmp-dir\") pod \"test-model-5a005d-kserve-844f5dc4c9-f72bf\" (UID: \"6f17cb1a-739d-4613-998a-ad43d62e6367\") " pod="ai-tenant-e2e-inf-b-5a005d/test-model-5a005d-kserve-844f5dc4c9-f72bf"
Sep 19 01:29:57.566962 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:57.566953    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f17cb1a-739d-4613-998a-ad43d62e6367-tls-certs\") pod \"test-model-5a005d-kserve-844f5dc4c9-f72bf\" (UID: \"6f17cb1a-739d-4613-998a-ad43d62e6367\") " pod="ai-tenant-e2e-inf-b-5a005d/test-model-5a005d-kserve-844f5dc4c9-f72bf"
Sep 19 01:29:57.567180 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:57.566986    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f17cb1a-739d-4613-998a-ad43d62e6367-home\") pod \"test-model-5a005d-kserve-844f5dc4c9-f72bf\" (UID: \"6f17cb1a-739d-4613-998a-ad43d62e6367\") " pod="ai-tenant-e2e-inf-b-5a005d/test-model-5a005d-kserve-844f5dc4c9-f72bf"
Sep 19 01:29:57.667431 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:57.667398    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29rsb\" (UniqueName: \"kubernetes.io/projected/6f17cb1a-739d-4613-998a-ad43d62e6367-kube-api-access-29rsb\") pod \"test-model-5a005d-kserve-844f5dc4c9-f72bf\" (UID: \"6f17cb1a-739d-4613-998a-ad43d62e6367\") " pod="ai-tenant-e2e-inf-b-5a005d/test-model-5a005d-kserve-844f5dc4c9-f72bf"
Sep 19 01:29:57.667590 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:57.667465    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f17cb1a-739d-4613-998a-ad43d62e6367-dshm\") pod \"test-model-5a005d-kserve-844f5dc4c9-f72bf\" (UID: \"6f17cb1a-739d-4613-998a-ad43d62e6367\") " pod="ai-tenant-e2e-inf-b-5a005d/test-model-5a005d-kserve-844f5dc4c9-f72bf"
Sep 19 01:29:57.667590 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:57.667523    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f17cb1a-739d-4613-998a-ad43d62e6367-model-cache\") pod \"test-model-5a005d-kserve-844f5dc4c9-f72bf\" (UID: \"6f17cb1a-739d-4613-998a-ad43d62e6367\") " pod="ai-tenant-e2e-inf-b-5a005d/test-model-5a005d-kserve-844f5dc4c9-f72bf"
Sep 19 01:29:57.667590 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:57.667565    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f17cb1a-739d-4613-998a-ad43d62e6367-tmp-dir\") pod \"test-model-5a005d-kserve-844f5dc4c9-f72bf\" (UID: \"6f17cb1a-739d-4613-998a-ad43d62e6367\") " pod="ai-tenant-e2e-inf-b-5a005d/test-model-5a005d-kserve-844f5dc4c9-f72bf"
Sep 19 01:29:57.667751 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:57.667588    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f17cb1a-739d-4613-998a-ad43d62e6367-tls-certs\") pod \"test-model-5a005d-kserve-844f5dc4c9-f72bf\" (UID: \"6f17cb1a-739d-4613-998a-ad43d62e6367\") " pod="ai-tenant-e2e-inf-b-5a005d/test-model-5a005d-kserve-844f5dc4c9-f72bf"
Sep 19 01:29:57.667751 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:57.667620    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f17cb1a-739d-4613-998a-ad43d62e6367-home\") pod \"test-model-5a005d-kserve-844f5dc4c9-f72bf\" (UID: \"6f17cb1a-739d-4613-998a-ad43d62e6367\") " pod="ai-tenant-e2e-inf-b-5a005d/test-model-5a005d-kserve-844f5dc4c9-f72bf"
Sep 19 01:29:57.668001 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:57.667980    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f17cb1a-739d-4613-998a-ad43d62e6367-model-cache\") pod \"test-model-5a005d-kserve-844f5dc4c9-f72bf\" (UID: \"6f17cb1a-739d-4613-998a-ad43d62e6367\") " pod="ai-tenant-e2e-inf-b-5a005d/test-model-5a005d-kserve-844f5dc4c9-f72bf"
Sep 19 01:29:57.668083 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:57.668011    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f17cb1a-739d-4613-998a-ad43d62e6367-home\") pod \"test-model-5a005d-kserve-844f5dc4c9-f72bf\" (UID: \"6f17cb1a-739d-4613-998a-ad43d62e6367\") " pod="ai-tenant-e2e-inf-b-5a005d/test-model-5a005d-kserve-844f5dc4c9-f72bf"
Sep 19 01:29:57.668121 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:57.668073    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f17cb1a-739d-4613-998a-ad43d62e6367-tmp-dir\") pod \"test-model-5a005d-kserve-844f5dc4c9-f72bf\" (UID: \"6f17cb1a-739d-4613-998a-ad43d62e6367\") " pod="ai-tenant-e2e-inf-b-5a005d/test-model-5a005d-kserve-844f5dc4c9-f72bf"
Sep 19 01:29:57.669978 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:57.669961    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f17cb1a-739d-4613-998a-ad43d62e6367-dshm\") pod \"test-model-5a005d-kserve-844f5dc4c9-f72bf\" (UID: \"6f17cb1a-739d-4613-998a-ad43d62e6367\") " pod="ai-tenant-e2e-inf-b-5a005d/test-model-5a005d-kserve-844f5dc4c9-f72bf"
Sep 19 01:29:57.670330 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:57.670311    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f17cb1a-739d-4613-998a-ad43d62e6367-tls-certs\") pod \"test-model-5a005d-kserve-844f5dc4c9-f72bf\" (UID: \"6f17cb1a-739d-4613-998a-ad43d62e6367\") " pod="ai-tenant-e2e-inf-b-5a005d/test-model-5a005d-kserve-844f5dc4c9-f72bf"
Sep 19 01:29:57.675073 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:57.675053    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29rsb\" (UniqueName: \"kubernetes.io/projected/6f17cb1a-739d-4613-998a-ad43d62e6367-kube-api-access-29rsb\") pod \"test-model-5a005d-kserve-844f5dc4c9-f72bf\" (UID: \"6f17cb1a-739d-4613-998a-ad43d62e6367\") " pod="ai-tenant-e2e-inf-b-5a005d/test-model-5a005d-kserve-844f5dc4c9-f72bf"
Sep 19 01:29:57.800937 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:57.800845    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5a005d/test-model-5a005d-kserve-844f5dc4c9-f72bf"
Sep 19 01:29:58.074979 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:58.074951    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-5a005d/test-model-5a005d-kserve-844f5dc4c9-f72bf"]
Sep 19 01:29:58.077568 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:29:58.077539    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6f17cb1a_739d_4613_998a_ad43d62e6367.slice/crio-27a15624204f48897f3f519567e5d6a51c04742a2be0c6db48daea89eacea3f8 WatchSource:0}: Error finding container 27a15624204f48897f3f519567e5d6a51c04742a2be0c6db48daea89eacea3f8: Status 404 returned error can't find the container with id 27a15624204f48897f3f519567e5d6a51c04742a2be0c6db48daea89eacea3f8
Sep 19 01:29:58.624495 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:58.624457    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5a005d/test-model-5a005d-kserve-844f5dc4c9-f72bf" event={"ID":"6f17cb1a-739d-4613-998a-ad43d62e6367","Type":"ContainerStarted","Data":"ae580c46d316d56f7b9477bb41105f7e0efdf0b67e1ff9b463b177c03b49bfae"}
Sep 19 01:29:58.624916 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:58.624506    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5a005d/test-model-5a005d-kserve-844f5dc4c9-f72bf" event={"ID":"6f17cb1a-739d-4613-998a-ad43d62e6367","Type":"ContainerStarted","Data":"27a15624204f48897f3f519567e5d6a51c04742a2be0c6db48daea89eacea3f8"}
Sep 19 01:29:58.641815 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:29:58.641769    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-5a005d/test-model-5a005d-kserve-844f5dc4c9-f72bf" podStartSLOduration=1.641754554 podStartE2EDuration="1.641754554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 01:29:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 01:29:58.64039519 +0000 UTC m=+1049.528064287" watchObservedRunningTime="2026-09-19 01:29:58.641754554 +0000 UTC m=+1049.529423631"
Sep 19 01:30:00.140869 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:00.140838    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c4ce82-29829690-hcmkb"]
Sep 19 01:30:00.144057 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:00.144041    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c4ce82-29829690-hcmkb"
Sep 19 01:30:00.147840 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:00.147819    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s7pvt\""
Sep 19 01:30:00.151775 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:00.151757    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29829690-lw8pt"]
Sep 19 01:30:00.154229 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:00.154207    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c4ce82-29829690-hcmkb"
Sep 19 01:30:00.154756 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:00.154742    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8e708f-29829690-2sn9s"]
Sep 19 01:30:00.154876 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:00.154844    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829690-lw8pt"
Sep 19 01:30:00.157665 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:00.157651    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8e708f-29829690-2sn9s"
Sep 19 01:30:00.165480 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:00.165441    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29829690-lw8pt"]
Sep 19 01:30:00.167276 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:00.167258    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c4ce82-29829690-hcmkb"]
Sep 19 01:30:00.169426 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:00.169407    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8e708f-29829690-2sn9s"]
Sep 19 01:30:00.185387 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:00.185362    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829690-lw8pt"
Sep 19 01:30:00.192391 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:00.192372    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8e708f-29829690-2sn9s"
Sep 19 01:30:00.437496 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:00.437422    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c4ce82-29829690-hcmkb"]
Sep 19 01:30:00.440735 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:30:00.440709    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7d53214_d510_4026_ba65_37ea04bece7d.slice/crio-a959d249b235b94732f100fbc15066fb072e6fdaedb18bccfa44bea41fefa863 WatchSource:0}: Error finding container a959d249b235b94732f100fbc15066fb072e6fdaedb18bccfa44bea41fefa863: Status 404 returned error can't find the container with id a959d249b235b94732f100fbc15066fb072e6fdaedb18bccfa44bea41fefa863
Sep 19 01:30:00.478335 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:00.478289    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29829690-lw8pt"]
Sep 19 01:30:00.480240 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:30:00.480213    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode69b83de_6d45_4d82_8ba0_364cc2ca6bdd.slice/crio-8d5aefb08c942d1a489a3c507b2eacbbf5732be2ad3ee3b2c74dfb62ca505b76 WatchSource:0}: Error finding container 8d5aefb08c942d1a489a3c507b2eacbbf5732be2ad3ee3b2c74dfb62ca505b76: Status 404 returned error can't find the container with id 8d5aefb08c942d1a489a3c507b2eacbbf5732be2ad3ee3b2c74dfb62ca505b76
Sep 19 01:30:00.489811 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:00.489766    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8e708f-29829690-2sn9s"]
Sep 19 01:30:00.492295 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:30:00.492272    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25527fb0_589a_44d6_9445_013a4a27f92b.slice/crio-1573e1f48616b6a2500ebe8dcda37e67548280cb322652b2d1867c478e7a0d9b WatchSource:0}: Error finding container 1573e1f48616b6a2500ebe8dcda37e67548280cb322652b2d1867c478e7a0d9b: Status 404 returned error can't find the container with id 1573e1f48616b6a2500ebe8dcda37e67548280cb322652b2d1867c478e7a0d9b
Sep 19 01:30:00.635139 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:00.635106    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8e708f-29829690-2sn9s" event={"ID":"25527fb0-589a-44d6-9445-013a4a27f92b","Type":"ContainerStarted","Data":"1573e1f48616b6a2500ebe8dcda37e67548280cb322652b2d1867c478e7a0d9b"}
Sep 19 01:30:00.636404 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:00.636381    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c4ce82-29829690-hcmkb" event={"ID":"a7d53214-d510-4026-ba65-37ea04bece7d","Type":"ContainerStarted","Data":"a959d249b235b94732f100fbc15066fb072e6fdaedb18bccfa44bea41fefa863"}
Sep 19 01:30:00.637445 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:00.637421    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829690-lw8pt" event={"ID":"e69b83de-6d45-4d82-8ba0-364cc2ca6bdd","Type":"ContainerStarted","Data":"8d5aefb08c942d1a489a3c507b2eacbbf5732be2ad3ee3b2c74dfb62ca505b76"}
Sep 19 01:30:01.643872 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:01.643836    2581 generic.go:356] "Generic (PLEG): container finished" podID="25527fb0-589a-44d6-9445-013a4a27f92b" containerID="20fe871e3a06b4387d5d3fc354165cd0b8ddc62277c26b1ccc5b2d7235d3dbcb" exitCode=0
Sep 19 01:30:01.644310 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:01.643907    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8e708f-29829690-2sn9s" event={"ID":"25527fb0-589a-44d6-9445-013a4a27f92b","Type":"ContainerDied","Data":"20fe871e3a06b4387d5d3fc354165cd0b8ddc62277c26b1ccc5b2d7235d3dbcb"}
Sep 19 01:30:01.645575 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:01.645551    2581 generic.go:356] "Generic (PLEG): container finished" podID="a7d53214-d510-4026-ba65-37ea04bece7d" containerID="e949de4ab1cf21962484bf39ea72b18866a747ff9e680971b8e898795af9301b" exitCode=0
Sep 19 01:30:01.645723 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:01.645622    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c4ce82-29829690-hcmkb" event={"ID":"a7d53214-d510-4026-ba65-37ea04bece7d","Type":"ContainerDied","Data":"e949de4ab1cf21962484bf39ea72b18866a747ff9e680971b8e898795af9301b"}
Sep 19 01:30:01.647056 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:01.647033    2581 generic.go:356] "Generic (PLEG): container finished" podID="e69b83de-6d45-4d82-8ba0-364cc2ca6bdd" containerID="b275c2bc70ed9cbb602a19720783eb75daa3329aee7229adeff3219a67902ca1" exitCode=0
Sep 19 01:30:01.647164 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:01.647080    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829690-lw8pt" event={"ID":"e69b83de-6d45-4d82-8ba0-364cc2ca6bdd","Type":"ContainerDied","Data":"b275c2bc70ed9cbb602a19720783eb75daa3329aee7229adeff3219a67902ca1"}
Sep 19 01:30:02.797303 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:02.797277    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8e708f-29829690-2sn9s"
Sep 19 01:30:02.836459 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:02.836436    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c4ce82-29829690-hcmkb"
Sep 19 01:30:02.839350 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:02.839332    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829690-lw8pt"
Sep 19 01:30:03.657556 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:03.657525    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c4ce82-29829690-hcmkb"
Sep 19 01:30:03.657735 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:03.657531    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c4ce82-29829690-hcmkb" event={"ID":"a7d53214-d510-4026-ba65-37ea04bece7d","Type":"ContainerDied","Data":"a959d249b235b94732f100fbc15066fb072e6fdaedb18bccfa44bea41fefa863"}
Sep 19 01:30:03.657735 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:03.657644    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a959d249b235b94732f100fbc15066fb072e6fdaedb18bccfa44bea41fefa863"
Sep 19 01:30:03.658970 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:03.658929    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829690-lw8pt"
Sep 19 01:30:03.658970 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:03.658952    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829690-lw8pt" event={"ID":"e69b83de-6d45-4d82-8ba0-364cc2ca6bdd","Type":"ContainerDied","Data":"8d5aefb08c942d1a489a3c507b2eacbbf5732be2ad3ee3b2c74dfb62ca505b76"}
Sep 19 01:30:03.659148 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:03.658994    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d5aefb08c942d1a489a3c507b2eacbbf5732be2ad3ee3b2c74dfb62ca505b76"
Sep 19 01:30:03.660611 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:03.660591    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8e708f-29829690-2sn9s"
Sep 19 01:30:03.660706 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:03.660611    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8e708f-29829690-2sn9s" event={"ID":"25527fb0-589a-44d6-9445-013a4a27f92b","Type":"ContainerDied","Data":"1573e1f48616b6a2500ebe8dcda37e67548280cb322652b2d1867c478e7a0d9b"}
Sep 19 01:30:03.660706 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:03.660646    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1573e1f48616b6a2500ebe8dcda37e67548280cb322652b2d1867c478e7a0d9b"
Sep 19 01:30:07.089984 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:07.089951    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c9nsz"]
Sep 19 01:30:07.801125 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:07.801090    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-5a005d/test-model-5a005d-kserve-844f5dc4c9-f72bf"
Sep 19 01:30:07.801328 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:07.801139    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-5a005d/test-model-5a005d-kserve-844f5dc4c9-f72bf"
Sep 19 01:30:07.813445 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:07.813411    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-5a005d/test-model-5a005d-kserve-844f5dc4c9-f72bf"
Sep 19 01:30:08.695481 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:08.695444    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-5a005d/test-model-5a005d-kserve-844f5dc4c9-f72bf"
Sep 19 01:30:11.473066 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:11.472989    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-728f63f6-77ddb8bf4-f47df"]
Sep 19 01:30:11.473401 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:11.473292    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25527fb0-589a-44d6-9445-013a4a27f92b" containerName="cleanup"
Sep 19 01:30:11.473401 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:11.473304    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="25527fb0-589a-44d6-9445-013a4a27f92b" containerName="cleanup"
Sep 19 01:30:11.473401 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:11.473324    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7d53214-d510-4026-ba65-37ea04bece7d" containerName="cleanup"
Sep 19 01:30:11.473401 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:11.473329    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7d53214-d510-4026-ba65-37ea04bece7d" containerName="cleanup"
Sep 19 01:30:11.473401 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:11.473338    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e69b83de-6d45-4d82-8ba0-364cc2ca6bdd" containerName="cleanup"
Sep 19 01:30:11.473401 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:11.473343    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e69b83de-6d45-4d82-8ba0-364cc2ca6bdd" containerName="cleanup"
Sep 19 01:30:11.473401 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:11.473391    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="25527fb0-589a-44d6-9445-013a4a27f92b" containerName="cleanup"
Sep 19 01:30:11.473401 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:11.473399    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e69b83de-6d45-4d82-8ba0-364cc2ca6bdd" containerName="cleanup"
Sep 19 01:30:11.473401 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:11.473404    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7d53214-d510-4026-ba65-37ea04bece7d" containerName="cleanup"
Sep 19 01:30:11.476487 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:11.476468    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-728f63f6-77ddb8bf4-f47df"
Sep 19 01:30:11.478602 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:11.478583    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-728f63f6\""
Sep 19 01:30:11.485514 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:11.485493    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-728f63f6-77ddb8bf4-f47df"]
Sep 19 01:30:11.588138 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:11.588112    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n677\" (UniqueName: \"kubernetes.io/projected/bad73288-a138-4b61-9a2e-316af32f0ca0-kube-api-access-2n677\") pod \"payload-pre-processing-e2e-mt-728f63f6-77ddb8bf4-f47df\" (UID: \"bad73288-a138-4b61-9a2e-316af32f0ca0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-728f63f6-77ddb8bf4-f47df"
Sep 19 01:30:11.588241 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:11.588158    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bad73288-a138-4b61-9a2e-316af32f0ca0-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-728f63f6-77ddb8bf4-f47df\" (UID: \"bad73288-a138-4b61-9a2e-316af32f0ca0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-728f63f6-77ddb8bf4-f47df"
Sep 19 01:30:11.689242 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:11.689209    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n677\" (UniqueName: \"kubernetes.io/projected/bad73288-a138-4b61-9a2e-316af32f0ca0-kube-api-access-2n677\") pod \"payload-pre-processing-e2e-mt-728f63f6-77ddb8bf4-f47df\" (UID: \"bad73288-a138-4b61-9a2e-316af32f0ca0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-728f63f6-77ddb8bf4-f47df"
Sep 19 01:30:11.689422 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:11.689258    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bad73288-a138-4b61-9a2e-316af32f0ca0-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-728f63f6-77ddb8bf4-f47df\" (UID: \"bad73288-a138-4b61-9a2e-316af32f0ca0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-728f63f6-77ddb8bf4-f47df"
Sep 19 01:30:11.689966 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:11.689949    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bad73288-a138-4b61-9a2e-316af32f0ca0-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-728f63f6-77ddb8bf4-f47df\" (UID: \"bad73288-a138-4b61-9a2e-316af32f0ca0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-728f63f6-77ddb8bf4-f47df"
Sep 19 01:30:11.697331 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:11.697306    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n677\" (UniqueName: \"kubernetes.io/projected/bad73288-a138-4b61-9a2e-316af32f0ca0-kube-api-access-2n677\") pod \"payload-pre-processing-e2e-mt-728f63f6-77ddb8bf4-f47df\" (UID: \"bad73288-a138-4b61-9a2e-316af32f0ca0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-728f63f6-77ddb8bf4-f47df"
Sep 19 01:30:11.787438 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:11.787373    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-728f63f6-77ddb8bf4-f47df"
Sep 19 01:30:12.062001 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:12.061972    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-728f63f6-77ddb8bf4-f47df"]
Sep 19 01:30:12.063697 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:30:12.063671    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbad73288_a138_4b61_9a2e_316af32f0ca0.slice/crio-85c3e09a30837c04d44f079ddf847f7bc4df1c60fd3a39f94ea3dbe2dc64dd48 WatchSource:0}: Error finding container 85c3e09a30837c04d44f079ddf847f7bc4df1c60fd3a39f94ea3dbe2dc64dd48: Status 404 returned error can't find the container with id 85c3e09a30837c04d44f079ddf847f7bc4df1c60fd3a39f94ea3dbe2dc64dd48
Sep 19 01:30:12.708126 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:12.708089    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-728f63f6-77ddb8bf4-f47df" event={"ID":"bad73288-a138-4b61-9a2e-316af32f0ca0","Type":"ContainerStarted","Data":"7da10cb6e1f43fbf027752f086e38cecd562cfd6a76acaf17beebb83c8df16dc"}
Sep 19 01:30:12.708509 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:12.708147    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-728f63f6-77ddb8bf4-f47df" event={"ID":"bad73288-a138-4b61-9a2e-316af32f0ca0","Type":"ContainerStarted","Data":"85c3e09a30837c04d44f079ddf847f7bc4df1c60fd3a39f94ea3dbe2dc64dd48"}
Sep 19 01:30:12.708509 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:12.708174    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-728f63f6-77ddb8bf4-f47df"
Sep 19 01:30:12.726846 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:12.726799    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-728f63f6-77ddb8bf4-f47df" podStartSLOduration=1.7267852700000001 podStartE2EDuration="1.72678527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 01:30:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 01:30:12.725668602 +0000 UTC m=+1063.613337704" watchObservedRunningTime="2026-09-19 01:30:12.72678527 +0000 UTC m=+1063.614454347"
Sep 19 01:30:23.714580 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:23.714550    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-728f63f6-77ddb8bf4-f47df"
Sep 19 01:30:26.988142 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:26.988103    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c9nsz"]
Sep 19 01:30:29.743422 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:29.743379    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-cf0ace/explicit-model-cf0ace-kserve-7bf66584b5-x55tm"]
Sep 19 01:30:29.746748 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:29.746731    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-cf0ace/explicit-model-cf0ace-kserve-7bf66584b5-x55tm"
Sep 19 01:30:29.749936 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:29.749913    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-cf0ace\"/\"default-dockercfg-5h44k\""
Sep 19 01:30:29.750126 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:29.749976    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-cf0ace\"/\"openshift-service-ca.crt\""
Sep 19 01:30:29.750549 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:29.750525    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-cf0ace\"/\"kube-root-ca.crt\""
Sep 19 01:30:29.750683 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:29.750663    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-cf0ace\"/\"explicit-model-cf0ace-kserve-self-signed-certs\""
Sep 19 01:30:29.770130 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:29.770110    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-cf0ace/explicit-model-cf0ace-kserve-7bf66584b5-x55tm"]
Sep 19 01:30:29.830818 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:29.830791    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57d37a6f-1f68-404c-9574-9339bd408486-tls-certs\") pod \"explicit-model-cf0ace-kserve-7bf66584b5-x55tm\" (UID: \"57d37a6f-1f68-404c-9574-9339bd408486\") " pod="ai-tenant-e2e-explicit-cf0ace/explicit-model-cf0ace-kserve-7bf66584b5-x55tm"
Sep 19 01:30:29.830948 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:29.830827    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57d37a6f-1f68-404c-9574-9339bd408486-home\") pod \"explicit-model-cf0ace-kserve-7bf66584b5-x55tm\" (UID: \"57d37a6f-1f68-404c-9574-9339bd408486\") " pod="ai-tenant-e2e-explicit-cf0ace/explicit-model-cf0ace-kserve-7bf66584b5-x55tm"
Sep 19 01:30:29.830948 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:29.830846    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57d37a6f-1f68-404c-9574-9339bd408486-dshm\") pod \"explicit-model-cf0ace-kserve-7bf66584b5-x55tm\" (UID: \"57d37a6f-1f68-404c-9574-9339bd408486\") " pod="ai-tenant-e2e-explicit-cf0ace/explicit-model-cf0ace-kserve-7bf66584b5-x55tm"
Sep 19 01:30:29.830948 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:29.830895    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57d37a6f-1f68-404c-9574-9339bd408486-tmp-dir\") pod \"explicit-model-cf0ace-kserve-7bf66584b5-x55tm\" (UID: \"57d37a6f-1f68-404c-9574-9339bd408486\") " pod="ai-tenant-e2e-explicit-cf0ace/explicit-model-cf0ace-kserve-7bf66584b5-x55tm"
Sep 19 01:30:29.831056 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:29.830961    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57d37a6f-1f68-404c-9574-9339bd408486-model-cache\") pod \"explicit-model-cf0ace-kserve-7bf66584b5-x55tm\" (UID: \"57d37a6f-1f68-404c-9574-9339bd408486\") " pod="ai-tenant-e2e-explicit-cf0ace/explicit-model-cf0ace-kserve-7bf66584b5-x55tm"
Sep 19 01:30:29.831056 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:29.830985    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwtzg\" (UniqueName: \"kubernetes.io/projected/57d37a6f-1f68-404c-9574-9339bd408486-kube-api-access-pwtzg\") pod \"explicit-model-cf0ace-kserve-7bf66584b5-x55tm\" (UID: \"57d37a6f-1f68-404c-9574-9339bd408486\") " pod="ai-tenant-e2e-explicit-cf0ace/explicit-model-cf0ace-kserve-7bf66584b5-x55tm"
Sep 19 01:30:29.932074 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:29.932043    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57d37a6f-1f68-404c-9574-9339bd408486-dshm\") pod \"explicit-model-cf0ace-kserve-7bf66584b5-x55tm\" (UID: \"57d37a6f-1f68-404c-9574-9339bd408486\") " pod="ai-tenant-e2e-explicit-cf0ace/explicit-model-cf0ace-kserve-7bf66584b5-x55tm"
Sep 19 01:30:29.932180 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:29.932077    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57d37a6f-1f68-404c-9574-9339bd408486-tmp-dir\") pod \"explicit-model-cf0ace-kserve-7bf66584b5-x55tm\" (UID: \"57d37a6f-1f68-404c-9574-9339bd408486\") " pod="ai-tenant-e2e-explicit-cf0ace/explicit-model-cf0ace-kserve-7bf66584b5-x55tm"
Sep 19 01:30:29.932180 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:29.932111    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57d37a6f-1f68-404c-9574-9339bd408486-model-cache\") pod \"explicit-model-cf0ace-kserve-7bf66584b5-x55tm\" (UID: \"57d37a6f-1f68-404c-9574-9339bd408486\") " pod="ai-tenant-e2e-explicit-cf0ace/explicit-model-cf0ace-kserve-7bf66584b5-x55tm"
Sep 19 01:30:29.932180 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:29.932128    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwtzg\" (UniqueName: \"kubernetes.io/projected/57d37a6f-1f68-404c-9574-9339bd408486-kube-api-access-pwtzg\") pod \"explicit-model-cf0ace-kserve-7bf66584b5-x55tm\" (UID: \"57d37a6f-1f68-404c-9574-9339bd408486\") " pod="ai-tenant-e2e-explicit-cf0ace/explicit-model-cf0ace-kserve-7bf66584b5-x55tm"
Sep 19 01:30:29.932298 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:29.932186    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57d37a6f-1f68-404c-9574-9339bd408486-tls-certs\") pod \"explicit-model-cf0ace-kserve-7bf66584b5-x55tm\" (UID: \"57d37a6f-1f68-404c-9574-9339bd408486\") " pod="ai-tenant-e2e-explicit-cf0ace/explicit-model-cf0ace-kserve-7bf66584b5-x55tm"
Sep 19 01:30:29.932298 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:29.932202    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57d37a6f-1f68-404c-9574-9339bd408486-home\") pod \"explicit-model-cf0ace-kserve-7bf66584b5-x55tm\" (UID: \"57d37a6f-1f68-404c-9574-9339bd408486\") " pod="ai-tenant-e2e-explicit-cf0ace/explicit-model-cf0ace-kserve-7bf66584b5-x55tm"
Sep 19 01:30:29.932587 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:29.932564    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57d37a6f-1f68-404c-9574-9339bd408486-tmp-dir\") pod \"explicit-model-cf0ace-kserve-7bf66584b5-x55tm\" (UID: \"57d37a6f-1f68-404c-9574-9339bd408486\") " pod="ai-tenant-e2e-explicit-cf0ace/explicit-model-cf0ace-kserve-7bf66584b5-x55tm"
Sep 19 01:30:29.932654 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:29.932588    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57d37a6f-1f68-404c-9574-9339bd408486-model-cache\") pod \"explicit-model-cf0ace-kserve-7bf66584b5-x55tm\" (UID: \"57d37a6f-1f68-404c-9574-9339bd408486\") " pod="ai-tenant-e2e-explicit-cf0ace/explicit-model-cf0ace-kserve-7bf66584b5-x55tm"
Sep 19 01:30:29.932654 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:29.932624    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57d37a6f-1f68-404c-9574-9339bd408486-home\") pod \"explicit-model-cf0ace-kserve-7bf66584b5-x55tm\" (UID: \"57d37a6f-1f68-404c-9574-9339bd408486\") " pod="ai-tenant-e2e-explicit-cf0ace/explicit-model-cf0ace-kserve-7bf66584b5-x55tm"
Sep 19 01:30:29.934636 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:29.934615    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57d37a6f-1f68-404c-9574-9339bd408486-dshm\") pod \"explicit-model-cf0ace-kserve-7bf66584b5-x55tm\" (UID: \"57d37a6f-1f68-404c-9574-9339bd408486\") " pod="ai-tenant-e2e-explicit-cf0ace/explicit-model-cf0ace-kserve-7bf66584b5-x55tm"
Sep 19 01:30:29.934719 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:29.934694    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57d37a6f-1f68-404c-9574-9339bd408486-tls-certs\") pod \"explicit-model-cf0ace-kserve-7bf66584b5-x55tm\" (UID: \"57d37a6f-1f68-404c-9574-9339bd408486\") " pod="ai-tenant-e2e-explicit-cf0ace/explicit-model-cf0ace-kserve-7bf66584b5-x55tm"
Sep 19 01:30:29.942720 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:29.942699    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwtzg\" (UniqueName: \"kubernetes.io/projected/57d37a6f-1f68-404c-9574-9339bd408486-kube-api-access-pwtzg\") pod \"explicit-model-cf0ace-kserve-7bf66584b5-x55tm\" (UID: \"57d37a6f-1f68-404c-9574-9339bd408486\") " pod="ai-tenant-e2e-explicit-cf0ace/explicit-model-cf0ace-kserve-7bf66584b5-x55tm"
Sep 19 01:30:30.056299 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:30.056250    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-cf0ace/explicit-model-cf0ace-kserve-7bf66584b5-x55tm"
Sep 19 01:30:30.329791 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:30.329762    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-cf0ace/explicit-model-cf0ace-kserve-7bf66584b5-x55tm"]
Sep 19 01:30:30.331499 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:30:30.331475    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod57d37a6f_1f68_404c_9574_9339bd408486.slice/crio-d7815199c111de19f30c12b2573b56547b5ca77264ad0ebba99294b204a63337 WatchSource:0}: Error finding container d7815199c111de19f30c12b2573b56547b5ca77264ad0ebba99294b204a63337: Status 404 returned error can't find the container with id d7815199c111de19f30c12b2573b56547b5ca77264ad0ebba99294b204a63337
Sep 19 01:30:30.798470 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:30.798429    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-cf0ace/explicit-model-cf0ace-kserve-7bf66584b5-x55tm" event={"ID":"57d37a6f-1f68-404c-9574-9339bd408486","Type":"ContainerStarted","Data":"3b86f161bd776aa6bea96200aff4e32461ce2d06ed7c9627ea446ee94ecdaf99"}
Sep 19 01:30:30.798470 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:30.798472    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-cf0ace/explicit-model-cf0ace-kserve-7bf66584b5-x55tm" event={"ID":"57d37a6f-1f68-404c-9574-9339bd408486","Type":"ContainerStarted","Data":"d7815199c111de19f30c12b2573b56547b5ca77264ad0ebba99294b204a63337"}
Sep 19 01:30:30.815897 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:30.815829    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-cf0ace/explicit-model-cf0ace-kserve-7bf66584b5-x55tm" podStartSLOduration=1.815811304 podStartE2EDuration="1.815811304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 01:30:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 01:30:30.815340422 +0000 UTC m=+1081.703009517" watchObservedRunningTime="2026-09-19 01:30:30.815811304 +0000 UTC m=+1081.703480382"
Sep 19 01:30:37.509392 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:37.509362    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-cf0ace/explicit-model-cf0ace-kserve-7bf66584b5-x55tm"]
Sep 19 01:30:37.636350 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:37.636313    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cf0ace-7g7lb"]
Sep 19 01:30:37.640581 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:37.640559    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cf0ace-7g7lb"
Sep 19 01:30:37.643952 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:37.643924    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 19 01:30:37.644148 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:37.644134    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s7pvt\""
Sep 19 01:30:37.660672 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:37.660649    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cf0ace-7g7lb"]
Sep 19 01:30:37.798259 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:37.798185    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16b0d14e-300c-4572-ba96-07f080de00da-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-cf0ace-7g7lb\" (UID: \"16b0d14e-300c-4572-ba96-07f080de00da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cf0ace-7g7lb"
Sep 19 01:30:37.899390 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:37.899362    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16b0d14e-300c-4572-ba96-07f080de00da-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-cf0ace-7g7lb\" (UID: \"16b0d14e-300c-4572-ba96-07f080de00da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cf0ace-7g7lb"
Sep 19 01:30:37.899986 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:37.899968    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16b0d14e-300c-4572-ba96-07f080de00da-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-cf0ace-7g7lb\" (UID: \"16b0d14e-300c-4572-ba96-07f080de00da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cf0ace-7g7lb"
Sep 19 01:30:37.950364 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:37.950327    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cf0ace-7g7lb"
Sep 19 01:30:38.218221 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:38.218047    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cf0ace-7g7lb"]
Sep 19 01:30:38.220947 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:30:38.220921    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16b0d14e_300c_4572_ba96_07f080de00da.slice/crio-d8f8dac6c24d9c75c9f3f8b99a015329d0d51b1a48e069db7cf64f7ed2e72e7b WatchSource:0}: Error finding container d8f8dac6c24d9c75c9f3f8b99a015329d0d51b1a48e069db7cf64f7ed2e72e7b: Status 404 returned error can't find the container with id d8f8dac6c24d9c75c9f3f8b99a015329d0d51b1a48e069db7cf64f7ed2e72e7b
Sep 19 01:30:38.851509 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:38.851477    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cf0ace-7g7lb" event={"ID":"16b0d14e-300c-4572-ba96-07f080de00da","Type":"ContainerStarted","Data":"6db9a10f219875cb499e9e790bf18bf0bd9cce4b5db16899647bc75a3ca24b49"}
Sep 19 01:30:38.851897 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:38.851531    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cf0ace-7g7lb" event={"ID":"16b0d14e-300c-4572-ba96-07f080de00da","Type":"ContainerStarted","Data":"d8f8dac6c24d9c75c9f3f8b99a015329d0d51b1a48e069db7cf64f7ed2e72e7b"}
Sep 19 01:30:38.901144 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:38.901106    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cf0ace-7g7lb" podStartSLOduration=1.901090892 podStartE2EDuration="1.901090892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 01:30:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 01:30:38.898181804 +0000 UTC m=+1089.785850882" watchObservedRunningTime="2026-09-19 01:30:38.901090892 +0000 UTC m=+1089.788759969"
Sep 19 01:30:39.858111 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:39.858074    2581 generic.go:356] "Generic (PLEG): container finished" podID="16b0d14e-300c-4572-ba96-07f080de00da" containerID="6db9a10f219875cb499e9e790bf18bf0bd9cce4b5db16899647bc75a3ca24b49" exitCode=0
Sep 19 01:30:39.858522 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:39.858158    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cf0ace-7g7lb" event={"ID":"16b0d14e-300c-4572-ba96-07f080de00da","Type":"ContainerDied","Data":"6db9a10f219875cb499e9e790bf18bf0bd9cce4b5db16899647bc75a3ca24b49"}
Sep 19 01:30:40.057220 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:40.057175    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-cf0ace/explicit-model-cf0ace-kserve-7bf66584b5-x55tm"
Sep 19 01:30:40.998794 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:40.998772    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cf0ace-7g7lb"
Sep 19 01:30:41.128004 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:41.127945    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16b0d14e-300c-4572-ba96-07f080de00da-maas-api-service-ca\") pod \"16b0d14e-300c-4572-ba96-07f080de00da\" (UID: \"16b0d14e-300c-4572-ba96-07f080de00da\") "
Sep 19 01:30:41.128270 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:41.128249    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16b0d14e-300c-4572-ba96-07f080de00da-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "16b0d14e-300c-4572-ba96-07f080de00da" (UID: "16b0d14e-300c-4572-ba96-07f080de00da"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 01:30:41.228934 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:41.228909    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16b0d14e-300c-4572-ba96-07f080de00da-maas-api-service-ca\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:30:41.868922 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:41.868887    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cf0ace-7g7lb" event={"ID":"16b0d14e-300c-4572-ba96-07f080de00da","Type":"ContainerDied","Data":"d8f8dac6c24d9c75c9f3f8b99a015329d0d51b1a48e069db7cf64f7ed2e72e7b"}
Sep 19 01:30:41.868922 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:41.868921    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cf0ace-7g7lb"
Sep 19 01:30:41.869139 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:41.868936    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d8f8dac6c24d9c75c9f3f8b99a015329d0d51b1a48e069db7cf64f7ed2e72e7b"
Sep 19 01:30:46.102043 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:46.102007    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-728f63f6-8hvxv"]
Sep 19 01:30:46.102412 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:46.102346    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16b0d14e-300c-4572-ba96-07f080de00da" containerName="revoke-keys"
Sep 19 01:30:46.102412 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:46.102357    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="16b0d14e-300c-4572-ba96-07f080de00da" containerName="revoke-keys"
Sep 19 01:30:46.102487 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:46.102429    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="16b0d14e-300c-4572-ba96-07f080de00da" containerName="revoke-keys"
Sep 19 01:30:46.105312 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:46.105297    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-728f63f6-8hvxv"
Sep 19 01:30:46.110880 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:46.110836    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 19 01:30:46.111246 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:46.111227    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s7pvt\""
Sep 19 01:30:46.130021 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:46.129999    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-728f63f6-8hvxv"]
Sep 19 01:30:46.267547 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:46.267515    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bcfffb5-087f-41a3-930a-9ca9c28d3bdc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-728f63f6-8hvxv\" (UID: \"3bcfffb5-087f-41a3-930a-9ca9c28d3bdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-728f63f6-8hvxv"
Sep 19 01:30:46.368468 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:46.368391    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bcfffb5-087f-41a3-930a-9ca9c28d3bdc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-728f63f6-8hvxv\" (UID: \"3bcfffb5-087f-41a3-930a-9ca9c28d3bdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-728f63f6-8hvxv"
Sep 19 01:30:46.368998 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:46.368979    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bcfffb5-087f-41a3-930a-9ca9c28d3bdc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-728f63f6-8hvxv\" (UID: \"3bcfffb5-087f-41a3-930a-9ca9c28d3bdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-728f63f6-8hvxv"
Sep 19 01:30:46.415434 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:46.415406    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-728f63f6-8hvxv"
Sep 19 01:30:46.687057 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:46.687034    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-728f63f6-8hvxv"]
Sep 19 01:30:46.689153 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:30:46.689124    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bcfffb5_087f_41a3_930a_9ca9c28d3bdc.slice/crio-0ad864a2fe87346642feea6b5478fe593b7760e3c742dd2087ea5e7a7627eb0d WatchSource:0}: Error finding container 0ad864a2fe87346642feea6b5478fe593b7760e3c742dd2087ea5e7a7627eb0d: Status 404 returned error can't find the container with id 0ad864a2fe87346642feea6b5478fe593b7760e3c742dd2087ea5e7a7627eb0d
Sep 19 01:30:46.895077 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:46.895047    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-728f63f6-8hvxv" event={"ID":"3bcfffb5-087f-41a3-930a-9ca9c28d3bdc","Type":"ContainerStarted","Data":"0ad864a2fe87346642feea6b5478fe593b7760e3c742dd2087ea5e7a7627eb0d"}
Sep 19 01:30:47.901565 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:47.901528    2581 generic.go:356] "Generic (PLEG): container finished" podID="3bcfffb5-087f-41a3-930a-9ca9c28d3bdc" containerID="b7b726aafd2f4427c203c2b4a3a2ff54062601b74d31874d0227576fae2f715c" exitCode=0
Sep 19 01:30:47.901957 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:47.901575    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-728f63f6-8hvxv" event={"ID":"3bcfffb5-087f-41a3-930a-9ca9c28d3bdc","Type":"ContainerDied","Data":"b7b726aafd2f4427c203c2b4a3a2ff54062601b74d31874d0227576fae2f715c"}
Sep 19 01:30:47.970519 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:47.970488    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-cf0ace-69dc9d55d6-p6wfl"]
Sep 19 01:30:47.970740 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:47.970716    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-cf0ace-69dc9d55d6-p6wfl" podUID="318478b1-28c2-4c07-a3bf-83b875f6e1e1" containerName="payload-pre-processing" containerID="cri-o://24ebeee39e2abdcf5d91a0924e5bcd3931b969716f8a777851da316113d35bb0" gracePeriod=30
Sep 19 01:30:48.199120 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:48.199049    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cf0ace-7g7lb"]
Sep 19 01:30:48.207366 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:48.207343    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cf0ace-7g7lb"]
Sep 19 01:30:48.222284 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:48.222266    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-cf0ace-69dc9d55d6-p6wfl"
Sep 19 01:30:48.387269 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:48.387235    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/318478b1-28c2-4c07-a3bf-83b875f6e1e1-plugins-config-volume\") pod \"318478b1-28c2-4c07-a3bf-83b875f6e1e1\" (UID: \"318478b1-28c2-4c07-a3bf-83b875f6e1e1\") "
Sep 19 01:30:48.387428 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:48.387297    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hv7dc\" (UniqueName: \"kubernetes.io/projected/318478b1-28c2-4c07-a3bf-83b875f6e1e1-kube-api-access-hv7dc\") pod \"318478b1-28c2-4c07-a3bf-83b875f6e1e1\" (UID: \"318478b1-28c2-4c07-a3bf-83b875f6e1e1\") "
Sep 19 01:30:48.387668 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:48.387639    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/318478b1-28c2-4c07-a3bf-83b875f6e1e1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "318478b1-28c2-4c07-a3bf-83b875f6e1e1" (UID: "318478b1-28c2-4c07-a3bf-83b875f6e1e1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 01:30:48.389558 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:48.389535    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/318478b1-28c2-4c07-a3bf-83b875f6e1e1-kube-api-access-hv7dc" (OuterVolumeSpecName: "kube-api-access-hv7dc") pod "318478b1-28c2-4c07-a3bf-83b875f6e1e1" (UID: "318478b1-28c2-4c07-a3bf-83b875f6e1e1"). InnerVolumeSpecName "kube-api-access-hv7dc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 01:30:48.489023 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:48.488964    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hv7dc\" (UniqueName: \"kubernetes.io/projected/318478b1-28c2-4c07-a3bf-83b875f6e1e1-kube-api-access-hv7dc\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:30:48.489023 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:48.488986    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/318478b1-28c2-4c07-a3bf-83b875f6e1e1-plugins-config-volume\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:30:48.908899 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:48.908846    2581 generic.go:356] "Generic (PLEG): container finished" podID="318478b1-28c2-4c07-a3bf-83b875f6e1e1" containerID="24ebeee39e2abdcf5d91a0924e5bcd3931b969716f8a777851da316113d35bb0" exitCode=0
Sep 19 01:30:48.909335 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:48.908907    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-cf0ace-69dc9d55d6-p6wfl"
Sep 19 01:30:48.909335 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:48.908934    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-cf0ace-69dc9d55d6-p6wfl" event={"ID":"318478b1-28c2-4c07-a3bf-83b875f6e1e1","Type":"ContainerDied","Data":"24ebeee39e2abdcf5d91a0924e5bcd3931b969716f8a777851da316113d35bb0"}
Sep 19 01:30:48.909335 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:48.908997    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-cf0ace-69dc9d55d6-p6wfl" event={"ID":"318478b1-28c2-4c07-a3bf-83b875f6e1e1","Type":"ContainerDied","Data":"c6473ab26a5ab8e5c39787ac2d56e6bcb69e9f424977415b181a9f94240fd959"}
Sep 19 01:30:48.909335 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:48.909029    2581 scope.go:117] "RemoveContainer" containerID="24ebeee39e2abdcf5d91a0924e5bcd3931b969716f8a777851da316113d35bb0"
Sep 19 01:30:48.918269 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:48.918251    2581 scope.go:117] "RemoveContainer" containerID="24ebeee39e2abdcf5d91a0924e5bcd3931b969716f8a777851da316113d35bb0"
Sep 19 01:30:48.918531 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:30:48.918514    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24ebeee39e2abdcf5d91a0924e5bcd3931b969716f8a777851da316113d35bb0\": container with ID starting with 24ebeee39e2abdcf5d91a0924e5bcd3931b969716f8a777851da316113d35bb0 not found: ID does not exist" containerID="24ebeee39e2abdcf5d91a0924e5bcd3931b969716f8a777851da316113d35bb0"
Sep 19 01:30:48.918601 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:48.918538    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24ebeee39e2abdcf5d91a0924e5bcd3931b969716f8a777851da316113d35bb0"} err="failed to get container status \"24ebeee39e2abdcf5d91a0924e5bcd3931b969716f8a777851da316113d35bb0\": rpc error: code = NotFound desc = could not find container \"24ebeee39e2abdcf5d91a0924e5bcd3931b969716f8a777851da316113d35bb0\": container with ID starting with 24ebeee39e2abdcf5d91a0924e5bcd3931b969716f8a777851da316113d35bb0 not found: ID does not exist"
Sep 19 01:30:48.932512 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:48.932479    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-cf0ace-69dc9d55d6-p6wfl"]
Sep 19 01:30:48.946581 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:48.943631    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-cf0ace-69dc9d55d6-p6wfl"]
Sep 19 01:30:49.041008 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:49.040981    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-728f63f6-8hvxv"
Sep 19 01:30:49.194374 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:49.194307    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bcfffb5-087f-41a3-930a-9ca9c28d3bdc-maas-api-service-ca\") pod \"3bcfffb5-087f-41a3-930a-9ca9c28d3bdc\" (UID: \"3bcfffb5-087f-41a3-930a-9ca9c28d3bdc\") "
Sep 19 01:30:49.194683 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:49.194658    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bcfffb5-087f-41a3-930a-9ca9c28d3bdc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3bcfffb5-087f-41a3-930a-9ca9c28d3bdc" (UID: "3bcfffb5-087f-41a3-930a-9ca9c28d3bdc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 01:30:49.272367 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:49.272341    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16b0d14e-300c-4572-ba96-07f080de00da" path="/var/lib/kubelet/pods/16b0d14e-300c-4572-ba96-07f080de00da/volumes"
Sep 19 01:30:49.272681 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:49.272668    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="318478b1-28c2-4c07-a3bf-83b875f6e1e1" path="/var/lib/kubelet/pods/318478b1-28c2-4c07-a3bf-83b875f6e1e1/volumes"
Sep 19 01:30:49.295077 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:49.295056    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bcfffb5-087f-41a3-930a-9ca9c28d3bdc-maas-api-service-ca\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:30:49.915894 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:49.915844    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-728f63f6-8hvxv"
Sep 19 01:30:49.916279 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:49.915897    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-728f63f6-8hvxv" event={"ID":"3bcfffb5-087f-41a3-930a-9ca9c28d3bdc","Type":"ContainerDied","Data":"0ad864a2fe87346642feea6b5478fe593b7760e3c742dd2087ea5e7a7627eb0d"}
Sep 19 01:30:49.916279 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:49.915952    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ad864a2fe87346642feea6b5478fe593b7760e3c742dd2087ea5e7a7627eb0d"
Sep 19 01:30:52.106299 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:52.106266    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c9nsz"]
Sep 19 01:30:52.528503 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:52.528420    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-cf0ace/explicit-model-cf0ace-kserve-7bf66584b5-x55tm" podUID="57d37a6f-1f68-404c-9574-9339bd408486" containerName="main" containerID="cri-o://3b86f161bd776aa6bea96200aff4e32461ce2d06ed7c9627ea446ee94ecdaf99" gracePeriod=45
Sep 19 01:30:52.771936 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:52.771913    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-cf0ace/explicit-model-cf0ace-kserve-7bf66584b5-x55tm"
Sep 19 01:30:52.923532 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:52.923504    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57d37a6f-1f68-404c-9574-9339bd408486-home\") pod \"57d37a6f-1f68-404c-9574-9339bd408486\" (UID: \"57d37a6f-1f68-404c-9574-9339bd408486\") "
Sep 19 01:30:52.923532 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:52.923536    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57d37a6f-1f68-404c-9574-9339bd408486-dshm\") pod \"57d37a6f-1f68-404c-9574-9339bd408486\" (UID: \"57d37a6f-1f68-404c-9574-9339bd408486\") "
Sep 19 01:30:52.923725 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:52.923556    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57d37a6f-1f68-404c-9574-9339bd408486-tmp-dir\") pod \"57d37a6f-1f68-404c-9574-9339bd408486\" (UID: \"57d37a6f-1f68-404c-9574-9339bd408486\") "
Sep 19 01:30:52.923725 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:52.923573    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57d37a6f-1f68-404c-9574-9339bd408486-tls-certs\") pod \"57d37a6f-1f68-404c-9574-9339bd408486\" (UID: \"57d37a6f-1f68-404c-9574-9339bd408486\") "
Sep 19 01:30:52.923725 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:52.923624    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57d37a6f-1f68-404c-9574-9339bd408486-model-cache\") pod \"57d37a6f-1f68-404c-9574-9339bd408486\" (UID: \"57d37a6f-1f68-404c-9574-9339bd408486\") "
Sep 19 01:30:52.923725 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:52.923661    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pwtzg\" (UniqueName: \"kubernetes.io/projected/57d37a6f-1f68-404c-9574-9339bd408486-kube-api-access-pwtzg\") pod \"57d37a6f-1f68-404c-9574-9339bd408486\" (UID: \"57d37a6f-1f68-404c-9574-9339bd408486\") "
Sep 19 01:30:52.923979 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:52.923755    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57d37a6f-1f68-404c-9574-9339bd408486-home" (OuterVolumeSpecName: "home") pod "57d37a6f-1f68-404c-9574-9339bd408486" (UID: "57d37a6f-1f68-404c-9574-9339bd408486"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 01:30:52.923979 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:52.923805    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57d37a6f-1f68-404c-9574-9339bd408486-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "57d37a6f-1f68-404c-9574-9339bd408486" (UID: "57d37a6f-1f68-404c-9574-9339bd408486"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 01:30:52.923979 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:52.923921    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57d37a6f-1f68-404c-9574-9339bd408486-model-cache" (OuterVolumeSpecName: "model-cache") pod "57d37a6f-1f68-404c-9574-9339bd408486" (UID: "57d37a6f-1f68-404c-9574-9339bd408486"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 01:30:52.924109 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:52.924024    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57d37a6f-1f68-404c-9574-9339bd408486-tmp-dir\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:30:52.924109 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:52.924038    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57d37a6f-1f68-404c-9574-9339bd408486-model-cache\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:30:52.924109 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:52.924045    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57d37a6f-1f68-404c-9574-9339bd408486-home\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:30:52.925881 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:52.925843    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57d37a6f-1f68-404c-9574-9339bd408486-dshm" (OuterVolumeSpecName: "dshm") pod "57d37a6f-1f68-404c-9574-9339bd408486" (UID: "57d37a6f-1f68-404c-9574-9339bd408486"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 01:30:52.925973 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:52.925956    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57d37a6f-1f68-404c-9574-9339bd408486-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "57d37a6f-1f68-404c-9574-9339bd408486" (UID: "57d37a6f-1f68-404c-9574-9339bd408486"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 01:30:52.926045 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:52.926011    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57d37a6f-1f68-404c-9574-9339bd408486-kube-api-access-pwtzg" (OuterVolumeSpecName: "kube-api-access-pwtzg") pod "57d37a6f-1f68-404c-9574-9339bd408486" (UID: "57d37a6f-1f68-404c-9574-9339bd408486"). InnerVolumeSpecName "kube-api-access-pwtzg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 01:30:52.932088 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:52.932048    2581 generic.go:356] "Generic (PLEG): container finished" podID="57d37a6f-1f68-404c-9574-9339bd408486" containerID="3b86f161bd776aa6bea96200aff4e32461ce2d06ed7c9627ea446ee94ecdaf99" exitCode=0
Sep 19 01:30:52.932211 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:52.932084    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-cf0ace/explicit-model-cf0ace-kserve-7bf66584b5-x55tm" event={"ID":"57d37a6f-1f68-404c-9574-9339bd408486","Type":"ContainerDied","Data":"3b86f161bd776aa6bea96200aff4e32461ce2d06ed7c9627ea446ee94ecdaf99"}
Sep 19 01:30:52.932211 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:52.932160    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-cf0ace/explicit-model-cf0ace-kserve-7bf66584b5-x55tm" event={"ID":"57d37a6f-1f68-404c-9574-9339bd408486","Type":"ContainerDied","Data":"d7815199c111de19f30c12b2573b56547b5ca77264ad0ebba99294b204a63337"}
Sep 19 01:30:52.932211 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:52.932189    2581 scope.go:117] "RemoveContainer" containerID="3b86f161bd776aa6bea96200aff4e32461ce2d06ed7c9627ea446ee94ecdaf99"
Sep 19 01:30:52.932362 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:52.932223    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-cf0ace/explicit-model-cf0ace-kserve-7bf66584b5-x55tm"
Sep 19 01:30:52.943596 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:52.943580    2581 scope.go:117] "RemoveContainer" containerID="3b86f161bd776aa6bea96200aff4e32461ce2d06ed7c9627ea446ee94ecdaf99"
Sep 19 01:30:52.943909 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:30:52.943886    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b86f161bd776aa6bea96200aff4e32461ce2d06ed7c9627ea446ee94ecdaf99\": container with ID starting with 3b86f161bd776aa6bea96200aff4e32461ce2d06ed7c9627ea446ee94ecdaf99 not found: ID does not exist" containerID="3b86f161bd776aa6bea96200aff4e32461ce2d06ed7c9627ea446ee94ecdaf99"
Sep 19 01:30:52.944003 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:52.943912    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b86f161bd776aa6bea96200aff4e32461ce2d06ed7c9627ea446ee94ecdaf99"} err="failed to get container status \"3b86f161bd776aa6bea96200aff4e32461ce2d06ed7c9627ea446ee94ecdaf99\": rpc error: code = NotFound desc = could not find container \"3b86f161bd776aa6bea96200aff4e32461ce2d06ed7c9627ea446ee94ecdaf99\": container with ID starting with 3b86f161bd776aa6bea96200aff4e32461ce2d06ed7c9627ea446ee94ecdaf99 not found: ID does not exist"
Sep 19 01:30:52.981656 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:52.981629    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-cf0ace/explicit-model-cf0ace-kserve-7bf66584b5-x55tm"]
Sep 19 01:30:52.993145 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:52.993120    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-cf0ace/explicit-model-cf0ace-kserve-7bf66584b5-x55tm"]
Sep 19 01:30:53.024744 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:53.024724    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57d37a6f-1f68-404c-9574-9339bd408486-dshm\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:30:53.024828 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:53.024748    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57d37a6f-1f68-404c-9574-9339bd408486-tls-certs\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:30:53.024828 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:53.024760    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwtzg\" (UniqueName: \"kubernetes.io/projected/57d37a6f-1f68-404c-9574-9339bd408486-kube-api-access-pwtzg\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:30:53.272444 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:53.272382    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57d37a6f-1f68-404c-9574-9339bd408486" path="/var/lib/kubelet/pods/57d37a6f-1f68-404c-9574-9339bd408486/volumes"
Sep 19 01:30:57.290239 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:57.290201    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-728f63f6-77ddb8bf4-f47df"]
Sep 19 01:30:57.290686 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:57.290439    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-728f63f6-77ddb8bf4-f47df" podUID="bad73288-a138-4b61-9a2e-316af32f0ca0" containerName="payload-pre-processing" containerID="cri-o://7da10cb6e1f43fbf027752f086e38cecd562cfd6a76acaf17beebb83c8df16dc" gracePeriod=30
Sep 19 01:30:57.463985 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:57.463955    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-728f63f6-8hvxv"]
Sep 19 01:30:57.469614 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:57.469588    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-728f63f6-8hvxv"]
Sep 19 01:30:57.534030 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:57.534005    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-728f63f6-77ddb8bf4-f47df"
Sep 19 01:30:57.554957 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:57.554888    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bad73288-a138-4b61-9a2e-316af32f0ca0-plugins-config-volume\") pod \"bad73288-a138-4b61-9a2e-316af32f0ca0\" (UID: \"bad73288-a138-4b61-9a2e-316af32f0ca0\") "
Sep 19 01:30:57.555066 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:57.554982    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2n677\" (UniqueName: \"kubernetes.io/projected/bad73288-a138-4b61-9a2e-316af32f0ca0-kube-api-access-2n677\") pod \"bad73288-a138-4b61-9a2e-316af32f0ca0\" (UID: \"bad73288-a138-4b61-9a2e-316af32f0ca0\") "
Sep 19 01:30:57.555230 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:57.555206    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bad73288-a138-4b61-9a2e-316af32f0ca0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bad73288-a138-4b61-9a2e-316af32f0ca0" (UID: "bad73288-a138-4b61-9a2e-316af32f0ca0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 01:30:57.557218 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:57.557185    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bad73288-a138-4b61-9a2e-316af32f0ca0-kube-api-access-2n677" (OuterVolumeSpecName: "kube-api-access-2n677") pod "bad73288-a138-4b61-9a2e-316af32f0ca0" (UID: "bad73288-a138-4b61-9a2e-316af32f0ca0"). InnerVolumeSpecName "kube-api-access-2n677". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 01:30:57.655792 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:57.655768    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2n677\" (UniqueName: \"kubernetes.io/projected/bad73288-a138-4b61-9a2e-316af32f0ca0-kube-api-access-2n677\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:30:57.655792 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:57.655789    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bad73288-a138-4b61-9a2e-316af32f0ca0-plugins-config-volume\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:30:57.958660 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:57.958627    2581 generic.go:356] "Generic (PLEG): container finished" podID="bad73288-a138-4b61-9a2e-316af32f0ca0" containerID="7da10cb6e1f43fbf027752f086e38cecd562cfd6a76acaf17beebb83c8df16dc" exitCode=0
Sep 19 01:30:57.958819 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:57.958672    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-728f63f6-77ddb8bf4-f47df"
Sep 19 01:30:57.958819 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:57.958709    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-728f63f6-77ddb8bf4-f47df" event={"ID":"bad73288-a138-4b61-9a2e-316af32f0ca0","Type":"ContainerDied","Data":"7da10cb6e1f43fbf027752f086e38cecd562cfd6a76acaf17beebb83c8df16dc"}
Sep 19 01:30:57.958819 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:57.958763    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-728f63f6-77ddb8bf4-f47df" event={"ID":"bad73288-a138-4b61-9a2e-316af32f0ca0","Type":"ContainerDied","Data":"85c3e09a30837c04d44f079ddf847f7bc4df1c60fd3a39f94ea3dbe2dc64dd48"}
Sep 19 01:30:57.958819 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:57.958787    2581 scope.go:117] "RemoveContainer" containerID="7da10cb6e1f43fbf027752f086e38cecd562cfd6a76acaf17beebb83c8df16dc"
Sep 19 01:30:57.967910 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:57.967891    2581 scope.go:117] "RemoveContainer" containerID="7da10cb6e1f43fbf027752f086e38cecd562cfd6a76acaf17beebb83c8df16dc"
Sep 19 01:30:57.968192 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:30:57.968171    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7da10cb6e1f43fbf027752f086e38cecd562cfd6a76acaf17beebb83c8df16dc\": container with ID starting with 7da10cb6e1f43fbf027752f086e38cecd562cfd6a76acaf17beebb83c8df16dc not found: ID does not exist" containerID="7da10cb6e1f43fbf027752f086e38cecd562cfd6a76acaf17beebb83c8df16dc"
Sep 19 01:30:57.968249 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:57.968198    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7da10cb6e1f43fbf027752f086e38cecd562cfd6a76acaf17beebb83c8df16dc"} err="failed to get container status \"7da10cb6e1f43fbf027752f086e38cecd562cfd6a76acaf17beebb83c8df16dc\": rpc error: code = NotFound desc = could not find container \"7da10cb6e1f43fbf027752f086e38cecd562cfd6a76acaf17beebb83c8df16dc\": container with ID starting with 7da10cb6e1f43fbf027752f086e38cecd562cfd6a76acaf17beebb83c8df16dc not found: ID does not exist"
Sep 19 01:30:57.989774 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:57.989748    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-728f63f6-77ddb8bf4-f47df"]
Sep 19 01:30:57.993941 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:57.993923    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-728f63f6-77ddb8bf4-f47df"]
Sep 19 01:30:59.271695 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:59.271660    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3bcfffb5-087f-41a3-930a-9ca9c28d3bdc" path="/var/lib/kubelet/pods/3bcfffb5-087f-41a3-930a-9ca9c28d3bdc/volumes"
Sep 19 01:30:59.272153 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:30:59.272046    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bad73288-a138-4b61-9a2e-316af32f0ca0" path="/var/lib/kubelet/pods/bad73288-a138-4b61-9a2e-316af32f0ca0/volumes"
Sep 19 01:31:09.496276 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:31:09.496246    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c9nsz"]
Sep 19 01:31:10.800616 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:31:10.800570    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-a136cc/noten-model-a136cc-kserve-6b54668f47-rsl52"]
Sep 19 01:31:10.801379 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:31:10.801351    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bcfffb5-087f-41a3-930a-9ca9c28d3bdc" containerName="revoke-keys"
Sep 19 01:31:10.801474 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:31:10.801384    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bcfffb5-087f-41a3-930a-9ca9c28d3bdc" containerName="revoke-keys"
Sep 19 01:31:10.801474 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:31:10.801410    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bad73288-a138-4b61-9a2e-316af32f0ca0" containerName="payload-pre-processing"
Sep 19 01:31:10.801474 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:31:10.801420    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="bad73288-a138-4b61-9a2e-316af32f0ca0" containerName="payload-pre-processing"
Sep 19 01:31:10.801474 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:31:10.801444    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="318478b1-28c2-4c07-a3bf-83b875f6e1e1" containerName="payload-pre-processing"
Sep 19 01:31:10.801474 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:31:10.801455    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="318478b1-28c2-4c07-a3bf-83b875f6e1e1" containerName="payload-pre-processing"
Sep 19 01:31:10.801722 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:31:10.801484    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57d37a6f-1f68-404c-9574-9339bd408486" containerName="main"
Sep 19 01:31:10.801722 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:31:10.801494    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="57d37a6f-1f68-404c-9574-9339bd408486" containerName="main"
Sep 19 01:31:10.801722 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:31:10.801674    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bcfffb5-087f-41a3-930a-9ca9c28d3bdc" containerName="revoke-keys"
Sep 19 01:31:10.801722 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:31:10.801697    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="bad73288-a138-4b61-9a2e-316af32f0ca0" containerName="payload-pre-processing"
Sep 19 01:31:10.801722 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:31:10.801715    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="318478b1-28c2-4c07-a3bf-83b875f6e1e1" containerName="payload-pre-processing"
Sep 19 01:31:10.801985 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:31:10.801726    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="57d37a6f-1f68-404c-9574-9339bd408486" containerName="main"
Sep 19 01:31:10.805660 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:31:10.805637    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-a136cc/noten-model-a136cc-kserve-6b54668f47-rsl52"
Sep 19 01:31:10.809015 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:31:10.808993    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-a136cc\"/\"noten-model-a136cc-kserve-self-signed-certs\""
Sep 19 01:31:10.809123 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:31:10.808993    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-a136cc\"/\"kube-root-ca.crt\""
Sep 19 01:31:10.809123 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:31:10.809048    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-a136cc\"/\"openshift-service-ca.crt\""
Sep 19 01:31:10.809123 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:31:10.809047    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-a136cc\"/\"default-dockercfg-vdk89\""
Sep 19 01:31:10.815819 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:31:10.815801    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-a136cc/noten-model-a136cc-kserve-6b54668f47-rsl52"]
Sep 19 01:31:10.855764 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:31:10.855739    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5jb5\" (UniqueName: \"kubernetes.io/projected/41bf5234-3580-4d96-8615-2c4e5a7ffabb-kube-api-access-j5jb5\") pod \"noten-model-a136cc-kserve-6b54668f47-rsl52\" (UID: \"41bf5234-3580-4d96-8615-2c4e5a7ffabb\") " pod="ai-tenant-e2e-noten-a136cc/noten-model-a136cc-kserve-6b54668f47-rsl52"
Sep 19 01:31:10.855853 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:31:10.855785    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41bf5234-3580-4d96-8615-2c4e5a7ffabb-home\") pod \"noten-model-a136cc-kserve-6b54668f47-rsl52\" (UID: \"41bf5234-3580-4d96-8615-2c4e5a7ffabb\") " pod="ai-tenant-e2e-noten-a136cc/noten-model-a136cc-kserve-6b54668f47-rsl52"
Sep 19 01:31:10.855853 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:31:10.855805    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41bf5234-3580-4d96-8615-2c4e5a7ffabb-dshm\") pod \"noten-model-a136cc-kserve-6b54668f47-rsl52\" (UID: \"41bf5234-3580-4d96-8615-2c4e5a7ffabb\") " pod="ai-tenant-e2e-noten-a136cc/noten-model-a136cc-kserve-6b54668f47-rsl52"
Sep 19 01:31:10.855853 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:31:10.855821    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41bf5234-3580-4d96-8615-2c4e5a7ffabb-tls-certs\") pod \"noten-model-a136cc-kserve-6b54668f47-rsl52\" (UID: \"41bf5234-3580-4d96-8615-2c4e5a7ffabb\") " pod="ai-tenant-e2e-noten-a136cc/noten-model-a136cc-kserve-6b54668f47-rsl52"
Sep 19 01:31:10.855993 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:31:10.855856    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41bf5234-3580-4d96-8615-2c4e5a7ffabb-tmp-dir\") pod \"noten-model-a136cc-kserve-6b54668f47-rsl52\" (UID: \"41bf5234-3580-4d96-8615-2c4e5a7ffabb\") " pod="ai-tenant-e2e-noten-a136cc/noten-model-a136cc-kserve-6b54668f47-rsl52"
Sep 19 01:31:10.855993 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:31:10.855938    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41bf5234-3580-4d96-8615-2c4e5a7ffabb-model-cache\") pod \"noten-model-a136cc-kserve-6b54668f47-rsl52\" (UID: \"41bf5234-3580-4d96-8615-2c4e5a7ffabb\") " pod="ai-tenant-e2e-noten-a136cc/noten-model-a136cc-kserve-6b54668f47-rsl52"
Sep 19 01:31:10.957075 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:31:10.957052    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41bf5234-3580-4d96-8615-2c4e5a7ffabb-model-cache\") pod \"noten-model-a136cc-kserve-6b54668f47-rsl52\" (UID: \"41bf5234-3580-4d96-8615-2c4e5a7ffabb\") " pod="ai-tenant-e2e-noten-a136cc/noten-model-a136cc-kserve-6b54668f47-rsl52"
Sep 19 01:31:10.957158 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:31:10.957087    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5jb5\" (UniqueName: \"kubernetes.io/projected/41bf5234-3580-4d96-8615-2c4e5a7ffabb-kube-api-access-j5jb5\") pod \"noten-model-a136cc-kserve-6b54668f47-rsl52\" (UID: \"41bf5234-3580-4d96-8615-2c4e5a7ffabb\") " pod="ai-tenant-e2e-noten-a136cc/noten-model-a136cc-kserve-6b54668f47-rsl52"
Sep 19 01:31:10.957250 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:31:10.957231    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41bf5234-3580-4d96-8615-2c4e5a7ffabb-home\") pod \"noten-model-a136cc-kserve-6b54668f47-rsl52\" (UID: \"41bf5234-3580-4d96-8615-2c4e5a7ffabb\") " pod="ai-tenant-e2e-noten-a136cc/noten-model-a136cc-kserve-6b54668f47-rsl52"
Sep 19 01:31:10.957312 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:31:10.957279    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41bf5234-3580-4d96-8615-2c4e5a7ffabb-dshm\") pod \"noten-model-a136cc-kserve-6b54668f47-rsl52\" (UID: \"41bf5234-3580-4d96-8615-2c4e5a7ffabb\") " pod="ai-tenant-e2e-noten-a136cc/noten-model-a136cc-kserve-6b54668f47-rsl52"
Sep 19 01:31:10.957369 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:31:10.957308    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41bf5234-3580-4d96-8615-2c4e5a7ffabb-tls-certs\") pod \"noten-model-a136cc-kserve-6b54668f47-rsl52\" (UID: \"41bf5234-3580-4d96-8615-2c4e5a7ffabb\") " pod="ai-tenant-e2e-noten-a136cc/noten-model-a136cc-kserve-6b54668f47-rsl52"
Sep 19 01:31:10.957369 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:31:10.957349    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41bf5234-3580-4d96-8615-2c4e5a7ffabb-tmp-dir\") pod \"noten-model-a136cc-kserve-6b54668f47-rsl52\" (UID: \"41bf5234-3580-4d96-8615-2c4e5a7ffabb\") " pod="ai-tenant-e2e-noten-a136cc/noten-model-a136cc-kserve-6b54668f47-rsl52"
Sep 19 01:31:10.957488 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:31:10.957469    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41bf5234-3580-4d96-8615-2c4e5a7ffabb-model-cache\") pod \"noten-model-a136cc-kserve-6b54668f47-rsl52\" (UID: \"41bf5234-3580-4d96-8615-2c4e5a7ffabb\") " pod="ai-tenant-e2e-noten-a136cc/noten-model-a136cc-kserve-6b54668f47-rsl52"
Sep 19 01:31:10.957549 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:31:10.957497    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41bf5234-3580-4d96-8615-2c4e5a7ffabb-home\") pod \"noten-model-a136cc-kserve-6b54668f47-rsl52\" (UID: \"41bf5234-3580-4d96-8615-2c4e5a7ffabb\") " pod="ai-tenant-e2e-noten-a136cc/noten-model-a136cc-kserve-6b54668f47-rsl52"
Sep 19 01:31:10.957700 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:31:10.957682    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41bf5234-3580-4d96-8615-2c4e5a7ffabb-tmp-dir\") pod \"noten-model-a136cc-kserve-6b54668f47-rsl52\" (UID: \"41bf5234-3580-4d96-8615-2c4e5a7ffabb\") " pod="ai-tenant-e2e-noten-a136cc/noten-model-a136cc-kserve-6b54668f47-rsl52"
Sep 19 01:31:10.959525 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:31:10.959506    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41bf5234-3580-4d96-8615-2c4e5a7ffabb-dshm\") pod \"noten-model-a136cc-kserve-6b54668f47-rsl52\" (UID: \"41bf5234-3580-4d96-8615-2c4e5a7ffabb\") " pod="ai-tenant-e2e-noten-a136cc/noten-model-a136cc-kserve-6b54668f47-rsl52"
Sep 19 01:31:10.959785 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:31:10.959767    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41bf5234-3580-4d96-8615-2c4e5a7ffabb-tls-certs\") pod \"noten-model-a136cc-kserve-6b54668f47-rsl52\" (UID: \"41bf5234-3580-4d96-8615-2c4e5a7ffabb\") " pod="ai-tenant-e2e-noten-a136cc/noten-model-a136cc-kserve-6b54668f47-rsl52"
Sep 19 01:31:10.965507 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:31:10.965490    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5jb5\" (UniqueName: \"kubernetes.io/projected/41bf5234-3580-4d96-8615-2c4e5a7ffabb-kube-api-access-j5jb5\") pod \"noten-model-a136cc-kserve-6b54668f47-rsl52\" (UID: \"41bf5234-3580-4d96-8615-2c4e5a7ffabb\") " pod="ai-tenant-e2e-noten-a136cc/noten-model-a136cc-kserve-6b54668f47-rsl52"
Sep 19 01:31:11.114383 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:31:11.114345    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-a136cc/noten-model-a136cc-kserve-6b54668f47-rsl52"
Sep 19 01:31:11.386400 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:31:11.386377    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-a136cc/noten-model-a136cc-kserve-6b54668f47-rsl52"]
Sep 19 01:31:11.388514 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:31:11.388484    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod41bf5234_3580_4d96_8615_2c4e5a7ffabb.slice/crio-0c5c39f90017bad8eaa8b5f9f1635edaf82fa309aafe474ea46c1140fb9469a5 WatchSource:0}: Error finding container 0c5c39f90017bad8eaa8b5f9f1635edaf82fa309aafe474ea46c1140fb9469a5: Status 404 returned error can't find the container with id 0c5c39f90017bad8eaa8b5f9f1635edaf82fa309aafe474ea46c1140fb9469a5
Sep 19 01:31:12.036139 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:31:12.036102    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-a136cc/noten-model-a136cc-kserve-6b54668f47-rsl52" event={"ID":"41bf5234-3580-4d96-8615-2c4e5a7ffabb","Type":"ContainerStarted","Data":"0c7e10e5672cf5dc71b0daab611328244a53d881877e328132747ebc28a8547b"}
Sep 19 01:31:12.036139 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:31:12.036143    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-a136cc/noten-model-a136cc-kserve-6b54668f47-rsl52" event={"ID":"41bf5234-3580-4d96-8615-2c4e5a7ffabb","Type":"ContainerStarted","Data":"0c5c39f90017bad8eaa8b5f9f1635edaf82fa309aafe474ea46c1140fb9469a5"}
Sep 19 01:31:12.054512 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:31:12.054464    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-a136cc/noten-model-a136cc-kserve-6b54668f47-rsl52" podStartSLOduration=2.054449908 podStartE2EDuration="2.054449908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 01:31:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 01:31:12.052617682 +0000 UTC m=+1122.940286759" watchObservedRunningTime="2026-09-19 01:31:12.054449908 +0000 UTC m=+1122.942119006"
Sep 19 01:31:21.115021 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:31:21.114987    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-a136cc/noten-model-a136cc-kserve-6b54668f47-rsl52"
Sep 19 01:31:21.115418 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:31:21.115050    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-a136cc/noten-model-a136cc-kserve-6b54668f47-rsl52"
Sep 19 01:31:21.127066 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:31:21.127040    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-a136cc/noten-model-a136cc-kserve-6b54668f47-rsl52"
Sep 19 01:31:22.098273 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:31:22.098241    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-a136cc/noten-model-a136cc-kserve-6b54668f47-rsl52"
Sep 19 01:31:34.200096 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:31:34.200062    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c9nsz"]
Sep 19 01:31:50.662794 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:31:50.662712    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-a136cc/noten-model-a136cc-kserve-6b54668f47-rsl52"]
Sep 19 01:32:05.682055 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:05.681994    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-a136cc/noten-model-a136cc-kserve-6b54668f47-rsl52" podUID="41bf5234-3580-4d96-8615-2c4e5a7ffabb" containerName="main" containerID="cri-o://0c7e10e5672cf5dc71b0daab611328244a53d881877e328132747ebc28a8547b" gracePeriod=45
Sep 19 01:32:05.929226 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:05.929198    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-a136cc/noten-model-a136cc-kserve-6b54668f47-rsl52"
Sep 19 01:32:06.000757 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:06.000677    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-8e708f/distinct-w0-8e708f-kserve-6f58868fbf-mfmbr"]
Sep 19 01:32:06.001017 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:06.001005    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41bf5234-3580-4d96-8615-2c4e5a7ffabb" containerName="main"
Sep 19 01:32:06.001071 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:06.001019    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="41bf5234-3580-4d96-8615-2c4e5a7ffabb" containerName="main"
Sep 19 01:32:06.001106 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:06.001083    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="41bf5234-3580-4d96-8615-2c4e5a7ffabb" containerName="main"
Sep 19 01:32:06.003893 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:06.003877    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-8e708f/distinct-w0-8e708f-kserve-6f58868fbf-mfmbr"
Sep 19 01:32:06.006673 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:06.006651    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-8e708f\"/\"distinct-w0-8e708f-kserve-self-signed-certs\""
Sep 19 01:32:06.011251 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:06.011229    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41bf5234-3580-4d96-8615-2c4e5a7ffabb-model-cache\") pod \"41bf5234-3580-4d96-8615-2c4e5a7ffabb\" (UID: \"41bf5234-3580-4d96-8615-2c4e5a7ffabb\") "
Sep 19 01:32:06.011353 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:06.011273    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j5jb5\" (UniqueName: \"kubernetes.io/projected/41bf5234-3580-4d96-8615-2c4e5a7ffabb-kube-api-access-j5jb5\") pod \"41bf5234-3580-4d96-8615-2c4e5a7ffabb\" (UID: \"41bf5234-3580-4d96-8615-2c4e5a7ffabb\") "
Sep 19 01:32:06.011353 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:06.011336    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41bf5234-3580-4d96-8615-2c4e5a7ffabb-tmp-dir\") pod \"41bf5234-3580-4d96-8615-2c4e5a7ffabb\" (UID: \"41bf5234-3580-4d96-8615-2c4e5a7ffabb\") "
Sep 19 01:32:06.011470 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:06.011386    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41bf5234-3580-4d96-8615-2c4e5a7ffabb-tls-certs\") pod \"41bf5234-3580-4d96-8615-2c4e5a7ffabb\" (UID: \"41bf5234-3580-4d96-8615-2c4e5a7ffabb\") "
Sep 19 01:32:06.011470 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:06.011437    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41bf5234-3580-4d96-8615-2c4e5a7ffabb-home\") pod \"41bf5234-3580-4d96-8615-2c4e5a7ffabb\" (UID: \"41bf5234-3580-4d96-8615-2c4e5a7ffabb\") "
Sep 19 01:32:06.011585 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:06.011468    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41bf5234-3580-4d96-8615-2c4e5a7ffabb-model-cache" (OuterVolumeSpecName: "model-cache") pod "41bf5234-3580-4d96-8615-2c4e5a7ffabb" (UID: "41bf5234-3580-4d96-8615-2c4e5a7ffabb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 01:32:06.011585 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:06.011478    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41bf5234-3580-4d96-8615-2c4e5a7ffabb-dshm\") pod \"41bf5234-3580-4d96-8615-2c4e5a7ffabb\" (UID: \"41bf5234-3580-4d96-8615-2c4e5a7ffabb\") "
Sep 19 01:32:06.011759 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:06.011733    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41bf5234-3580-4d96-8615-2c4e5a7ffabb-home" (OuterVolumeSpecName: "home") pod "41bf5234-3580-4d96-8615-2c4e5a7ffabb" (UID: "41bf5234-3580-4d96-8615-2c4e5a7ffabb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 01:32:06.011828 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:06.011806    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41bf5234-3580-4d96-8615-2c4e5a7ffabb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "41bf5234-3580-4d96-8615-2c4e5a7ffabb" (UID: "41bf5234-3580-4d96-8615-2c4e5a7ffabb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 01:32:06.011926 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:06.011835    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41bf5234-3580-4d96-8615-2c4e5a7ffabb-home\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:32:06.011926 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:06.011854    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41bf5234-3580-4d96-8615-2c4e5a7ffabb-model-cache\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:32:06.013714 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:06.013687    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41bf5234-3580-4d96-8615-2c4e5a7ffabb-kube-api-access-j5jb5" (OuterVolumeSpecName: "kube-api-access-j5jb5") pod "41bf5234-3580-4d96-8615-2c4e5a7ffabb" (UID: "41bf5234-3580-4d96-8615-2c4e5a7ffabb"). InnerVolumeSpecName "kube-api-access-j5jb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 01:32:06.013881 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:06.013830    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41bf5234-3580-4d96-8615-2c4e5a7ffabb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "41bf5234-3580-4d96-8615-2c4e5a7ffabb" (UID: "41bf5234-3580-4d96-8615-2c4e5a7ffabb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 01:32:06.013983 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:06.013883    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41bf5234-3580-4d96-8615-2c4e5a7ffabb-dshm" (OuterVolumeSpecName: "dshm") pod "41bf5234-3580-4d96-8615-2c4e5a7ffabb" (UID: "41bf5234-3580-4d96-8615-2c4e5a7ffabb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 01:32:06.018043 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:06.018021    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-8e708f/distinct-w0-8e708f-kserve-6f58868fbf-mfmbr"]
Sep 19 01:32:06.112390 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:06.112363    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99de318b-4f35-4f13-8a91-099ad5aaa66a-dshm\") pod \"distinct-w0-8e708f-kserve-6f58868fbf-mfmbr\" (UID: \"99de318b-4f35-4f13-8a91-099ad5aaa66a\") " pod="e2e-models-e2e-worker-w0-8e708f/distinct-w0-8e708f-kserve-6f58868fbf-mfmbr"
Sep 19 01:32:06.112501 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:06.112396    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99de318b-4f35-4f13-8a91-099ad5aaa66a-model-cache\") pod \"distinct-w0-8e708f-kserve-6f58868fbf-mfmbr\" (UID: \"99de318b-4f35-4f13-8a91-099ad5aaa66a\") " pod="e2e-models-e2e-worker-w0-8e708f/distinct-w0-8e708f-kserve-6f58868fbf-mfmbr"
Sep 19 01:32:06.112501 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:06.112457    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99de318b-4f35-4f13-8a91-099ad5aaa66a-home\") pod \"distinct-w0-8e708f-kserve-6f58868fbf-mfmbr\" (UID: \"99de318b-4f35-4f13-8a91-099ad5aaa66a\") " pod="e2e-models-e2e-worker-w0-8e708f/distinct-w0-8e708f-kserve-6f58868fbf-mfmbr"
Sep 19 01:32:06.112501 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:06.112486    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99de318b-4f35-4f13-8a91-099ad5aaa66a-tmp-dir\") pod \"distinct-w0-8e708f-kserve-6f58868fbf-mfmbr\" (UID: \"99de318b-4f35-4f13-8a91-099ad5aaa66a\") " pod="e2e-models-e2e-worker-w0-8e708f/distinct-w0-8e708f-kserve-6f58868fbf-mfmbr"
Sep 19 01:32:06.112605 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:06.112554    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99de318b-4f35-4f13-8a91-099ad5aaa66a-tls-certs\") pod \"distinct-w0-8e708f-kserve-6f58868fbf-mfmbr\" (UID: \"99de318b-4f35-4f13-8a91-099ad5aaa66a\") " pod="e2e-models-e2e-worker-w0-8e708f/distinct-w0-8e708f-kserve-6f58868fbf-mfmbr"
Sep 19 01:32:06.112605 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:06.112588    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbglr\" (UniqueName: \"kubernetes.io/projected/99de318b-4f35-4f13-8a91-099ad5aaa66a-kube-api-access-dbglr\") pod \"distinct-w0-8e708f-kserve-6f58868fbf-mfmbr\" (UID: \"99de318b-4f35-4f13-8a91-099ad5aaa66a\") " pod="e2e-models-e2e-worker-w0-8e708f/distinct-w0-8e708f-kserve-6f58868fbf-mfmbr"
Sep 19 01:32:06.112670 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:06.112624    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41bf5234-3580-4d96-8615-2c4e5a7ffabb-tmp-dir\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:32:06.112670 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:06.112635    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41bf5234-3580-4d96-8615-2c4e5a7ffabb-tls-certs\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:32:06.112670 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:06.112643    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41bf5234-3580-4d96-8615-2c4e5a7ffabb-dshm\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:32:06.112670 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:06.112657    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5jb5\" (UniqueName: \"kubernetes.io/projected/41bf5234-3580-4d96-8615-2c4e5a7ffabb-kube-api-access-j5jb5\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:32:06.213764 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:06.213735    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99de318b-4f35-4f13-8a91-099ad5aaa66a-home\") pod \"distinct-w0-8e708f-kserve-6f58868fbf-mfmbr\" (UID: \"99de318b-4f35-4f13-8a91-099ad5aaa66a\") " pod="e2e-models-e2e-worker-w0-8e708f/distinct-w0-8e708f-kserve-6f58868fbf-mfmbr"
Sep 19 01:32:06.213922 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:06.213771    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99de318b-4f35-4f13-8a91-099ad5aaa66a-tmp-dir\") pod \"distinct-w0-8e708f-kserve-6f58868fbf-mfmbr\" (UID: \"99de318b-4f35-4f13-8a91-099ad5aaa66a\") " pod="e2e-models-e2e-worker-w0-8e708f/distinct-w0-8e708f-kserve-6f58868fbf-mfmbr"
Sep 19 01:32:06.213922 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:06.213823    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99de318b-4f35-4f13-8a91-099ad5aaa66a-tls-certs\") pod \"distinct-w0-8e708f-kserve-6f58868fbf-mfmbr\" (UID: \"99de318b-4f35-4f13-8a91-099ad5aaa66a\") " pod="e2e-models-e2e-worker-w0-8e708f/distinct-w0-8e708f-kserve-6f58868fbf-mfmbr"
Sep 19 01:32:06.213922 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:06.213857    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbglr\" (UniqueName: \"kubernetes.io/projected/99de318b-4f35-4f13-8a91-099ad5aaa66a-kube-api-access-dbglr\") pod \"distinct-w0-8e708f-kserve-6f58868fbf-mfmbr\" (UID: \"99de318b-4f35-4f13-8a91-099ad5aaa66a\") " pod="e2e-models-e2e-worker-w0-8e708f/distinct-w0-8e708f-kserve-6f58868fbf-mfmbr"
Sep 19 01:32:06.214039 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:06.213936    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99de318b-4f35-4f13-8a91-099ad5aaa66a-dshm\") pod \"distinct-w0-8e708f-kserve-6f58868fbf-mfmbr\" (UID: \"99de318b-4f35-4f13-8a91-099ad5aaa66a\") " pod="e2e-models-e2e-worker-w0-8e708f/distinct-w0-8e708f-kserve-6f58868fbf-mfmbr"
Sep 19 01:32:06.214039 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:06.213961    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99de318b-4f35-4f13-8a91-099ad5aaa66a-model-cache\") pod \"distinct-w0-8e708f-kserve-6f58868fbf-mfmbr\" (UID: \"99de318b-4f35-4f13-8a91-099ad5aaa66a\") " pod="e2e-models-e2e-worker-w0-8e708f/distinct-w0-8e708f-kserve-6f58868fbf-mfmbr"
Sep 19 01:32:06.214215 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:06.214197    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99de318b-4f35-4f13-8a91-099ad5aaa66a-home\") pod \"distinct-w0-8e708f-kserve-6f58868fbf-mfmbr\" (UID: \"99de318b-4f35-4f13-8a91-099ad5aaa66a\") " pod="e2e-models-e2e-worker-w0-8e708f/distinct-w0-8e708f-kserve-6f58868fbf-mfmbr"
Sep 19 01:32:06.214281 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:06.214265    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99de318b-4f35-4f13-8a91-099ad5aaa66a-tmp-dir\") pod \"distinct-w0-8e708f-kserve-6f58868fbf-mfmbr\" (UID: \"99de318b-4f35-4f13-8a91-099ad5aaa66a\") " pod="e2e-models-e2e-worker-w0-8e708f/distinct-w0-8e708f-kserve-6f58868fbf-mfmbr"
Sep 19 01:32:06.214322 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:06.214269    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99de318b-4f35-4f13-8a91-099ad5aaa66a-model-cache\") pod \"distinct-w0-8e708f-kserve-6f58868fbf-mfmbr\" (UID: \"99de318b-4f35-4f13-8a91-099ad5aaa66a\") " pod="e2e-models-e2e-worker-w0-8e708f/distinct-w0-8e708f-kserve-6f58868fbf-mfmbr"
Sep 19 01:32:06.216382 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:06.216357    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99de318b-4f35-4f13-8a91-099ad5aaa66a-dshm\") pod \"distinct-w0-8e708f-kserve-6f58868fbf-mfmbr\" (UID: \"99de318b-4f35-4f13-8a91-099ad5aaa66a\") " pod="e2e-models-e2e-worker-w0-8e708f/distinct-w0-8e708f-kserve-6f58868fbf-mfmbr"
Sep 19 01:32:06.216594 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:06.216579    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99de318b-4f35-4f13-8a91-099ad5aaa66a-tls-certs\") pod \"distinct-w0-8e708f-kserve-6f58868fbf-mfmbr\" (UID: \"99de318b-4f35-4f13-8a91-099ad5aaa66a\") " pod="e2e-models-e2e-worker-w0-8e708f/distinct-w0-8e708f-kserve-6f58868fbf-mfmbr"
Sep 19 01:32:06.222317 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:06.222299    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbglr\" (UniqueName: \"kubernetes.io/projected/99de318b-4f35-4f13-8a91-099ad5aaa66a-kube-api-access-dbglr\") pod \"distinct-w0-8e708f-kserve-6f58868fbf-mfmbr\" (UID: \"99de318b-4f35-4f13-8a91-099ad5aaa66a\") " pod="e2e-models-e2e-worker-w0-8e708f/distinct-w0-8e708f-kserve-6f58868fbf-mfmbr"
Sep 19 01:32:06.302873 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:06.302766    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c9nsz"]
Sep 19 01:32:06.306201 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:06.306179    2581 generic.go:356] "Generic (PLEG): container finished" podID="41bf5234-3580-4d96-8615-2c4e5a7ffabb" containerID="0c7e10e5672cf5dc71b0daab611328244a53d881877e328132747ebc28a8547b" exitCode=0
Sep 19 01:32:06.306296 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:06.306224    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-a136cc/noten-model-a136cc-kserve-6b54668f47-rsl52" event={"ID":"41bf5234-3580-4d96-8615-2c4e5a7ffabb","Type":"ContainerDied","Data":"0c7e10e5672cf5dc71b0daab611328244a53d881877e328132747ebc28a8547b"}
Sep 19 01:32:06.306296 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:06.306258    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-a136cc/noten-model-a136cc-kserve-6b54668f47-rsl52" event={"ID":"41bf5234-3580-4d96-8615-2c4e5a7ffabb","Type":"ContainerDied","Data":"0c5c39f90017bad8eaa8b5f9f1635edaf82fa309aafe474ea46c1140fb9469a5"}
Sep 19 01:32:06.306296 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:06.306274    2581 scope.go:117] "RemoveContainer" containerID="0c7e10e5672cf5dc71b0daab611328244a53d881877e328132747ebc28a8547b"
Sep 19 01:32:06.306417 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:06.306236    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-a136cc/noten-model-a136cc-kserve-6b54668f47-rsl52"
Sep 19 01:32:06.314943 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:06.314771    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-8e708f/distinct-w0-8e708f-kserve-6f58868fbf-mfmbr"
Sep 19 01:32:06.317154 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:06.317135    2581 scope.go:117] "RemoveContainer" containerID="0c7e10e5672cf5dc71b0daab611328244a53d881877e328132747ebc28a8547b"
Sep 19 01:32:06.317440 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:32:06.317422    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c7e10e5672cf5dc71b0daab611328244a53d881877e328132747ebc28a8547b\": container with ID starting with 0c7e10e5672cf5dc71b0daab611328244a53d881877e328132747ebc28a8547b not found: ID does not exist" containerID="0c7e10e5672cf5dc71b0daab611328244a53d881877e328132747ebc28a8547b"
Sep 19 01:32:06.317493 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:06.317449    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c7e10e5672cf5dc71b0daab611328244a53d881877e328132747ebc28a8547b"} err="failed to get container status \"0c7e10e5672cf5dc71b0daab611328244a53d881877e328132747ebc28a8547b\": rpc error: code = NotFound desc = could not find container \"0c7e10e5672cf5dc71b0daab611328244a53d881877e328132747ebc28a8547b\": container with ID starting with 0c7e10e5672cf5dc71b0daab611328244a53d881877e328132747ebc28a8547b not found: ID does not exist"
Sep 19 01:32:06.331066 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:06.331045    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-a136cc/noten-model-a136cc-kserve-6b54668f47-rsl52"]
Sep 19 01:32:06.337325 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:06.337299    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-a136cc/noten-model-a136cc-kserve-6b54668f47-rsl52"]
Sep 19 01:32:06.594191 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:06.594150    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-8e708f/distinct-w0-8e708f-kserve-6f58868fbf-mfmbr"]
Sep 19 01:32:06.596367 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:32:06.596326    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod99de318b_4f35_4f13_8a91_099ad5aaa66a.slice/crio-a8bd3de27fa25c904d5b4078c5347b69b9e24e6d8a0547dd27cbb8d1a43c84c6 WatchSource:0}: Error finding container a8bd3de27fa25c904d5b4078c5347b69b9e24e6d8a0547dd27cbb8d1a43c84c6: Status 404 returned error can't find the container with id a8bd3de27fa25c904d5b4078c5347b69b9e24e6d8a0547dd27cbb8d1a43c84c6
Sep 19 01:32:07.271554 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:07.271522    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41bf5234-3580-4d96-8615-2c4e5a7ffabb" path="/var/lib/kubelet/pods/41bf5234-3580-4d96-8615-2c4e5a7ffabb/volumes"
Sep 19 01:32:07.314545 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:07.314514    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-8e708f/distinct-w0-8e708f-kserve-6f58868fbf-mfmbr" event={"ID":"99de318b-4f35-4f13-8a91-099ad5aaa66a","Type":"ContainerStarted","Data":"77472ae496c2dfc75e7d36ffaf6ee6941a6f4d3fbd82a6295f9502b34edb4869"}
Sep 19 01:32:07.314545 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:07.314550    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-8e708f/distinct-w0-8e708f-kserve-6f58868fbf-mfmbr" event={"ID":"99de318b-4f35-4f13-8a91-099ad5aaa66a","Type":"ContainerStarted","Data":"a8bd3de27fa25c904d5b4078c5347b69b9e24e6d8a0547dd27cbb8d1a43c84c6"}
Sep 19 01:32:07.337660 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:07.337604    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-8e708f/distinct-w0-8e708f-kserve-6f58868fbf-mfmbr" podStartSLOduration=2.337586655 podStartE2EDuration="2.337586655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 01:32:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 01:32:07.336795854 +0000 UTC m=+1178.224464930" watchObservedRunningTime="2026-09-19 01:32:07.337586655 +0000 UTC m=+1178.225255733"
Sep 19 01:32:09.272684 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:09.272651    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-c4ce82/test-model-c4ce82-kserve-7b9cfc5c47-tqrkm"]
Sep 19 01:32:12.069524 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:12.069487    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-5a005d/test-model-5a005d-kserve-844f5dc4c9-f72bf"]
Sep 19 01:32:12.216431 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:12.216399    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c4ce82-qnwbw"]
Sep 19 01:32:12.220068 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:12.220051    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c4ce82-qnwbw"
Sep 19 01:32:12.222235 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:12.222211    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 19 01:32:12.222331 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:12.222290    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s7pvt\""
Sep 19 01:32:12.227824 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:12.227775    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c4ce82-qnwbw"]
Sep 19 01:32:12.267204 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:12.267174    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e08e5e7-4fb9-4431-bf87-f03015627c3a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c4ce82-qnwbw\" (UID: \"5e08e5e7-4fb9-4431-bf87-f03015627c3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c4ce82-qnwbw"
Sep 19 01:32:12.367882 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:12.367837    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e08e5e7-4fb9-4431-bf87-f03015627c3a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c4ce82-qnwbw\" (UID: \"5e08e5e7-4fb9-4431-bf87-f03015627c3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c4ce82-qnwbw"
Sep 19 01:32:12.368503 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:12.368480    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e08e5e7-4fb9-4431-bf87-f03015627c3a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c4ce82-qnwbw\" (UID: \"5e08e5e7-4fb9-4431-bf87-f03015627c3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c4ce82-qnwbw"
Sep 19 01:32:12.531655 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:12.531624    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c4ce82-qnwbw"
Sep 19 01:32:12.801709 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:12.801638    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c4ce82-qnwbw"]
Sep 19 01:32:12.805199 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:32:12.805172    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e08e5e7_4fb9_4431_bf87_f03015627c3a.slice/crio-373636f3a293dbb22dda5bc35f11a29c7c811a76af53a3b647bb62801bc06e26 WatchSource:0}: Error finding container 373636f3a293dbb22dda5bc35f11a29c7c811a76af53a3b647bb62801bc06e26: Status 404 returned error can't find the container with id 373636f3a293dbb22dda5bc35f11a29c7c811a76af53a3b647bb62801bc06e26
Sep 19 01:32:13.344636 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:13.344603    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c4ce82-qnwbw" event={"ID":"5e08e5e7-4fb9-4431-bf87-f03015627c3a","Type":"ContainerStarted","Data":"7ad6ea9ccda53c113f865a2cd9ae68844013c2cefc9b6f008de981002746b265"}
Sep 19 01:32:13.345007 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:13.344654    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c4ce82-qnwbw" event={"ID":"5e08e5e7-4fb9-4431-bf87-f03015627c3a","Type":"ContainerStarted","Data":"373636f3a293dbb22dda5bc35f11a29c7c811a76af53a3b647bb62801bc06e26"}
Sep 19 01:32:13.359967 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:13.359921    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c4ce82-qnwbw" podStartSLOduration=1.359901587 podStartE2EDuration="1.359901587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 01:32:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 01:32:13.359261565 +0000 UTC m=+1184.246930668" watchObservedRunningTime="2026-09-19 01:32:13.359901587 +0000 UTC m=+1184.247570666"
Sep 19 01:32:14.351743 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:14.351707    2581 generic.go:356] "Generic (PLEG): container finished" podID="5e08e5e7-4fb9-4431-bf87-f03015627c3a" containerID="7ad6ea9ccda53c113f865a2cd9ae68844013c2cefc9b6f008de981002746b265" exitCode=0
Sep 19 01:32:14.352192 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:14.351789    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c4ce82-qnwbw" event={"ID":"5e08e5e7-4fb9-4431-bf87-f03015627c3a","Type":"ContainerDied","Data":"7ad6ea9ccda53c113f865a2cd9ae68844013c2cefc9b6f008de981002746b265"}
Sep 19 01:32:15.485272 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:15.485250    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c4ce82-qnwbw"
Sep 19 01:32:15.596110 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:15.596074    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e08e5e7-4fb9-4431-bf87-f03015627c3a-maas-api-service-ca\") pod \"5e08e5e7-4fb9-4431-bf87-f03015627c3a\" (UID: \"5e08e5e7-4fb9-4431-bf87-f03015627c3a\") "
Sep 19 01:32:15.596400 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:15.596376    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e08e5e7-4fb9-4431-bf87-f03015627c3a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5e08e5e7-4fb9-4431-bf87-f03015627c3a" (UID: "5e08e5e7-4fb9-4431-bf87-f03015627c3a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 01:32:15.697088 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:15.697012    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e08e5e7-4fb9-4431-bf87-f03015627c3a-maas-api-service-ca\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:32:16.315877 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:16.315826    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-8e708f/distinct-w0-8e708f-kserve-6f58868fbf-mfmbr"
Sep 19 01:32:16.316027 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:16.315914    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-8e708f/distinct-w0-8e708f-kserve-6f58868fbf-mfmbr"
Sep 19 01:32:16.328510 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:16.328485    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-8e708f/distinct-w0-8e708f-kserve-6f58868fbf-mfmbr"
Sep 19 01:32:16.362224 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:16.362199    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c4ce82-qnwbw"
Sep 19 01:32:16.362390 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:16.362243    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c4ce82-qnwbw" event={"ID":"5e08e5e7-4fb9-4431-bf87-f03015627c3a","Type":"ContainerDied","Data":"373636f3a293dbb22dda5bc35f11a29c7c811a76af53a3b647bb62801bc06e26"}
Sep 19 01:32:16.362390 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:16.362284    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="373636f3a293dbb22dda5bc35f11a29c7c811a76af53a3b647bb62801bc06e26"
Sep 19 01:32:16.374096 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:16.374075    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-8e708f/distinct-w0-8e708f-kserve-6f58868fbf-mfmbr"
Sep 19 01:32:23.168781 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:23.168746    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c4ce82-29829690-hcmkb"]
Sep 19 01:32:23.173212 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:23.173189    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c4ce82-29829690-hcmkb"]
Sep 19 01:32:23.272918 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:23.272886    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7d53214-d510-4026-ba65-37ea04bece7d" path="/var/lib/kubelet/pods/a7d53214-d510-4026-ba65-37ea04bece7d/volumes"
Sep 19 01:32:23.512366 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:23.512276    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c4ce82-qnwbw"]
Sep 19 01:32:23.513817 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:23.513797    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c4ce82-qnwbw"]
Sep 19 01:32:24.292099 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:24.292057    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-c4ce82/test-model-c4ce82-kserve-7b9cfc5c47-tqrkm" podUID="b0d58b7b-f528-444c-96fa-e6bea067bb5b" containerName="main" containerID="cri-o://2972f21973002caa3c8221a4f619b3b3b96b661e951e3583aec397a6e7a5eaf7" gracePeriod=45
Sep 19 01:32:24.540601 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:24.540582    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c4ce82/test-model-c4ce82-kserve-7b9cfc5c47-tqrkm"
Sep 19 01:32:24.674029 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:24.674002    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0d58b7b-f528-444c-96fa-e6bea067bb5b-dshm\") pod \"b0d58b7b-f528-444c-96fa-e6bea067bb5b\" (UID: \"b0d58b7b-f528-444c-96fa-e6bea067bb5b\") "
Sep 19 01:32:24.674029 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:24.674033    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0d58b7b-f528-444c-96fa-e6bea067bb5b-tmp-dir\") pod \"b0d58b7b-f528-444c-96fa-e6bea067bb5b\" (UID: \"b0d58b7b-f528-444c-96fa-e6bea067bb5b\") "
Sep 19 01:32:24.674231 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:24.674063    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0d58b7b-f528-444c-96fa-e6bea067bb5b-model-cache\") pod \"b0d58b7b-f528-444c-96fa-e6bea067bb5b\" (UID: \"b0d58b7b-f528-444c-96fa-e6bea067bb5b\") "
Sep 19 01:32:24.674231 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:24.674115    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0d58b7b-f528-444c-96fa-e6bea067bb5b-home\") pod \"b0d58b7b-f528-444c-96fa-e6bea067bb5b\" (UID: \"b0d58b7b-f528-444c-96fa-e6bea067bb5b\") "
Sep 19 01:32:24.674231 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:24.674153    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7w9f2\" (UniqueName: \"kubernetes.io/projected/b0d58b7b-f528-444c-96fa-e6bea067bb5b-kube-api-access-7w9f2\") pod \"b0d58b7b-f528-444c-96fa-e6bea067bb5b\" (UID: \"b0d58b7b-f528-444c-96fa-e6bea067bb5b\") "
Sep 19 01:32:24.674231 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:24.674183    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0d58b7b-f528-444c-96fa-e6bea067bb5b-tls-certs\") pod \"b0d58b7b-f528-444c-96fa-e6bea067bb5b\" (UID: \"b0d58b7b-f528-444c-96fa-e6bea067bb5b\") "
Sep 19 01:32:24.674423 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:24.674329    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0d58b7b-f528-444c-96fa-e6bea067bb5b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b0d58b7b-f528-444c-96fa-e6bea067bb5b" (UID: "b0d58b7b-f528-444c-96fa-e6bea067bb5b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 01:32:24.674423 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:24.674367    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0d58b7b-f528-444c-96fa-e6bea067bb5b-home" (OuterVolumeSpecName: "home") pod "b0d58b7b-f528-444c-96fa-e6bea067bb5b" (UID: "b0d58b7b-f528-444c-96fa-e6bea067bb5b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 01:32:24.674423 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:24.674381    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0d58b7b-f528-444c-96fa-e6bea067bb5b-model-cache" (OuterVolumeSpecName: "model-cache") pod "b0d58b7b-f528-444c-96fa-e6bea067bb5b" (UID: "b0d58b7b-f528-444c-96fa-e6bea067bb5b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 01:32:24.674589 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:24.674545    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0d58b7b-f528-444c-96fa-e6bea067bb5b-tmp-dir\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:32:24.674589 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:24.674560    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0d58b7b-f528-444c-96fa-e6bea067bb5b-model-cache\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:32:24.674589 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:24.674569    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0d58b7b-f528-444c-96fa-e6bea067bb5b-home\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:32:24.676421 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:24.676399    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0d58b7b-f528-444c-96fa-e6bea067bb5b-dshm" (OuterVolumeSpecName: "dshm") pod "b0d58b7b-f528-444c-96fa-e6bea067bb5b" (UID: "b0d58b7b-f528-444c-96fa-e6bea067bb5b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 01:32:24.676512 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:24.676480    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0d58b7b-f528-444c-96fa-e6bea067bb5b-kube-api-access-7w9f2" (OuterVolumeSpecName: "kube-api-access-7w9f2") pod "b0d58b7b-f528-444c-96fa-e6bea067bb5b" (UID: "b0d58b7b-f528-444c-96fa-e6bea067bb5b"). InnerVolumeSpecName "kube-api-access-7w9f2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 01:32:24.676577 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:24.676541    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0d58b7b-f528-444c-96fa-e6bea067bb5b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b0d58b7b-f528-444c-96fa-e6bea067bb5b" (UID: "b0d58b7b-f528-444c-96fa-e6bea067bb5b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 01:32:24.775085 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:24.775059    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7w9f2\" (UniqueName: \"kubernetes.io/projected/b0d58b7b-f528-444c-96fa-e6bea067bb5b-kube-api-access-7w9f2\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:32:24.775085 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:24.775081    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0d58b7b-f528-444c-96fa-e6bea067bb5b-tls-certs\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:32:24.775085 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:24.775090    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0d58b7b-f528-444c-96fa-e6bea067bb5b-dshm\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:32:25.273288 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:25.273248    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e08e5e7-4fb9-4431-bf87-f03015627c3a" path="/var/lib/kubelet/pods/5e08e5e7-4fb9-4431-bf87-f03015627c3a/volumes"
Sep 19 01:32:25.408281 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:25.408253    2581 generic.go:356] "Generic (PLEG): container finished" podID="b0d58b7b-f528-444c-96fa-e6bea067bb5b" containerID="2972f21973002caa3c8221a4f619b3b3b96b661e951e3583aec397a6e7a5eaf7" exitCode=0
Sep 19 01:32:25.408612 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:25.408290    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c4ce82/test-model-c4ce82-kserve-7b9cfc5c47-tqrkm" event={"ID":"b0d58b7b-f528-444c-96fa-e6bea067bb5b","Type":"ContainerDied","Data":"2972f21973002caa3c8221a4f619b3b3b96b661e951e3583aec397a6e7a5eaf7"}
Sep 19 01:32:25.408612 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:25.408300    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c4ce82/test-model-c4ce82-kserve-7b9cfc5c47-tqrkm"
Sep 19 01:32:25.408612 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:25.408317    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c4ce82/test-model-c4ce82-kserve-7b9cfc5c47-tqrkm" event={"ID":"b0d58b7b-f528-444c-96fa-e6bea067bb5b","Type":"ContainerDied","Data":"8ed1900f0c0c622d4ade74305fb9bb4213aa9950d5778034baf3236ef042ffe8"}
Sep 19 01:32:25.408612 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:25.408341    2581 scope.go:117] "RemoveContainer" containerID="2972f21973002caa3c8221a4f619b3b3b96b661e951e3583aec397a6e7a5eaf7"
Sep 19 01:32:25.417104 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:25.417084    2581 scope.go:117] "RemoveContainer" containerID="2972f21973002caa3c8221a4f619b3b3b96b661e951e3583aec397a6e7a5eaf7"
Sep 19 01:32:25.417394 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:32:25.417371    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2972f21973002caa3c8221a4f619b3b3b96b661e951e3583aec397a6e7a5eaf7\": container with ID starting with 2972f21973002caa3c8221a4f619b3b3b96b661e951e3583aec397a6e7a5eaf7 not found: ID does not exist" containerID="2972f21973002caa3c8221a4f619b3b3b96b661e951e3583aec397a6e7a5eaf7"
Sep 19 01:32:25.417447 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:25.417401    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2972f21973002caa3c8221a4f619b3b3b96b661e951e3583aec397a6e7a5eaf7"} err="failed to get container status \"2972f21973002caa3c8221a4f619b3b3b96b661e951e3583aec397a6e7a5eaf7\": rpc error: code = NotFound desc = could not find container \"2972f21973002caa3c8221a4f619b3b3b96b661e951e3583aec397a6e7a5eaf7\": container with ID starting with 2972f21973002caa3c8221a4f619b3b3b96b661e951e3583aec397a6e7a5eaf7 not found: ID does not exist"
Sep 19 01:32:25.427588 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:25.427568    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-c4ce82/test-model-c4ce82-kserve-7b9cfc5c47-tqrkm"]
Sep 19 01:32:25.431658 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:25.431630    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-c4ce82/test-model-c4ce82-kserve-7b9cfc5c47-tqrkm"]
Sep 19 01:32:26.326373 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:26.326344    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-b5fe4519-openshift-default-84cfc86584-vxfnt"]
Sep 19 01:32:26.326769 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:26.326752    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0d58b7b-f528-444c-96fa-e6bea067bb5b" containerName="main"
Sep 19 01:32:26.326814 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:26.326771    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0d58b7b-f528-444c-96fa-e6bea067bb5b" containerName="main"
Sep 19 01:32:26.326814 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:26.326792    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e08e5e7-4fb9-4431-bf87-f03015627c3a" containerName="revoke-keys"
Sep 19 01:32:26.326814 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:26.326798    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e08e5e7-4fb9-4431-bf87-f03015627c3a" containerName="revoke-keys"
Sep 19 01:32:26.326927 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:26.326848    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0d58b7b-f528-444c-96fa-e6bea067bb5b" containerName="main"
Sep 19 01:32:26.326927 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:26.326877    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e08e5e7-4fb9-4431-bf87-f03015627c3a" containerName="revoke-keys"
Sep 19 01:32:26.331045 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:26.331029    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b5fe4519-openshift-default-84cfc86584-vxfnt"
Sep 19 01:32:26.335368 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:26.335350    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-b5fe4519-openshift-default-dockercfg-zxpbg\""
Sep 19 01:32:26.363097 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:26.363072    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b5fe4519-openshift-default-84cfc86584-vxfnt"]
Sep 19 01:32:26.489521 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:26.489491    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62b1c519-4609-4138-a75c-7bcc23dcbf10-istio-envoy\") pod \"e2e-mt-b5fe4519-openshift-default-84cfc86584-vxfnt\" (UID: \"62b1c519-4609-4138-a75c-7bcc23dcbf10\") " pod="openshift-ingress/e2e-mt-b5fe4519-openshift-default-84cfc86584-vxfnt"
Sep 19 01:32:26.489521 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:26.489522    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62b1c519-4609-4138-a75c-7bcc23dcbf10-istiod-ca-cert\") pod \"e2e-mt-b5fe4519-openshift-default-84cfc86584-vxfnt\" (UID: \"62b1c519-4609-4138-a75c-7bcc23dcbf10\") " pod="openshift-ingress/e2e-mt-b5fe4519-openshift-default-84cfc86584-vxfnt"
Sep 19 01:32:26.489926 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:26.489543    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62b1c519-4609-4138-a75c-7bcc23dcbf10-workload-certs\") pod \"e2e-mt-b5fe4519-openshift-default-84cfc86584-vxfnt\" (UID: \"62b1c519-4609-4138-a75c-7bcc23dcbf10\") " pod="openshift-ingress/e2e-mt-b5fe4519-openshift-default-84cfc86584-vxfnt"
Sep 19 01:32:26.489926 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:26.489565    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62b1c519-4609-4138-a75c-7bcc23dcbf10-istio-data\") pod \"e2e-mt-b5fe4519-openshift-default-84cfc86584-vxfnt\" (UID: \"62b1c519-4609-4138-a75c-7bcc23dcbf10\") " pod="openshift-ingress/e2e-mt-b5fe4519-openshift-default-84cfc86584-vxfnt"
Sep 19 01:32:26.489926 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:26.489624    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpkkb\" (UniqueName: \"kubernetes.io/projected/62b1c519-4609-4138-a75c-7bcc23dcbf10-kube-api-access-kpkkb\") pod \"e2e-mt-b5fe4519-openshift-default-84cfc86584-vxfnt\" (UID: \"62b1c519-4609-4138-a75c-7bcc23dcbf10\") " pod="openshift-ingress/e2e-mt-b5fe4519-openshift-default-84cfc86584-vxfnt"
Sep 19 01:32:26.489926 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:26.489661    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62b1c519-4609-4138-a75c-7bcc23dcbf10-credential-socket\") pod \"e2e-mt-b5fe4519-openshift-default-84cfc86584-vxfnt\" (UID: \"62b1c519-4609-4138-a75c-7bcc23dcbf10\") " pod="openshift-ingress/e2e-mt-b5fe4519-openshift-default-84cfc86584-vxfnt"
Sep 19 01:32:26.489926 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:26.489679    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62b1c519-4609-4138-a75c-7bcc23dcbf10-istio-podinfo\") pod \"e2e-mt-b5fe4519-openshift-default-84cfc86584-vxfnt\" (UID: \"62b1c519-4609-4138-a75c-7bcc23dcbf10\") " pod="openshift-ingress/e2e-mt-b5fe4519-openshift-default-84cfc86584-vxfnt"
Sep 19 01:32:26.489926 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:26.489696    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62b1c519-4609-4138-a75c-7bcc23dcbf10-istio-token\") pod \"e2e-mt-b5fe4519-openshift-default-84cfc86584-vxfnt\" (UID: \"62b1c519-4609-4138-a75c-7bcc23dcbf10\") " pod="openshift-ingress/e2e-mt-b5fe4519-openshift-default-84cfc86584-vxfnt"
Sep 19 01:32:26.489926 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:26.489771    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62b1c519-4609-4138-a75c-7bcc23dcbf10-workload-socket\") pod \"e2e-mt-b5fe4519-openshift-default-84cfc86584-vxfnt\" (UID: \"62b1c519-4609-4138-a75c-7bcc23dcbf10\") " pod="openshift-ingress/e2e-mt-b5fe4519-openshift-default-84cfc86584-vxfnt"
Sep 19 01:32:26.590988 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:26.590925    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62b1c519-4609-4138-a75c-7bcc23dcbf10-credential-socket\") pod \"e2e-mt-b5fe4519-openshift-default-84cfc86584-vxfnt\" (UID: \"62b1c519-4609-4138-a75c-7bcc23dcbf10\") " pod="openshift-ingress/e2e-mt-b5fe4519-openshift-default-84cfc86584-vxfnt"
Sep 19 01:32:26.590988 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:26.590954    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62b1c519-4609-4138-a75c-7bcc23dcbf10-istio-podinfo\") pod \"e2e-mt-b5fe4519-openshift-default-84cfc86584-vxfnt\" (UID: \"62b1c519-4609-4138-a75c-7bcc23dcbf10\") " pod="openshift-ingress/e2e-mt-b5fe4519-openshift-default-84cfc86584-vxfnt"
Sep 19 01:32:26.590988 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:26.590972    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62b1c519-4609-4138-a75c-7bcc23dcbf10-istio-token\") pod \"e2e-mt-b5fe4519-openshift-default-84cfc86584-vxfnt\" (UID: \"62b1c519-4609-4138-a75c-7bcc23dcbf10\") " pod="openshift-ingress/e2e-mt-b5fe4519-openshift-default-84cfc86584-vxfnt"
Sep 19 01:32:26.591201 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:26.591037    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62b1c519-4609-4138-a75c-7bcc23dcbf10-workload-socket\") pod \"e2e-mt-b5fe4519-openshift-default-84cfc86584-vxfnt\" (UID: \"62b1c519-4609-4138-a75c-7bcc23dcbf10\") " pod="openshift-ingress/e2e-mt-b5fe4519-openshift-default-84cfc86584-vxfnt"
Sep 19 01:32:26.591201 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:26.591067    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62b1c519-4609-4138-a75c-7bcc23dcbf10-istio-envoy\") pod \"e2e-mt-b5fe4519-openshift-default-84cfc86584-vxfnt\" (UID: \"62b1c519-4609-4138-a75c-7bcc23dcbf10\") " pod="openshift-ingress/e2e-mt-b5fe4519-openshift-default-84cfc86584-vxfnt"
Sep 19 01:32:26.591201 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:26.591090    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62b1c519-4609-4138-a75c-7bcc23dcbf10-istiod-ca-cert\") pod \"e2e-mt-b5fe4519-openshift-default-84cfc86584-vxfnt\" (UID: \"62b1c519-4609-4138-a75c-7bcc23dcbf10\") " pod="openshift-ingress/e2e-mt-b5fe4519-openshift-default-84cfc86584-vxfnt"
Sep 19 01:32:26.591201 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:26.591118    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62b1c519-4609-4138-a75c-7bcc23dcbf10-workload-certs\") pod \"e2e-mt-b5fe4519-openshift-default-84cfc86584-vxfnt\" (UID: \"62b1c519-4609-4138-a75c-7bcc23dcbf10\") " pod="openshift-ingress/e2e-mt-b5fe4519-openshift-default-84cfc86584-vxfnt"
Sep 19 01:32:26.591201 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:26.591153    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62b1c519-4609-4138-a75c-7bcc23dcbf10-istio-data\") pod \"e2e-mt-b5fe4519-openshift-default-84cfc86584-vxfnt\" (UID: \"62b1c519-4609-4138-a75c-7bcc23dcbf10\") " pod="openshift-ingress/e2e-mt-b5fe4519-openshift-default-84cfc86584-vxfnt"
Sep 19 01:32:26.591201 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:26.591194    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpkkb\" (UniqueName: \"kubernetes.io/projected/62b1c519-4609-4138-a75c-7bcc23dcbf10-kube-api-access-kpkkb\") pod \"e2e-mt-b5fe4519-openshift-default-84cfc86584-vxfnt\" (UID: \"62b1c519-4609-4138-a75c-7bcc23dcbf10\") " pod="openshift-ingress/e2e-mt-b5fe4519-openshift-default-84cfc86584-vxfnt"
Sep 19 01:32:26.591535 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:26.591383    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62b1c519-4609-4138-a75c-7bcc23dcbf10-workload-socket\") pod \"e2e-mt-b5fe4519-openshift-default-84cfc86584-vxfnt\" (UID: \"62b1c519-4609-4138-a75c-7bcc23dcbf10\") " pod="openshift-ingress/e2e-mt-b5fe4519-openshift-default-84cfc86584-vxfnt"
Sep 19 01:32:26.591535 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:26.591394    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62b1c519-4609-4138-a75c-7bcc23dcbf10-credential-socket\") pod \"e2e-mt-b5fe4519-openshift-default-84cfc86584-vxfnt\" (UID: \"62b1c519-4609-4138-a75c-7bcc23dcbf10\") " pod="openshift-ingress/e2e-mt-b5fe4519-openshift-default-84cfc86584-vxfnt"
Sep 19 01:32:26.591680 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:26.591651    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62b1c519-4609-4138-a75c-7bcc23dcbf10-istio-data\") pod \"e2e-mt-b5fe4519-openshift-default-84cfc86584-vxfnt\" (UID: \"62b1c519-4609-4138-a75c-7bcc23dcbf10\") " pod="openshift-ingress/e2e-mt-b5fe4519-openshift-default-84cfc86584-vxfnt"
Sep 19 01:32:26.591745 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:26.591675    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62b1c519-4609-4138-a75c-7bcc23dcbf10-workload-certs\") pod \"e2e-mt-b5fe4519-openshift-default-84cfc86584-vxfnt\" (UID: \"62b1c519-4609-4138-a75c-7bcc23dcbf10\") " pod="openshift-ingress/e2e-mt-b5fe4519-openshift-default-84cfc86584-vxfnt"
Sep 19 01:32:26.591887 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:26.591815    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62b1c519-4609-4138-a75c-7bcc23dcbf10-istiod-ca-cert\") pod \"e2e-mt-b5fe4519-openshift-default-84cfc86584-vxfnt\" (UID: \"62b1c519-4609-4138-a75c-7bcc23dcbf10\") " pod="openshift-ingress/e2e-mt-b5fe4519-openshift-default-84cfc86584-vxfnt"
Sep 19 01:32:26.593494 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:26.593466    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62b1c519-4609-4138-a75c-7bcc23dcbf10-istio-envoy\") pod \"e2e-mt-b5fe4519-openshift-default-84cfc86584-vxfnt\" (UID: \"62b1c519-4609-4138-a75c-7bcc23dcbf10\") " pod="openshift-ingress/e2e-mt-b5fe4519-openshift-default-84cfc86584-vxfnt"
Sep 19 01:32:26.593584 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:26.593537    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62b1c519-4609-4138-a75c-7bcc23dcbf10-istio-podinfo\") pod \"e2e-mt-b5fe4519-openshift-default-84cfc86584-vxfnt\" (UID: \"62b1c519-4609-4138-a75c-7bcc23dcbf10\") " pod="openshift-ingress/e2e-mt-b5fe4519-openshift-default-84cfc86584-vxfnt"
Sep 19 01:32:26.606034 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:26.606011    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62b1c519-4609-4138-a75c-7bcc23dcbf10-istio-token\") pod \"e2e-mt-b5fe4519-openshift-default-84cfc86584-vxfnt\" (UID: \"62b1c519-4609-4138-a75c-7bcc23dcbf10\") " pod="openshift-ingress/e2e-mt-b5fe4519-openshift-default-84cfc86584-vxfnt"
Sep 19 01:32:26.606176 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:26.606158    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpkkb\" (UniqueName: \"kubernetes.io/projected/62b1c519-4609-4138-a75c-7bcc23dcbf10-kube-api-access-kpkkb\") pod \"e2e-mt-b5fe4519-openshift-default-84cfc86584-vxfnt\" (UID: \"62b1c519-4609-4138-a75c-7bcc23dcbf10\") " pod="openshift-ingress/e2e-mt-b5fe4519-openshift-default-84cfc86584-vxfnt"
Sep 19 01:32:26.640233 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:26.640211    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b5fe4519-openshift-default-84cfc86584-vxfnt"
Sep 19 01:32:26.947935 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:26.947903    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b5fe4519-openshift-default-84cfc86584-vxfnt"]
Sep 19 01:32:26.949024 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:32:26.948994    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62b1c519_4609_4138_a75c_7bcc23dcbf10.slice/crio-5490179ec2ae77798af857c4f80403bd4a77b8eb0c297424281de9cf88ccf1eb WatchSource:0}: Error finding container 5490179ec2ae77798af857c4f80403bd4a77b8eb0c297424281de9cf88ccf1eb: Status 404 returned error can't find the container with id 5490179ec2ae77798af857c4f80403bd4a77b8eb0c297424281de9cf88ccf1eb
Sep 19 01:32:26.951251 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:26.951224    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 19 01:32:26.951307 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:26.951288    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 19 01:32:26.951355 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:26.951317    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 19 01:32:27.089101 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:27.089055    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-5a005d/test-model-5a005d-kserve-844f5dc4c9-f72bf" podUID="6f17cb1a-739d-4613-998a-ad43d62e6367" containerName="main" containerID="cri-o://ae580c46d316d56f7b9477bb41105f7e0efdf0b67e1ff9b463b177c03b49bfae" gracePeriod=45
Sep 19 01:32:27.273887 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:27.273838    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0d58b7b-f528-444c-96fa-e6bea067bb5b" path="/var/lib/kubelet/pods/b0d58b7b-f528-444c-96fa-e6bea067bb5b/volumes"
Sep 19 01:32:27.330298 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:27.330273    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5a005d/test-model-5a005d-kserve-844f5dc4c9-f72bf"
Sep 19 01:32:27.397711 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:27.397688    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f17cb1a-739d-4613-998a-ad43d62e6367-model-cache\") pod \"6f17cb1a-739d-4613-998a-ad43d62e6367\" (UID: \"6f17cb1a-739d-4613-998a-ad43d62e6367\") "
Sep 19 01:32:27.397848 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:27.397739    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f17cb1a-739d-4613-998a-ad43d62e6367-tmp-dir\") pod \"6f17cb1a-739d-4613-998a-ad43d62e6367\" (UID: \"6f17cb1a-739d-4613-998a-ad43d62e6367\") "
Sep 19 01:32:27.397848 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:27.397759    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f17cb1a-739d-4613-998a-ad43d62e6367-tls-certs\") pod \"6f17cb1a-739d-4613-998a-ad43d62e6367\" (UID: \"6f17cb1a-739d-4613-998a-ad43d62e6367\") "
Sep 19 01:32:27.397848 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:27.397805    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f17cb1a-739d-4613-998a-ad43d62e6367-dshm\") pod \"6f17cb1a-739d-4613-998a-ad43d62e6367\" (UID: \"6f17cb1a-739d-4613-998a-ad43d62e6367\") "
Sep 19 01:32:27.397848 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:27.397827    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f17cb1a-739d-4613-998a-ad43d62e6367-home\") pod \"6f17cb1a-739d-4613-998a-ad43d62e6367\" (UID: \"6f17cb1a-739d-4613-998a-ad43d62e6367\") "
Sep 19 01:32:27.398085 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:27.397897    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-29rsb\" (UniqueName: \"kubernetes.io/projected/6f17cb1a-739d-4613-998a-ad43d62e6367-kube-api-access-29rsb\") pod \"6f17cb1a-739d-4613-998a-ad43d62e6367\" (UID: \"6f17cb1a-739d-4613-998a-ad43d62e6367\") "
Sep 19 01:32:27.398085 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:27.398006    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f17cb1a-739d-4613-998a-ad43d62e6367-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6f17cb1a-739d-4613-998a-ad43d62e6367" (UID: "6f17cb1a-739d-4613-998a-ad43d62e6367"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 01:32:27.398085 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:27.398019    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f17cb1a-739d-4613-998a-ad43d62e6367-model-cache" (OuterVolumeSpecName: "model-cache") pod "6f17cb1a-739d-4613-998a-ad43d62e6367" (UID: "6f17cb1a-739d-4613-998a-ad43d62e6367"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 01:32:27.398237 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:27.398175    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f17cb1a-739d-4613-998a-ad43d62e6367-home" (OuterVolumeSpecName: "home") pod "6f17cb1a-739d-4613-998a-ad43d62e6367" (UID: "6f17cb1a-739d-4613-998a-ad43d62e6367"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 01:32:27.398237 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:27.398198    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f17cb1a-739d-4613-998a-ad43d62e6367-model-cache\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:32:27.398237 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:27.398213    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f17cb1a-739d-4613-998a-ad43d62e6367-tmp-dir\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:32:27.400126 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:27.400102    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f17cb1a-739d-4613-998a-ad43d62e6367-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6f17cb1a-739d-4613-998a-ad43d62e6367" (UID: "6f17cb1a-739d-4613-998a-ad43d62e6367"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 01:32:27.400608 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:27.400587    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f17cb1a-739d-4613-998a-ad43d62e6367-dshm" (OuterVolumeSpecName: "dshm") pod "6f17cb1a-739d-4613-998a-ad43d62e6367" (UID: "6f17cb1a-739d-4613-998a-ad43d62e6367"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 01:32:27.400660 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:27.400603    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f17cb1a-739d-4613-998a-ad43d62e6367-kube-api-access-29rsb" (OuterVolumeSpecName: "kube-api-access-29rsb") pod "6f17cb1a-739d-4613-998a-ad43d62e6367" (UID: "6f17cb1a-739d-4613-998a-ad43d62e6367"). InnerVolumeSpecName "kube-api-access-29rsb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 01:32:27.421219 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:27.421194    2581 generic.go:356] "Generic (PLEG): container finished" podID="6f17cb1a-739d-4613-998a-ad43d62e6367" containerID="ae580c46d316d56f7b9477bb41105f7e0efdf0b67e1ff9b463b177c03b49bfae" exitCode=0
Sep 19 01:32:27.421325 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:27.421236    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5a005d/test-model-5a005d-kserve-844f5dc4c9-f72bf"
Sep 19 01:32:27.421325 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:27.421279    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5a005d/test-model-5a005d-kserve-844f5dc4c9-f72bf" event={"ID":"6f17cb1a-739d-4613-998a-ad43d62e6367","Type":"ContainerDied","Data":"ae580c46d316d56f7b9477bb41105f7e0efdf0b67e1ff9b463b177c03b49bfae"}
Sep 19 01:32:27.421325 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:27.421318    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5a005d/test-model-5a005d-kserve-844f5dc4c9-f72bf" event={"ID":"6f17cb1a-739d-4613-998a-ad43d62e6367","Type":"ContainerDied","Data":"27a15624204f48897f3f519567e5d6a51c04742a2be0c6db48daea89eacea3f8"}
Sep 19 01:32:27.421439 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:27.421338    2581 scope.go:117] "RemoveContainer" containerID="ae580c46d316d56f7b9477bb41105f7e0efdf0b67e1ff9b463b177c03b49bfae"
Sep 19 01:32:27.423051 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:27.423030    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b5fe4519-openshift-default-84cfc86584-vxfnt" event={"ID":"62b1c519-4609-4138-a75c-7bcc23dcbf10","Type":"ContainerStarted","Data":"df1658114676d73ee724993ad81890e6cc11b4550ff0bd12d3b61329b8229945"}
Sep 19 01:32:27.423051 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:27.423067    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b5fe4519-openshift-default-84cfc86584-vxfnt" event={"ID":"62b1c519-4609-4138-a75c-7bcc23dcbf10","Type":"ContainerStarted","Data":"5490179ec2ae77798af857c4f80403bd4a77b8eb0c297424281de9cf88ccf1eb"}
Sep 19 01:32:27.429728 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:27.429710    2581 scope.go:117] "RemoveContainer" containerID="ae580c46d316d56f7b9477bb41105f7e0efdf0b67e1ff9b463b177c03b49bfae"
Sep 19 01:32:27.430003 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:32:27.429987    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae580c46d316d56f7b9477bb41105f7e0efdf0b67e1ff9b463b177c03b49bfae\": container with ID starting with ae580c46d316d56f7b9477bb41105f7e0efdf0b67e1ff9b463b177c03b49bfae not found: ID does not exist" containerID="ae580c46d316d56f7b9477bb41105f7e0efdf0b67e1ff9b463b177c03b49bfae"
Sep 19 01:32:27.430060 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:27.430011    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae580c46d316d56f7b9477bb41105f7e0efdf0b67e1ff9b463b177c03b49bfae"} err="failed to get container status \"ae580c46d316d56f7b9477bb41105f7e0efdf0b67e1ff9b463b177c03b49bfae\": rpc error: code = NotFound desc = could not find container \"ae580c46d316d56f7b9477bb41105f7e0efdf0b67e1ff9b463b177c03b49bfae\": container with ID starting with ae580c46d316d56f7b9477bb41105f7e0efdf0b67e1ff9b463b177c03b49bfae not found: ID does not exist"
Sep 19 01:32:27.473409 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:27.473332    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-b5fe4519-openshift-default-84cfc86584-vxfnt" podStartSLOduration=1.473316061 podStartE2EDuration="1.473316061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 01:32:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 01:32:27.471918228 +0000 UTC m=+1198.359587320" watchObservedRunningTime="2026-09-19 01:32:27.473316061 +0000 UTC m=+1198.360985142"
Sep 19 01:32:27.493275 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:27.493253    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-5a005d/test-model-5a005d-kserve-844f5dc4c9-f72bf"]
Sep 19 01:32:27.498967 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:27.498940    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f17cb1a-739d-4613-998a-ad43d62e6367-tls-certs\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:32:27.498967 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:27.498964    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f17cb1a-739d-4613-998a-ad43d62e6367-dshm\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:32:27.499089 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:27.498978    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f17cb1a-739d-4613-998a-ad43d62e6367-home\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:32:27.499089 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:27.498990    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29rsb\" (UniqueName: \"kubernetes.io/projected/6f17cb1a-739d-4613-998a-ad43d62e6367-kube-api-access-29rsb\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:32:27.502306 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:27.502286    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-5a005d/test-model-5a005d-kserve-844f5dc4c9-f72bf"]
Sep 19 01:32:27.640333 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:27.640306    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-b5fe4519-openshift-default-84cfc86584-vxfnt"
Sep 19 01:32:28.645786 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:28.645750    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-b5fe4519-openshift-default-84cfc86584-vxfnt"
Sep 19 01:32:29.275187 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:29.275148    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f17cb1a-739d-4613-998a-ad43d62e6367" path="/var/lib/kubelet/pods/6f17cb1a-739d-4613-998a-ad43d62e6367/volumes"
Sep 19 01:32:29.434164 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:29.434138    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b5fe4519-openshift-default-84cfc86584-vxfnt"
Sep 19 01:32:29.435039 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:29.435023    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-b5fe4519-openshift-default-84cfc86584-vxfnt"
Sep 19 01:32:31.234052 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:31.234026    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x785d_f759a12f-5b6f-4924-96f2-dd5e00fc1348/ovn-acl-logging/0.log"
Sep 19 01:32:31.234483 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:31.234136    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x785d_f759a12f-5b6f-4924-96f2-dd5e00fc1348/ovn-acl-logging/0.log"
Sep 19 01:32:31.308625 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:31.308600    2581 scope.go:117] "RemoveContainer" containerID="1cc3a09dc21a2928826cbfd30e5700db72143dd1d3290ba908d78cc4d560678d"
Sep 19 01:32:32.589612 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:32.589580    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c9nsz"]
Sep 19 01:32:32.702966 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:32.702938    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5a005d-xq6ch"]
Sep 19 01:32:32.703264 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:32.703252    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f17cb1a-739d-4613-998a-ad43d62e6367" containerName="main"
Sep 19 01:32:32.703307 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:32.703266    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f17cb1a-739d-4613-998a-ad43d62e6367" containerName="main"
Sep 19 01:32:32.703339 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:32.703321    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f17cb1a-739d-4613-998a-ad43d62e6367" containerName="main"
Sep 19 01:32:32.707891 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:32.707850    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5a005d-xq6ch"
Sep 19 01:32:32.709903 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:32.709879    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 19 01:32:32.710115 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:32.710102    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s7pvt\""
Sep 19 01:32:32.714439 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:32.714408    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5a005d-xq6ch"]
Sep 19 01:32:32.840566 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:32.840484    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16c05980-c8aa-4cc1-b5a4-181637c27cba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5a005d-xq6ch\" (UID: \"16c05980-c8aa-4cc1-b5a4-181637c27cba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5a005d-xq6ch"
Sep 19 01:32:32.941512 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:32.941481    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16c05980-c8aa-4cc1-b5a4-181637c27cba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5a005d-xq6ch\" (UID: \"16c05980-c8aa-4cc1-b5a4-181637c27cba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5a005d-xq6ch"
Sep 19 01:32:32.942063 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:32.942046    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16c05980-c8aa-4cc1-b5a4-181637c27cba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5a005d-xq6ch\" (UID: \"16c05980-c8aa-4cc1-b5a4-181637c27cba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5a005d-xq6ch"
Sep 19 01:32:33.019447 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:33.019425    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5a005d-xq6ch"
Sep 19 01:32:33.288670 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:33.288640    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5a005d-xq6ch"]
Sep 19 01:32:33.291119 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:32:33.291087    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16c05980_c8aa_4cc1_b5a4_181637c27cba.slice/crio-da6fc2f0df0eae33099be69ee3b21b148c2407b8f911b442c170e30480d86171 WatchSource:0}: Error finding container da6fc2f0df0eae33099be69ee3b21b148c2407b8f911b442c170e30480d86171: Status 404 returned error can't find the container with id da6fc2f0df0eae33099be69ee3b21b148c2407b8f911b442c170e30480d86171
Sep 19 01:32:33.455958 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:33.455925    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5a005d-xq6ch" event={"ID":"16c05980-c8aa-4cc1-b5a4-181637c27cba","Type":"ContainerStarted","Data":"da6fc2f0df0eae33099be69ee3b21b148c2407b8f911b442c170e30480d86171"}
Sep 19 01:32:34.462593 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:34.462562    2581 generic.go:356] "Generic (PLEG): container finished" podID="16c05980-c8aa-4cc1-b5a4-181637c27cba" containerID="d49496c10ac2e25a03114f39fe26a5e8fa1a3015d616c2baf1caaad732ea2797" exitCode=0
Sep 19 01:32:34.463045 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:34.462620    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5a005d-xq6ch" event={"ID":"16c05980-c8aa-4cc1-b5a4-181637c27cba","Type":"ContainerDied","Data":"d49496c10ac2e25a03114f39fe26a5e8fa1a3015d616c2baf1caaad732ea2797"}
Sep 19 01:32:34.751471 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:34.751382    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b5fe4519-b4d97c6f8-n7pdx"]
Sep 19 01:32:34.754654 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:34.754633    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b5fe4519-b4d97c6f8-n7pdx"
Sep 19 01:32:34.757109 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:34.757089    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-b5fe4519\""
Sep 19 01:32:34.768401 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:34.768374    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b5fe4519-b4d97c6f8-n7pdx"]
Sep 19 01:32:34.859890 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:34.859837    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/271c7dde-6e52-47ab-ba47-7c8aaac9c549-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-b5fe4519-b4d97c6f8-n7pdx\" (UID: \"271c7dde-6e52-47ab-ba47-7c8aaac9c549\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b5fe4519-b4d97c6f8-n7pdx"
Sep 19 01:32:34.860047 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:34.859896    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zltlg\" (UniqueName: \"kubernetes.io/projected/271c7dde-6e52-47ab-ba47-7c8aaac9c549-kube-api-access-zltlg\") pod \"payload-pre-processing-e2e-mt-b5fe4519-b4d97c6f8-n7pdx\" (UID: \"271c7dde-6e52-47ab-ba47-7c8aaac9c549\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b5fe4519-b4d97c6f8-n7pdx"
Sep 19 01:32:34.961289 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:34.961250    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/271c7dde-6e52-47ab-ba47-7c8aaac9c549-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-b5fe4519-b4d97c6f8-n7pdx\" (UID: \"271c7dde-6e52-47ab-ba47-7c8aaac9c549\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b5fe4519-b4d97c6f8-n7pdx"
Sep 19 01:32:34.961289 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:34.961289    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zltlg\" (UniqueName: \"kubernetes.io/projected/271c7dde-6e52-47ab-ba47-7c8aaac9c549-kube-api-access-zltlg\") pod \"payload-pre-processing-e2e-mt-b5fe4519-b4d97c6f8-n7pdx\" (UID: \"271c7dde-6e52-47ab-ba47-7c8aaac9c549\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b5fe4519-b4d97c6f8-n7pdx"
Sep 19 01:32:34.962241 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:34.962219    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/271c7dde-6e52-47ab-ba47-7c8aaac9c549-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-b5fe4519-b4d97c6f8-n7pdx\" (UID: \"271c7dde-6e52-47ab-ba47-7c8aaac9c549\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b5fe4519-b4d97c6f8-n7pdx"
Sep 19 01:32:34.970374 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:34.970352    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zltlg\" (UniqueName: \"kubernetes.io/projected/271c7dde-6e52-47ab-ba47-7c8aaac9c549-kube-api-access-zltlg\") pod \"payload-pre-processing-e2e-mt-b5fe4519-b4d97c6f8-n7pdx\" (UID: \"271c7dde-6e52-47ab-ba47-7c8aaac9c549\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b5fe4519-b4d97c6f8-n7pdx"
Sep 19 01:32:35.064646 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:35.064564    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b5fe4519-b4d97c6f8-n7pdx"
Sep 19 01:32:35.343258 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:35.343223    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b5fe4519-b4d97c6f8-n7pdx"]
Sep 19 01:32:35.345384 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:32:35.345355    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod271c7dde_6e52_47ab_ba47_7c8aaac9c549.slice/crio-7874432e5ddcc65d35dfb56221dc4a15fd92794d765fd94075bb3bf57d194b53 WatchSource:0}: Error finding container 7874432e5ddcc65d35dfb56221dc4a15fd92794d765fd94075bb3bf57d194b53: Status 404 returned error can't find the container with id 7874432e5ddcc65d35dfb56221dc4a15fd92794d765fd94075bb3bf57d194b53
Sep 19 01:32:35.469699 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:35.469663    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b5fe4519-b4d97c6f8-n7pdx" event={"ID":"271c7dde-6e52-47ab-ba47-7c8aaac9c549","Type":"ContainerStarted","Data":"563793a68b90da1b8fccd0c9cc081547e65437e6b2f0562ed9c68fba3bdc74c7"}
Sep 19 01:32:35.470120 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:35.469720    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b5fe4519-b4d97c6f8-n7pdx" event={"ID":"271c7dde-6e52-47ab-ba47-7c8aaac9c549","Type":"ContainerStarted","Data":"7874432e5ddcc65d35dfb56221dc4a15fd92794d765fd94075bb3bf57d194b53"}
Sep 19 01:32:35.470120 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:35.469927    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-b5fe4519-b4d97c6f8-n7pdx"
Sep 19 01:32:35.488676 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:35.488622    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-b5fe4519-b4d97c6f8-n7pdx" podStartSLOduration=1.488606042 podStartE2EDuration="1.488606042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 01:32:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 01:32:35.486831566 +0000 UTC m=+1206.374500677" watchObservedRunningTime="2026-09-19 01:32:35.488606042 +0000 UTC m=+1206.376275119"
Sep 19 01:32:35.590689 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:35.590650    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5a005d-xq6ch"
Sep 19 01:32:35.767031 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:35.766954    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16c05980-c8aa-4cc1-b5a4-181637c27cba-maas-api-service-ca\") pod \"16c05980-c8aa-4cc1-b5a4-181637c27cba\" (UID: \"16c05980-c8aa-4cc1-b5a4-181637c27cba\") "
Sep 19 01:32:35.767320 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:35.767299    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16c05980-c8aa-4cc1-b5a4-181637c27cba-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "16c05980-c8aa-4cc1-b5a4-181637c27cba" (UID: "16c05980-c8aa-4cc1-b5a4-181637c27cba"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 01:32:35.868062 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:35.868022    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16c05980-c8aa-4cc1-b5a4-181637c27cba-maas-api-service-ca\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:32:36.475082 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:36.475053    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5a005d-xq6ch"
Sep 19 01:32:36.475481 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:36.475094    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5a005d-xq6ch" event={"ID":"16c05980-c8aa-4cc1-b5a4-181637c27cba","Type":"ContainerDied","Data":"da6fc2f0df0eae33099be69ee3b21b148c2407b8f911b442c170e30480d86171"}
Sep 19 01:32:36.475481 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:36.475138    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da6fc2f0df0eae33099be69ee3b21b148c2407b8f911b442c170e30480d86171"
Sep 19 01:32:43.895275 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:43.895234    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5a005d-756b74fb85-bmnpx"]
Sep 19 01:32:43.895757 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:43.895567    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-5a005d-756b74fb85-bmnpx" podUID="b03a0fdb-7a5d-44a8-bf5c-88fcfd06b734" containerName="payload-processing" containerID="cri-o://b3cf9ed3612b47fa9996297fcd700e747a52656ca3fcbf70553da46af123ff43" gracePeriod=30
Sep 19 01:32:44.141208 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:44.141182    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5a005d-756b74fb85-bmnpx"
Sep 19 01:32:44.243411 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:44.242720    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hbcx9\" (UniqueName: \"kubernetes.io/projected/b03a0fdb-7a5d-44a8-bf5c-88fcfd06b734-kube-api-access-hbcx9\") pod \"b03a0fdb-7a5d-44a8-bf5c-88fcfd06b734\" (UID: \"b03a0fdb-7a5d-44a8-bf5c-88fcfd06b734\") "
Sep 19 01:32:44.243411 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:44.242775    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b03a0fdb-7a5d-44a8-bf5c-88fcfd06b734-plugins-config-volume\") pod \"b03a0fdb-7a5d-44a8-bf5c-88fcfd06b734\" (UID: \"b03a0fdb-7a5d-44a8-bf5c-88fcfd06b734\") "
Sep 19 01:32:44.244007 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:44.243424    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b03a0fdb-7a5d-44a8-bf5c-88fcfd06b734-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b03a0fdb-7a5d-44a8-bf5c-88fcfd06b734" (UID: "b03a0fdb-7a5d-44a8-bf5c-88fcfd06b734"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 01:32:44.245758 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:44.245732    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b03a0fdb-7a5d-44a8-bf5c-88fcfd06b734-kube-api-access-hbcx9" (OuterVolumeSpecName: "kube-api-access-hbcx9") pod "b03a0fdb-7a5d-44a8-bf5c-88fcfd06b734" (UID: "b03a0fdb-7a5d-44a8-bf5c-88fcfd06b734"). InnerVolumeSpecName "kube-api-access-hbcx9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 01:32:44.343808 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:44.343779    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbcx9\" (UniqueName: \"kubernetes.io/projected/b03a0fdb-7a5d-44a8-bf5c-88fcfd06b734-kube-api-access-hbcx9\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:32:44.343808 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:44.343802    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b03a0fdb-7a5d-44a8-bf5c-88fcfd06b734-plugins-config-volume\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:32:44.382476 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:44.382442    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5a005d-xq6ch"]
Sep 19 01:32:44.387037 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:44.387013    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5a005d-xq6ch"]
Sep 19 01:32:44.515064 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:44.515006    2581 generic.go:356] "Generic (PLEG): container finished" podID="b03a0fdb-7a5d-44a8-bf5c-88fcfd06b734" containerID="b3cf9ed3612b47fa9996297fcd700e747a52656ca3fcbf70553da46af123ff43" exitCode=0
Sep 19 01:32:44.515064 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:44.515045    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5a005d-756b74fb85-bmnpx"
Sep 19 01:32:44.515208 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:44.515092    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5a005d-756b74fb85-bmnpx" event={"ID":"b03a0fdb-7a5d-44a8-bf5c-88fcfd06b734","Type":"ContainerDied","Data":"b3cf9ed3612b47fa9996297fcd700e747a52656ca3fcbf70553da46af123ff43"}
Sep 19 01:32:44.515208 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:44.515137    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5a005d-756b74fb85-bmnpx" event={"ID":"b03a0fdb-7a5d-44a8-bf5c-88fcfd06b734","Type":"ContainerDied","Data":"68a2fa37a094c55d6edf0fd425d02dd160581951b115646ba6eaacffa557e0b1"}
Sep 19 01:32:44.515208 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:44.515159    2581 scope.go:117] "RemoveContainer" containerID="b3cf9ed3612b47fa9996297fcd700e747a52656ca3fcbf70553da46af123ff43"
Sep 19 01:32:44.524876 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:44.524840    2581 scope.go:117] "RemoveContainer" containerID="b3cf9ed3612b47fa9996297fcd700e747a52656ca3fcbf70553da46af123ff43"
Sep 19 01:32:44.525115 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:32:44.525096    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3cf9ed3612b47fa9996297fcd700e747a52656ca3fcbf70553da46af123ff43\": container with ID starting with b3cf9ed3612b47fa9996297fcd700e747a52656ca3fcbf70553da46af123ff43 not found: ID does not exist" containerID="b3cf9ed3612b47fa9996297fcd700e747a52656ca3fcbf70553da46af123ff43"
Sep 19 01:32:44.525176 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:44.525124    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3cf9ed3612b47fa9996297fcd700e747a52656ca3fcbf70553da46af123ff43"} err="failed to get container status \"b3cf9ed3612b47fa9996297fcd700e747a52656ca3fcbf70553da46af123ff43\": rpc error: code = NotFound desc = could not find container \"b3cf9ed3612b47fa9996297fcd700e747a52656ca3fcbf70553da46af123ff43\": container with ID starting with b3cf9ed3612b47fa9996297fcd700e747a52656ca3fcbf70553da46af123ff43 not found: ID does not exist"
Sep 19 01:32:44.545473 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:44.545452    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5a005d-756b74fb85-bmnpx"]
Sep 19 01:32:44.554042 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:44.554025    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5a005d-756b74fb85-bmnpx"]
Sep 19 01:32:45.272444 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:45.272405    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16c05980-c8aa-4cc1-b5a4-181637c27cba" path="/var/lib/kubelet/pods/16c05980-c8aa-4cc1-b5a4-181637c27cba/volumes"
Sep 19 01:32:45.272813 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:45.272761    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b03a0fdb-7a5d-44a8-bf5c-88fcfd06b734" path="/var/lib/kubelet/pods/b03a0fdb-7a5d-44a8-bf5c-88fcfd06b734/volumes"
Sep 19 01:32:46.478039 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:46.478006    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-b5fe4519-b4d97c6f8-n7pdx"
Sep 19 01:32:51.678582 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:51.678551    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-5a005d-openshift-default-cc9788f6b-twz5d"]
Sep 19 01:32:51.679037 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:51.678751    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-5a005d-openshift-default-cc9788f6b-twz5d" podUID="2302d85f-6dba-4f71-bc69-6c64172f0f93" containerName="istio-proxy" containerID="cri-o://02374b819abfffc5abb51b208e9cd18dc7d7ee6f29d926f9e3a10ff82eb0bb4b" gracePeriod=30
Sep 19 01:32:52.752476 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:52.752445    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-751f6a-openshift-default-549c5bdf4d-78c97"]
Sep 19 01:32:52.752850 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:52.752752    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b03a0fdb-7a5d-44a8-bf5c-88fcfd06b734" containerName="payload-processing"
Sep 19 01:32:52.752850 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:52.752764    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b03a0fdb-7a5d-44a8-bf5c-88fcfd06b734" containerName="payload-processing"
Sep 19 01:32:52.752850 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:52.752786    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16c05980-c8aa-4cc1-b5a4-181637c27cba" containerName="revoke-keys"
Sep 19 01:32:52.752850 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:52.752791    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="16c05980-c8aa-4cc1-b5a4-181637c27cba" containerName="revoke-keys"
Sep 19 01:32:52.752850 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:52.752842    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="16c05980-c8aa-4cc1-b5a4-181637c27cba" containerName="revoke-keys"
Sep 19 01:32:52.752850 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:52.752849    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b03a0fdb-7a5d-44a8-bf5c-88fcfd06b734" containerName="payload-processing"
Sep 19 01:32:52.754978 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:52.754955    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-751f6a-openshift-default-549c5bdf4d-78c97"
Sep 19 01:32:52.757606 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:52.757584    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-751f6a-openshift-default-dockercfg-ndb2v\""
Sep 19 01:32:52.820216 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:52.820185    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89c7c2d8-487c-4c80-a630-dc0ffc760475-credential-socket\") pod \"e2e-shared-a-w3-751f6a-openshift-default-549c5bdf4d-78c97\" (UID: \"89c7c2d8-487c-4c80-a630-dc0ffc760475\") " pod="openshift-ingress/e2e-shared-a-w3-751f6a-openshift-default-549c5bdf4d-78c97"
Sep 19 01:32:52.820216 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:52.820220    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89c7c2d8-487c-4c80-a630-dc0ffc760475-istio-envoy\") pod \"e2e-shared-a-w3-751f6a-openshift-default-549c5bdf4d-78c97\" (UID: \"89c7c2d8-487c-4c80-a630-dc0ffc760475\") " pod="openshift-ingress/e2e-shared-a-w3-751f6a-openshift-default-549c5bdf4d-78c97"
Sep 19 01:32:52.820459 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:52.820239    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89c7c2d8-487c-4c80-a630-dc0ffc760475-workload-socket\") pod \"e2e-shared-a-w3-751f6a-openshift-default-549c5bdf4d-78c97\" (UID: \"89c7c2d8-487c-4c80-a630-dc0ffc760475\") " pod="openshift-ingress/e2e-shared-a-w3-751f6a-openshift-default-549c5bdf4d-78c97"
Sep 19 01:32:52.820459 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:52.820294    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89c7c2d8-487c-4c80-a630-dc0ffc760475-istio-token\") pod \"e2e-shared-a-w3-751f6a-openshift-default-549c5bdf4d-78c97\" (UID: \"89c7c2d8-487c-4c80-a630-dc0ffc760475\") " pod="openshift-ingress/e2e-shared-a-w3-751f6a-openshift-default-549c5bdf4d-78c97"
Sep 19 01:32:52.820459 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:52.820363    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89c7c2d8-487c-4c80-a630-dc0ffc760475-istio-data\") pod \"e2e-shared-a-w3-751f6a-openshift-default-549c5bdf4d-78c97\" (UID: \"89c7c2d8-487c-4c80-a630-dc0ffc760475\") " pod="openshift-ingress/e2e-shared-a-w3-751f6a-openshift-default-549c5bdf4d-78c97"
Sep 19 01:32:52.820459 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:52.820402    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89c7c2d8-487c-4c80-a630-dc0ffc760475-istiod-ca-cert\") pod \"e2e-shared-a-w3-751f6a-openshift-default-549c5bdf4d-78c97\" (UID: \"89c7c2d8-487c-4c80-a630-dc0ffc760475\") " pod="openshift-ingress/e2e-shared-a-w3-751f6a-openshift-default-549c5bdf4d-78c97"
Sep 19 01:32:52.820459 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:52.820436    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89c7c2d8-487c-4c80-a630-dc0ffc760475-workload-certs\") pod \"e2e-shared-a-w3-751f6a-openshift-default-549c5bdf4d-78c97\" (UID: \"89c7c2d8-487c-4c80-a630-dc0ffc760475\") " pod="openshift-ingress/e2e-shared-a-w3-751f6a-openshift-default-549c5bdf4d-78c97"
Sep 19 01:32:52.820671 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:52.820461    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89c7c2d8-487c-4c80-a630-dc0ffc760475-istio-podinfo\") pod \"e2e-shared-a-w3-751f6a-openshift-default-549c5bdf4d-78c97\" (UID: \"89c7c2d8-487c-4c80-a630-dc0ffc760475\") " pod="openshift-ingress/e2e-shared-a-w3-751f6a-openshift-default-549c5bdf4d-78c97"
Sep 19 01:32:52.820671 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:52.820496    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdsbj\" (UniqueName: \"kubernetes.io/projected/89c7c2d8-487c-4c80-a630-dc0ffc760475-kube-api-access-rdsbj\") pod \"e2e-shared-a-w3-751f6a-openshift-default-549c5bdf4d-78c97\" (UID: \"89c7c2d8-487c-4c80-a630-dc0ffc760475\") " pod="openshift-ingress/e2e-shared-a-w3-751f6a-openshift-default-549c5bdf4d-78c97"
Sep 19 01:32:52.888619 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:52.888585    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-751f6a-openshift-default-549c5bdf4d-78c97"]
Sep 19 01:32:52.920984 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:52.920959    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89c7c2d8-487c-4c80-a630-dc0ffc760475-istio-data\") pod \"e2e-shared-a-w3-751f6a-openshift-default-549c5bdf4d-78c97\" (UID: \"89c7c2d8-487c-4c80-a630-dc0ffc760475\") " pod="openshift-ingress/e2e-shared-a-w3-751f6a-openshift-default-549c5bdf4d-78c97"
Sep 19 01:32:52.921130 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:52.920990    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89c7c2d8-487c-4c80-a630-dc0ffc760475-istiod-ca-cert\") pod \"e2e-shared-a-w3-751f6a-openshift-default-549c5bdf4d-78c97\" (UID: \"89c7c2d8-487c-4c80-a630-dc0ffc760475\") " pod="openshift-ingress/e2e-shared-a-w3-751f6a-openshift-default-549c5bdf4d-78c97"
Sep 19 01:32:52.921130 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:52.921011    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89c7c2d8-487c-4c80-a630-dc0ffc760475-workload-certs\") pod \"e2e-shared-a-w3-751f6a-openshift-default-549c5bdf4d-78c97\" (UID: \"89c7c2d8-487c-4c80-a630-dc0ffc760475\") " pod="openshift-ingress/e2e-shared-a-w3-751f6a-openshift-default-549c5bdf4d-78c97"
Sep 19 01:32:52.921130 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:52.921027    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89c7c2d8-487c-4c80-a630-dc0ffc760475-istio-podinfo\") pod \"e2e-shared-a-w3-751f6a-openshift-default-549c5bdf4d-78c97\" (UID: \"89c7c2d8-487c-4c80-a630-dc0ffc760475\") " pod="openshift-ingress/e2e-shared-a-w3-751f6a-openshift-default-549c5bdf4d-78c97"
Sep 19 01:32:52.921130 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:52.921048    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdsbj\" (UniqueName: \"kubernetes.io/projected/89c7c2d8-487c-4c80-a630-dc0ffc760475-kube-api-access-rdsbj\") pod \"e2e-shared-a-w3-751f6a-openshift-default-549c5bdf4d-78c97\" (UID: \"89c7c2d8-487c-4c80-a630-dc0ffc760475\") " pod="openshift-ingress/e2e-shared-a-w3-751f6a-openshift-default-549c5bdf4d-78c97"
Sep 19 01:32:52.921130 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:52.921080    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89c7c2d8-487c-4c80-a630-dc0ffc760475-credential-socket\") pod \"e2e-shared-a-w3-751f6a-openshift-default-549c5bdf4d-78c97\" (UID: \"89c7c2d8-487c-4c80-a630-dc0ffc760475\") " pod="openshift-ingress/e2e-shared-a-w3-751f6a-openshift-default-549c5bdf4d-78c97"
Sep 19 01:32:52.921130 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:52.921111    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89c7c2d8-487c-4c80-a630-dc0ffc760475-istio-envoy\") pod \"e2e-shared-a-w3-751f6a-openshift-default-549c5bdf4d-78c97\" (UID: \"89c7c2d8-487c-4c80-a630-dc0ffc760475\") " pod="openshift-ingress/e2e-shared-a-w3-751f6a-openshift-default-549c5bdf4d-78c97"
Sep 19 01:32:52.921459 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:52.921138    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89c7c2d8-487c-4c80-a630-dc0ffc760475-workload-socket\") pod \"e2e-shared-a-w3-751f6a-openshift-default-549c5bdf4d-78c97\" (UID: \"89c7c2d8-487c-4c80-a630-dc0ffc760475\") " pod="openshift-ingress/e2e-shared-a-w3-751f6a-openshift-default-549c5bdf4d-78c97"
Sep 19 01:32:52.921459 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:52.921182    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89c7c2d8-487c-4c80-a630-dc0ffc760475-istio-token\") pod \"e2e-shared-a-w3-751f6a-openshift-default-549c5bdf4d-78c97\" (UID: \"89c7c2d8-487c-4c80-a630-dc0ffc760475\") " pod="openshift-ingress/e2e-shared-a-w3-751f6a-openshift-default-549c5bdf4d-78c97"
Sep 19 01:32:52.921459 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:52.921374    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89c7c2d8-487c-4c80-a630-dc0ffc760475-istio-data\") pod \"e2e-shared-a-w3-751f6a-openshift-default-549c5bdf4d-78c97\" (UID: \"89c7c2d8-487c-4c80-a630-dc0ffc760475\") " pod="openshift-ingress/e2e-shared-a-w3-751f6a-openshift-default-549c5bdf4d-78c97"
Sep 19 01:32:52.921459 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:52.921403    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89c7c2d8-487c-4c80-a630-dc0ffc760475-workload-certs\") pod \"e2e-shared-a-w3-751f6a-openshift-default-549c5bdf4d-78c97\" (UID: \"89c7c2d8-487c-4c80-a630-dc0ffc760475\") " pod="openshift-ingress/e2e-shared-a-w3-751f6a-openshift-default-549c5bdf4d-78c97"
Sep 19 01:32:52.921627 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:52.921576    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89c7c2d8-487c-4c80-a630-dc0ffc760475-workload-socket\") pod \"e2e-shared-a-w3-751f6a-openshift-default-549c5bdf4d-78c97\" (UID: \"89c7c2d8-487c-4c80-a630-dc0ffc760475\") " pod="openshift-ingress/e2e-shared-a-w3-751f6a-openshift-default-549c5bdf4d-78c97"
Sep 19 01:32:52.921627 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:52.921583    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89c7c2d8-487c-4c80-a630-dc0ffc760475-credential-socket\") pod \"e2e-shared-a-w3-751f6a-openshift-default-549c5bdf4d-78c97\" (UID: \"89c7c2d8-487c-4c80-a630-dc0ffc760475\") " pod="openshift-ingress/e2e-shared-a-w3-751f6a-openshift-default-549c5bdf4d-78c97"
Sep 19 01:32:52.922272 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:52.922249    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89c7c2d8-487c-4c80-a630-dc0ffc760475-istiod-ca-cert\") pod \"e2e-shared-a-w3-751f6a-openshift-default-549c5bdf4d-78c97\" (UID: \"89c7c2d8-487c-4c80-a630-dc0ffc760475\") " pod="openshift-ingress/e2e-shared-a-w3-751f6a-openshift-default-549c5bdf4d-78c97"
Sep 19 01:32:52.923523 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:52.923501    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89c7c2d8-487c-4c80-a630-dc0ffc760475-istio-envoy\") pod \"e2e-shared-a-w3-751f6a-openshift-default-549c5bdf4d-78c97\" (UID: \"89c7c2d8-487c-4c80-a630-dc0ffc760475\") " pod="openshift-ingress/e2e-shared-a-w3-751f6a-openshift-default-549c5bdf4d-78c97"
Sep 19 01:32:52.923981 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:52.923961    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89c7c2d8-487c-4c80-a630-dc0ffc760475-istio-podinfo\") pod \"e2e-shared-a-w3-751f6a-openshift-default-549c5bdf4d-78c97\" (UID: \"89c7c2d8-487c-4c80-a630-dc0ffc760475\") " pod="openshift-ingress/e2e-shared-a-w3-751f6a-openshift-default-549c5bdf4d-78c97"
Sep 19 01:32:52.943014 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:52.942986    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89c7c2d8-487c-4c80-a630-dc0ffc760475-istio-token\") pod \"e2e-shared-a-w3-751f6a-openshift-default-549c5bdf4d-78c97\" (UID: \"89c7c2d8-487c-4c80-a630-dc0ffc760475\") " pod="openshift-ingress/e2e-shared-a-w3-751f6a-openshift-default-549c5bdf4d-78c97"
Sep 19 01:32:52.943413 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:52.943384    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdsbj\" (UniqueName: \"kubernetes.io/projected/89c7c2d8-487c-4c80-a630-dc0ffc760475-kube-api-access-rdsbj\") pod \"e2e-shared-a-w3-751f6a-openshift-default-549c5bdf4d-78c97\" (UID: \"89c7c2d8-487c-4c80-a630-dc0ffc760475\") " pod="openshift-ingress/e2e-shared-a-w3-751f6a-openshift-default-549c5bdf4d-78c97"
Sep 19 01:32:53.065944 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:53.065890    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-751f6a-openshift-default-549c5bdf4d-78c97"
Sep 19 01:32:53.356131 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:53.356106    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-751f6a-openshift-default-549c5bdf4d-78c97"]
Sep 19 01:32:53.357909 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:32:53.357883    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89c7c2d8_487c_4c80_a630_dc0ffc760475.slice/crio-304086b076e7820a915a7a3c81b57c49120326ffc88e7bf477ffd96343188a8d WatchSource:0}: Error finding container 304086b076e7820a915a7a3c81b57c49120326ffc88e7bf477ffd96343188a8d: Status 404 returned error can't find the container with id 304086b076e7820a915a7a3c81b57c49120326ffc88e7bf477ffd96343188a8d
Sep 19 01:32:53.364433 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:53.364393    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 19 01:32:53.364517 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:53.364487    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 19 01:32:53.364561 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:53.364538    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 19 01:32:53.566273 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:53.566235    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-751f6a-openshift-default-549c5bdf4d-78c97" event={"ID":"89c7c2d8-487c-4c80-a630-dc0ffc760475","Type":"ContainerStarted","Data":"30a726cf43deddaf51c8914f3a6e84104c511c46ccd290b406d5491592e2aa3e"}
Sep 19 01:32:53.566460 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:53.566302    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-751f6a-openshift-default-549c5bdf4d-78c97" event={"ID":"89c7c2d8-487c-4c80-a630-dc0ffc760475","Type":"ContainerStarted","Data":"304086b076e7820a915a7a3c81b57c49120326ffc88e7bf477ffd96343188a8d"}
Sep 19 01:32:53.595135 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:53.595074    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-751f6a-openshift-default-549c5bdf4d-78c97" podStartSLOduration=1.595052827 podStartE2EDuration="1.595052827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 01:32:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 01:32:53.593293646 +0000 UTC m=+1224.480962741" watchObservedRunningTime="2026-09-19 01:32:53.595052827 +0000 UTC m=+1224.482721905"
Sep 19 01:32:54.066808 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:54.066780    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-751f6a-openshift-default-549c5bdf4d-78c97"
Sep 19 01:32:54.072339 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:54.072319    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-751f6a-openshift-default-549c5bdf4d-78c97"
Sep 19 01:32:54.572492 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:54.572457    2581 generic.go:356] "Generic (PLEG): container finished" podID="62b1c519-4609-4138-a75c-7bcc23dcbf10" containerID="df1658114676d73ee724993ad81890e6cc11b4550ff0bd12d3b61329b8229945" exitCode=0
Sep 19 01:32:54.572652 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:54.572534    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b5fe4519-openshift-default-84cfc86584-vxfnt" event={"ID":"62b1c519-4609-4138-a75c-7bcc23dcbf10","Type":"ContainerDied","Data":"df1658114676d73ee724993ad81890e6cc11b4550ff0bd12d3b61329b8229945"}
Sep 19 01:32:54.572916 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:54.572895    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-751f6a-openshift-default-549c5bdf4d-78c97"
Sep 19 01:32:54.573061 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:54.573048    2581 scope.go:117] "RemoveContainer" containerID="df1658114676d73ee724993ad81890e6cc11b4550ff0bd12d3b61329b8229945"
Sep 19 01:32:54.573850 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:54.573751    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-751f6a-openshift-default-549c5bdf4d-78c97"
Sep 19 01:32:54.574196 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:54.574167    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 19 01:32:54.574255 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:54.574241    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 19 01:32:54.574293 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:54.574278    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 19 01:32:55.580068 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:55.580033    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b5fe4519-openshift-default-84cfc86584-vxfnt" event={"ID":"62b1c519-4609-4138-a75c-7bcc23dcbf10","Type":"ContainerStarted","Data":"d1ed080b26198bd2b779f38ab755e6a358a750a0bd85da0be6ca2e2a07e39e3d"}
Sep 19 01:32:55.641345 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:55.641317    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-b5fe4519-openshift-default-84cfc86584-vxfnt"
Sep 19 01:32:55.642223 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:55.642201    2581 patch_prober.go:28] interesting pod/e2e-mt-b5fe4519-openshift-default-84cfc86584-vxfnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Sep 19 01:32:55.642338 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:55.642252    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b5fe4519-openshift-default-84cfc86584-vxfnt" podUID="62b1c519-4609-4138-a75c-7bcc23dcbf10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Sep 19 01:32:55.996464 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:55.996415    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-8e708f/distinct-2-w0-8e708f-kserve-59b54cd666-tg2h7"]
Sep 19 01:32:55.999791 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:55.999768    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-8e708f/distinct-2-w0-8e708f-kserve-59b54cd666-tg2h7"
Sep 19 01:32:56.002747 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:56.002720    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-8e708f\"/\"distinct-2-w0-8e708f-kserve-self-signed-certs\""
Sep 19 01:32:56.009951 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:56.009927    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-8e708f/distinct-2-w0-8e708f-kserve-59b54cd666-tg2h7"]
Sep 19 01:32:56.050104 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:56.050075    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9430ca55-a991-45b3-840c-9fc6fe68d4d6-tls-certs\") pod \"distinct-2-w0-8e708f-kserve-59b54cd666-tg2h7\" (UID: \"9430ca55-a991-45b3-840c-9fc6fe68d4d6\") " pod="e2e-models-e2e-worker-w0-8e708f/distinct-2-w0-8e708f-kserve-59b54cd666-tg2h7"
Sep 19 01:32:56.050205 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:56.050123    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9430ca55-a991-45b3-840c-9fc6fe68d4d6-model-cache\") pod \"distinct-2-w0-8e708f-kserve-59b54cd666-tg2h7\" (UID: \"9430ca55-a991-45b3-840c-9fc6fe68d4d6\") " pod="e2e-models-e2e-worker-w0-8e708f/distinct-2-w0-8e708f-kserve-59b54cd666-tg2h7"
Sep 19 01:32:56.050205 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:56.050174    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7bnb\" (UniqueName: \"kubernetes.io/projected/9430ca55-a991-45b3-840c-9fc6fe68d4d6-kube-api-access-n7bnb\") pod \"distinct-2-w0-8e708f-kserve-59b54cd666-tg2h7\" (UID: \"9430ca55-a991-45b3-840c-9fc6fe68d4d6\") " pod="e2e-models-e2e-worker-w0-8e708f/distinct-2-w0-8e708f-kserve-59b54cd666-tg2h7"
Sep 19 01:32:56.050336 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:56.050213    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9430ca55-a991-45b3-840c-9fc6fe68d4d6-tmp-dir\") pod \"distinct-2-w0-8e708f-kserve-59b54cd666-tg2h7\" (UID: \"9430ca55-a991-45b3-840c-9fc6fe68d4d6\") " pod="e2e-models-e2e-worker-w0-8e708f/distinct-2-w0-8e708f-kserve-59b54cd666-tg2h7"
Sep 19 01:32:56.050336 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:56.050250    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9430ca55-a991-45b3-840c-9fc6fe68d4d6-home\") pod \"distinct-2-w0-8e708f-kserve-59b54cd666-tg2h7\" (UID: \"9430ca55-a991-45b3-840c-9fc6fe68d4d6\") " pod="e2e-models-e2e-worker-w0-8e708f/distinct-2-w0-8e708f-kserve-59b54cd666-tg2h7"
Sep 19 01:32:56.050440 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:56.050360    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9430ca55-a991-45b3-840c-9fc6fe68d4d6-dshm\") pod \"distinct-2-w0-8e708f-kserve-59b54cd666-tg2h7\" (UID: \"9430ca55-a991-45b3-840c-9fc6fe68d4d6\") " pod="e2e-models-e2e-worker-w0-8e708f/distinct-2-w0-8e708f-kserve-59b54cd666-tg2h7"
Sep 19 01:32:56.151738 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:56.151698    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9430ca55-a991-45b3-840c-9fc6fe68d4d6-tls-certs\") pod \"distinct-2-w0-8e708f-kserve-59b54cd666-tg2h7\" (UID: \"9430ca55-a991-45b3-840c-9fc6fe68d4d6\") " pod="e2e-models-e2e-worker-w0-8e708f/distinct-2-w0-8e708f-kserve-59b54cd666-tg2h7"
Sep 19 01:32:56.151965 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:56.151750    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9430ca55-a991-45b3-840c-9fc6fe68d4d6-model-cache\") pod \"distinct-2-w0-8e708f-kserve-59b54cd666-tg2h7\" (UID: \"9430ca55-a991-45b3-840c-9fc6fe68d4d6\") " pod="e2e-models-e2e-worker-w0-8e708f/distinct-2-w0-8e708f-kserve-59b54cd666-tg2h7"
Sep 19 01:32:56.151965 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:56.151886    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7bnb\" (UniqueName: \"kubernetes.io/projected/9430ca55-a991-45b3-840c-9fc6fe68d4d6-kube-api-access-n7bnb\") pod \"distinct-2-w0-8e708f-kserve-59b54cd666-tg2h7\" (UID: \"9430ca55-a991-45b3-840c-9fc6fe68d4d6\") " pod="e2e-models-e2e-worker-w0-8e708f/distinct-2-w0-8e708f-kserve-59b54cd666-tg2h7"
Sep 19 01:32:56.151965 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:56.151923    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9430ca55-a991-45b3-840c-9fc6fe68d4d6-tmp-dir\") pod \"distinct-2-w0-8e708f-kserve-59b54cd666-tg2h7\" (UID: \"9430ca55-a991-45b3-840c-9fc6fe68d4d6\") " pod="e2e-models-e2e-worker-w0-8e708f/distinct-2-w0-8e708f-kserve-59b54cd666-tg2h7"
Sep 19 01:32:56.151965 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:56.151949    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9430ca55-a991-45b3-840c-9fc6fe68d4d6-home\") pod \"distinct-2-w0-8e708f-kserve-59b54cd666-tg2h7\" (UID: \"9430ca55-a991-45b3-840c-9fc6fe68d4d6\") " pod="e2e-models-e2e-worker-w0-8e708f/distinct-2-w0-8e708f-kserve-59b54cd666-tg2h7"
Sep 19 01:32:56.152201 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:56.152000    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9430ca55-a991-45b3-840c-9fc6fe68d4d6-dshm\") pod \"distinct-2-w0-8e708f-kserve-59b54cd666-tg2h7\" (UID: \"9430ca55-a991-45b3-840c-9fc6fe68d4d6\") " pod="e2e-models-e2e-worker-w0-8e708f/distinct-2-w0-8e708f-kserve-59b54cd666-tg2h7"
Sep 19 01:32:56.152201 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:56.152180    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9430ca55-a991-45b3-840c-9fc6fe68d4d6-model-cache\") pod \"distinct-2-w0-8e708f-kserve-59b54cd666-tg2h7\" (UID: \"9430ca55-a991-45b3-840c-9fc6fe68d4d6\") " pod="e2e-models-e2e-worker-w0-8e708f/distinct-2-w0-8e708f-kserve-59b54cd666-tg2h7"
Sep 19 01:32:56.152299 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:56.152278    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9430ca55-a991-45b3-840c-9fc6fe68d4d6-tmp-dir\") pod \"distinct-2-w0-8e708f-kserve-59b54cd666-tg2h7\" (UID: \"9430ca55-a991-45b3-840c-9fc6fe68d4d6\") " pod="e2e-models-e2e-worker-w0-8e708f/distinct-2-w0-8e708f-kserve-59b54cd666-tg2h7"
Sep 19 01:32:56.152344 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:56.152327    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9430ca55-a991-45b3-840c-9fc6fe68d4d6-home\") pod \"distinct-2-w0-8e708f-kserve-59b54cd666-tg2h7\" (UID: \"9430ca55-a991-45b3-840c-9fc6fe68d4d6\") " pod="e2e-models-e2e-worker-w0-8e708f/distinct-2-w0-8e708f-kserve-59b54cd666-tg2h7"
Sep 19 01:32:56.154335 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:56.154312    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9430ca55-a991-45b3-840c-9fc6fe68d4d6-dshm\") pod \"distinct-2-w0-8e708f-kserve-59b54cd666-tg2h7\" (UID: \"9430ca55-a991-45b3-840c-9fc6fe68d4d6\") " pod="e2e-models-e2e-worker-w0-8e708f/distinct-2-w0-8e708f-kserve-59b54cd666-tg2h7"
Sep 19 01:32:56.154538 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:56.154522    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9430ca55-a991-45b3-840c-9fc6fe68d4d6-tls-certs\") pod \"distinct-2-w0-8e708f-kserve-59b54cd666-tg2h7\" (UID: \"9430ca55-a991-45b3-840c-9fc6fe68d4d6\") " pod="e2e-models-e2e-worker-w0-8e708f/distinct-2-w0-8e708f-kserve-59b54cd666-tg2h7"
Sep 19 01:32:56.163118 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:56.163098    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7bnb\" (UniqueName: \"kubernetes.io/projected/9430ca55-a991-45b3-840c-9fc6fe68d4d6-kube-api-access-n7bnb\") pod \"distinct-2-w0-8e708f-kserve-59b54cd666-tg2h7\" (UID: \"9430ca55-a991-45b3-840c-9fc6fe68d4d6\") " pod="e2e-models-e2e-worker-w0-8e708f/distinct-2-w0-8e708f-kserve-59b54cd666-tg2h7"
Sep 19 01:32:56.311003 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:56.310913    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-8e708f/distinct-2-w0-8e708f-kserve-59b54cd666-tg2h7"
Sep 19 01:32:56.589467 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:56.589368    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-8e708f/distinct-2-w0-8e708f-kserve-59b54cd666-tg2h7"]
Sep 19 01:32:56.593223 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:32:56.593170    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9430ca55_a991_45b3_840c_9fc6fe68d4d6.slice/crio-aa0bab60b71c3d906c5bfc5a3e44576fc01b64b78d8c11f1ddbf4d8c2d16e2b1 WatchSource:0}: Error finding container aa0bab60b71c3d906c5bfc5a3e44576fc01b64b78d8c11f1ddbf4d8c2d16e2b1: Status 404 returned error can't find the container with id aa0bab60b71c3d906c5bfc5a3e44576fc01b64b78d8c11f1ddbf4d8c2d16e2b1
Sep 19 01:32:56.640822 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:56.640788    2581 patch_prober.go:28] interesting pod/e2e-mt-b5fe4519-openshift-default-84cfc86584-vxfnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Sep 19 01:32:56.640946 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:56.640842    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b5fe4519-openshift-default-84cfc86584-vxfnt" podUID="62b1c519-4609-4138-a75c-7bcc23dcbf10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Sep 19 01:32:56.941186 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:56.941159    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5a005d-openshift-default-cc9788f6b-twz5d"
Sep 19 01:32:57.061635 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:57.061603    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2302d85f-6dba-4f71-bc69-6c64172f0f93-istiod-ca-cert\") pod \"2302d85f-6dba-4f71-bc69-6c64172f0f93\" (UID: \"2302d85f-6dba-4f71-bc69-6c64172f0f93\") "
Sep 19 01:32:57.061838 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:57.061664    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2302d85f-6dba-4f71-bc69-6c64172f0f93-workload-socket\") pod \"2302d85f-6dba-4f71-bc69-6c64172f0f93\" (UID: \"2302d85f-6dba-4f71-bc69-6c64172f0f93\") "
Sep 19 01:32:57.061838 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:57.061715    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2302d85f-6dba-4f71-bc69-6c64172f0f93-istio-data\") pod \"2302d85f-6dba-4f71-bc69-6c64172f0f93\" (UID: \"2302d85f-6dba-4f71-bc69-6c64172f0f93\") "
Sep 19 01:32:57.061838 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:57.061744    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2302d85f-6dba-4f71-bc69-6c64172f0f93-istio-token\") pod \"2302d85f-6dba-4f71-bc69-6c64172f0f93\" (UID: \"2302d85f-6dba-4f71-bc69-6c64172f0f93\") "
Sep 19 01:32:57.061838 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:57.061761    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2302d85f-6dba-4f71-bc69-6c64172f0f93-istio-envoy\") pod \"2302d85f-6dba-4f71-bc69-6c64172f0f93\" (UID: \"2302d85f-6dba-4f71-bc69-6c64172f0f93\") "
Sep 19 01:32:57.061838 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:57.061780    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2302d85f-6dba-4f71-bc69-6c64172f0f93-workload-certs\") pod \"2302d85f-6dba-4f71-bc69-6c64172f0f93\" (UID: \"2302d85f-6dba-4f71-bc69-6c64172f0f93\") "
Sep 19 01:32:57.062159 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:57.061885    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2302d85f-6dba-4f71-bc69-6c64172f0f93-credential-socket\") pod \"2302d85f-6dba-4f71-bc69-6c64172f0f93\" (UID: \"2302d85f-6dba-4f71-bc69-6c64172f0f93\") "
Sep 19 01:32:57.062159 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:57.061939    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2302d85f-6dba-4f71-bc69-6c64172f0f93-istio-podinfo\") pod \"2302d85f-6dba-4f71-bc69-6c64172f0f93\" (UID: \"2302d85f-6dba-4f71-bc69-6c64172f0f93\") "
Sep 19 01:32:57.062159 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:57.061977    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bxlh4\" (UniqueName: \"kubernetes.io/projected/2302d85f-6dba-4f71-bc69-6c64172f0f93-kube-api-access-bxlh4\") pod \"2302d85f-6dba-4f71-bc69-6c64172f0f93\" (UID: \"2302d85f-6dba-4f71-bc69-6c64172f0f93\") "
Sep 19 01:32:57.062159 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:57.062030    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2302d85f-6dba-4f71-bc69-6c64172f0f93-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2302d85f-6dba-4f71-bc69-6c64172f0f93" (UID: "2302d85f-6dba-4f71-bc69-6c64172f0f93"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 01:32:57.062159 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:57.062113    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2302d85f-6dba-4f71-bc69-6c64172f0f93-istio-data" (OuterVolumeSpecName: "istio-data") pod "2302d85f-6dba-4f71-bc69-6c64172f0f93" (UID: "2302d85f-6dba-4f71-bc69-6c64172f0f93"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 01:32:57.062159 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:57.062132    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2302d85f-6dba-4f71-bc69-6c64172f0f93-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2302d85f-6dba-4f71-bc69-6c64172f0f93" (UID: "2302d85f-6dba-4f71-bc69-6c64172f0f93"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 01:32:57.062159 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:57.062060    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2302d85f-6dba-4f71-bc69-6c64172f0f93-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2302d85f-6dba-4f71-bc69-6c64172f0f93" (UID: "2302d85f-6dba-4f71-bc69-6c64172f0f93"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 01:32:57.062533 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:57.062325    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2302d85f-6dba-4f71-bc69-6c64172f0f93-istio-data\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:32:57.062533 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:57.062344    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2302d85f-6dba-4f71-bc69-6c64172f0f93-workload-certs\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:32:57.062533 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:57.062345    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2302d85f-6dba-4f71-bc69-6c64172f0f93-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2302d85f-6dba-4f71-bc69-6c64172f0f93" (UID: "2302d85f-6dba-4f71-bc69-6c64172f0f93"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 01:32:57.062533 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:57.062357    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2302d85f-6dba-4f71-bc69-6c64172f0f93-istiod-ca-cert\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:32:57.062533 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:57.062372    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2302d85f-6dba-4f71-bc69-6c64172f0f93-workload-socket\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:32:57.064155 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:57.064135    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2302d85f-6dba-4f71-bc69-6c64172f0f93-istio-token" (OuterVolumeSpecName: "istio-token") pod "2302d85f-6dba-4f71-bc69-6c64172f0f93" (UID: "2302d85f-6dba-4f71-bc69-6c64172f0f93"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 01:32:57.064246 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:57.064182    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2302d85f-6dba-4f71-bc69-6c64172f0f93-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2302d85f-6dba-4f71-bc69-6c64172f0f93" (UID: "2302d85f-6dba-4f71-bc69-6c64172f0f93"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 19 01:32:57.064631 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:57.064613    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2302d85f-6dba-4f71-bc69-6c64172f0f93-kube-api-access-bxlh4" (OuterVolumeSpecName: "kube-api-access-bxlh4") pod "2302d85f-6dba-4f71-bc69-6c64172f0f93" (UID: "2302d85f-6dba-4f71-bc69-6c64172f0f93"). InnerVolumeSpecName "kube-api-access-bxlh4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 01:32:57.064631 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:57.064618    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2302d85f-6dba-4f71-bc69-6c64172f0f93-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2302d85f-6dba-4f71-bc69-6c64172f0f93" (UID: "2302d85f-6dba-4f71-bc69-6c64172f0f93"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 01:32:57.111733 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:57.111642    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c9nsz"]
Sep 19 01:32:57.163974 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:57.163951    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2302d85f-6dba-4f71-bc69-6c64172f0f93-istio-podinfo\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:32:57.164085 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:57.163978    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxlh4\" (UniqueName: \"kubernetes.io/projected/2302d85f-6dba-4f71-bc69-6c64172f0f93-kube-api-access-bxlh4\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:32:57.164085 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:57.163994    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2302d85f-6dba-4f71-bc69-6c64172f0f93-istio-token\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:32:57.164085 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:57.164007    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2302d85f-6dba-4f71-bc69-6c64172f0f93-istio-envoy\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:32:57.164085 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:57.164019    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2302d85f-6dba-4f71-bc69-6c64172f0f93-credential-socket\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:32:57.597759 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:57.597712    2581 generic.go:356] "Generic (PLEG): container finished" podID="2302d85f-6dba-4f71-bc69-6c64172f0f93" containerID="02374b819abfffc5abb51b208e9cd18dc7d7ee6f29d926f9e3a10ff82eb0bb4b" exitCode=0
Sep 19 01:32:57.598273 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:57.597814    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5a005d-openshift-default-cc9788f6b-twz5d" event={"ID":"2302d85f-6dba-4f71-bc69-6c64172f0f93","Type":"ContainerDied","Data":"02374b819abfffc5abb51b208e9cd18dc7d7ee6f29d926f9e3a10ff82eb0bb4b"}
Sep 19 01:32:57.598273 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:57.597890    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5a005d-openshift-default-cc9788f6b-twz5d" event={"ID":"2302d85f-6dba-4f71-bc69-6c64172f0f93","Type":"ContainerDied","Data":"04db7ad36bc10ea97a77818b18e3f6f7e4420067a099e765c64ae4945c9cbd99"}
Sep 19 01:32:57.598273 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:57.597925    2581 scope.go:117] "RemoveContainer" containerID="02374b819abfffc5abb51b208e9cd18dc7d7ee6f29d926f9e3a10ff82eb0bb4b"
Sep 19 01:32:57.598273 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:57.598106    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5a005d-openshift-default-cc9788f6b-twz5d"
Sep 19 01:32:57.603644 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:57.602682    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-8e708f/distinct-2-w0-8e708f-kserve-59b54cd666-tg2h7" event={"ID":"9430ca55-a991-45b3-840c-9fc6fe68d4d6","Type":"ContainerStarted","Data":"e6b1ad4704cec6fa600617e539c79d9ac3186e3942d205c25630d6c277e2931a"}
Sep 19 01:32:57.603644 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:57.602725    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-8e708f/distinct-2-w0-8e708f-kserve-59b54cd666-tg2h7" event={"ID":"9430ca55-a991-45b3-840c-9fc6fe68d4d6","Type":"ContainerStarted","Data":"aa0bab60b71c3d906c5bfc5a3e44576fc01b64b78d8c11f1ddbf4d8c2d16e2b1"}
Sep 19 01:32:57.611180 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:57.611161    2581 scope.go:117] "RemoveContainer" containerID="c8acbd4b6408ab58b32f26ce0478f8b58e17a2f5a39eb764f0bfaa7d9a1df10f"
Sep 19 01:32:57.619101 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:57.619084    2581 scope.go:117] "RemoveContainer" containerID="02374b819abfffc5abb51b208e9cd18dc7d7ee6f29d926f9e3a10ff82eb0bb4b"
Sep 19 01:32:57.619381 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:32:57.619362    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02374b819abfffc5abb51b208e9cd18dc7d7ee6f29d926f9e3a10ff82eb0bb4b\": container with ID starting with 02374b819abfffc5abb51b208e9cd18dc7d7ee6f29d926f9e3a10ff82eb0bb4b not found: ID does not exist" containerID="02374b819abfffc5abb51b208e9cd18dc7d7ee6f29d926f9e3a10ff82eb0bb4b"
Sep 19 01:32:57.619460 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:57.619393    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02374b819abfffc5abb51b208e9cd18dc7d7ee6f29d926f9e3a10ff82eb0bb4b"} err="failed to get container status \"02374b819abfffc5abb51b208e9cd18dc7d7ee6f29d926f9e3a10ff82eb0bb4b\": rpc error: code = NotFound desc = could not find container \"02374b819abfffc5abb51b208e9cd18dc7d7ee6f29d926f9e3a10ff82eb0bb4b\": container with ID starting with 02374b819abfffc5abb51b208e9cd18dc7d7ee6f29d926f9e3a10ff82eb0bb4b not found: ID does not exist"
Sep 19 01:32:57.619460 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:57.619424    2581 scope.go:117] "RemoveContainer" containerID="c8acbd4b6408ab58b32f26ce0478f8b58e17a2f5a39eb764f0bfaa7d9a1df10f"
Sep 19 01:32:57.619685 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:32:57.619669    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8acbd4b6408ab58b32f26ce0478f8b58e17a2f5a39eb764f0bfaa7d9a1df10f\": container with ID starting with c8acbd4b6408ab58b32f26ce0478f8b58e17a2f5a39eb764f0bfaa7d9a1df10f not found: ID does not exist" containerID="c8acbd4b6408ab58b32f26ce0478f8b58e17a2f5a39eb764f0bfaa7d9a1df10f"
Sep 19 01:32:57.619729 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:57.619692    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8acbd4b6408ab58b32f26ce0478f8b58e17a2f5a39eb764f0bfaa7d9a1df10f"} err="failed to get container status \"c8acbd4b6408ab58b32f26ce0478f8b58e17a2f5a39eb764f0bfaa7d9a1df10f\": rpc error: code = NotFound desc = could not find container \"c8acbd4b6408ab58b32f26ce0478f8b58e17a2f5a39eb764f0bfaa7d9a1df10f\": container with ID starting with c8acbd4b6408ab58b32f26ce0478f8b58e17a2f5a39eb764f0bfaa7d9a1df10f not found: ID does not exist"
Sep 19 01:32:57.627733 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:57.627687    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-8e708f/distinct-2-w0-8e708f-kserve-59b54cd666-tg2h7" podStartSLOduration=2.627673036 podStartE2EDuration="2.627673036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 01:32:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 01:32:57.625435944 +0000 UTC m=+1228.513105021" watchObservedRunningTime="2026-09-19 01:32:57.627673036 +0000 UTC m=+1228.515342113"
Sep 19 01:32:57.641160 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:57.641139    2581 patch_prober.go:28] interesting pod/e2e-mt-b5fe4519-openshift-default-84cfc86584-vxfnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Sep 19 01:32:57.641223 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:57.641180    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b5fe4519-openshift-default-84cfc86584-vxfnt" podUID="62b1c519-4609-4138-a75c-7bcc23dcbf10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Sep 19 01:32:57.649278 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:57.649256    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-5a005d-openshift-default-cc9788f6b-twz5d"]
Sep 19 01:32:57.652416 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:57.652396    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-5a005d-openshift-default-cc9788f6b-twz5d"]
Sep 19 01:32:58.640649 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:58.640615    2581 patch_prober.go:28] interesting pod/e2e-mt-b5fe4519-openshift-default-84cfc86584-vxfnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Sep 19 01:32:58.641058 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:58.640672    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b5fe4519-openshift-default-84cfc86584-vxfnt" podUID="62b1c519-4609-4138-a75c-7bcc23dcbf10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Sep 19 01:32:59.272154 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:59.272114    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2302d85f-6dba-4f71-bc69-6c64172f0f93" path="/var/lib/kubelet/pods/2302d85f-6dba-4f71-bc69-6c64172f0f93/volumes"
Sep 19 01:32:59.434349 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:59.434315    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b5fe4519-openshift-default-84cfc86584-vxfnt"
Sep 19 01:32:59.640914 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:59.640848    2581 patch_prober.go:28] interesting pod/e2e-mt-b5fe4519-openshift-default-84cfc86584-vxfnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Sep 19 01:32:59.641368 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:32:59.640925    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b5fe4519-openshift-default-84cfc86584-vxfnt" podUID="62b1c519-4609-4138-a75c-7bcc23dcbf10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Sep 19 01:33:00.640840 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:00.640801    2581 patch_prober.go:28] interesting pod/e2e-mt-b5fe4519-openshift-default-84cfc86584-vxfnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Sep 19 01:33:00.641066 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:00.640887    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b5fe4519-openshift-default-84cfc86584-vxfnt" podUID="62b1c519-4609-4138-a75c-7bcc23dcbf10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Sep 19 01:33:01.474810 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:01.474779    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-751f6a-f5b56d759-n267t"]
Sep 19 01:33:01.475118 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:01.475104    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2302d85f-6dba-4f71-bc69-6c64172f0f93" containerName="istio-proxy"
Sep 19 01:33:01.475170 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:01.475119    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2302d85f-6dba-4f71-bc69-6c64172f0f93" containerName="istio-proxy"
Sep 19 01:33:01.475203 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:01.475177    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="2302d85f-6dba-4f71-bc69-6c64172f0f93" containerName="istio-proxy"
Sep 19 01:33:01.475203 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:01.475186    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="2302d85f-6dba-4f71-bc69-6c64172f0f93" containerName="istio-proxy"
Sep 19 01:33:01.478258 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:01.478242    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-751f6a-f5b56d759-n267t"
Sep 19 01:33:01.493546 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:01.493526    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-751f6a\""
Sep 19 01:33:01.493798 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:01.493782    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-751f6a-dockercfg-fczpp\""
Sep 19 01:33:01.506319 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:01.506296    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-751f6a-f5b56d759-n267t"]
Sep 19 01:33:01.586131 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:01.586102    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-751f6a-8546c5d9f8-zw744"]
Sep 19 01:33:01.586469 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:01.586456    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2302d85f-6dba-4f71-bc69-6c64172f0f93" containerName="istio-proxy"
Sep 19 01:33:01.586513 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:01.586472    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2302d85f-6dba-4f71-bc69-6c64172f0f93" containerName="istio-proxy"
Sep 19 01:33:01.588400 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:01.588386    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-751f6a-8546c5d9f8-zw744"
Sep 19 01:33:01.606785 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:01.606762    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-751f6a-8546c5d9f8-zw744"]
Sep 19 01:33:01.616607 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:01.616577    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drqx2\" (UniqueName: \"kubernetes.io/projected/950dbb17-8019-4b84-859b-dbc14565592b-kube-api-access-drqx2\") pod \"payload-processing-e2e-shared-a-w3-751f6a-f5b56d759-n267t\" (UID: \"950dbb17-8019-4b84-859b-dbc14565592b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-751f6a-f5b56d759-n267t"
Sep 19 01:33:01.616712 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:01.616690    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/950dbb17-8019-4b84-859b-dbc14565592b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-751f6a-f5b56d759-n267t\" (UID: \"950dbb17-8019-4b84-859b-dbc14565592b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-751f6a-f5b56d759-n267t"
Sep 19 01:33:01.645196 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:01.645173    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-b5fe4519-openshift-default-84cfc86584-vxfnt"
Sep 19 01:33:01.646317 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:01.646299    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-b5fe4519-openshift-default-84cfc86584-vxfnt"
Sep 19 01:33:01.718160 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:01.718134    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9nb4\" (UniqueName: \"kubernetes.io/projected/e985fb28-5b5b-4515-b453-e3c0e1721e04-kube-api-access-r9nb4\") pod \"payload-pre-processing-e2e-shared-a-w3-751f6a-8546c5d9f8-zw744\" (UID: \"e985fb28-5b5b-4515-b453-e3c0e1721e04\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-751f6a-8546c5d9f8-zw744"
Sep 19 01:33:01.718336 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:01.718310    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e985fb28-5b5b-4515-b453-e3c0e1721e04-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-751f6a-8546c5d9f8-zw744\" (UID: \"e985fb28-5b5b-4515-b453-e3c0e1721e04\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-751f6a-8546c5d9f8-zw744"
Sep 19 01:33:01.718409 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:01.718365    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/950dbb17-8019-4b84-859b-dbc14565592b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-751f6a-f5b56d759-n267t\" (UID: \"950dbb17-8019-4b84-859b-dbc14565592b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-751f6a-f5b56d759-n267t"
Sep 19 01:33:01.718480 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:01.718465    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drqx2\" (UniqueName: \"kubernetes.io/projected/950dbb17-8019-4b84-859b-dbc14565592b-kube-api-access-drqx2\") pod \"payload-processing-e2e-shared-a-w3-751f6a-f5b56d759-n267t\" (UID: \"950dbb17-8019-4b84-859b-dbc14565592b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-751f6a-f5b56d759-n267t"
Sep 19 01:33:01.719052 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:01.719035    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/950dbb17-8019-4b84-859b-dbc14565592b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-751f6a-f5b56d759-n267t\" (UID: \"950dbb17-8019-4b84-859b-dbc14565592b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-751f6a-f5b56d759-n267t"
Sep 19 01:33:01.743567 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:01.743503    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drqx2\" (UniqueName: \"kubernetes.io/projected/950dbb17-8019-4b84-859b-dbc14565592b-kube-api-access-drqx2\") pod \"payload-processing-e2e-shared-a-w3-751f6a-f5b56d759-n267t\" (UID: \"950dbb17-8019-4b84-859b-dbc14565592b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-751f6a-f5b56d759-n267t"
Sep 19 01:33:01.788438 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:01.788412    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-751f6a-f5b56d759-n267t"
Sep 19 01:33:01.819323 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:01.819297    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e985fb28-5b5b-4515-b453-e3c0e1721e04-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-751f6a-8546c5d9f8-zw744\" (UID: \"e985fb28-5b5b-4515-b453-e3c0e1721e04\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-751f6a-8546c5d9f8-zw744"
Sep 19 01:33:01.819452 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:01.819379    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9nb4\" (UniqueName: \"kubernetes.io/projected/e985fb28-5b5b-4515-b453-e3c0e1721e04-kube-api-access-r9nb4\") pod \"payload-pre-processing-e2e-shared-a-w3-751f6a-8546c5d9f8-zw744\" (UID: \"e985fb28-5b5b-4515-b453-e3c0e1721e04\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-751f6a-8546c5d9f8-zw744"
Sep 19 01:33:01.819995 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:01.819974    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e985fb28-5b5b-4515-b453-e3c0e1721e04-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-751f6a-8546c5d9f8-zw744\" (UID: \"e985fb28-5b5b-4515-b453-e3c0e1721e04\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-751f6a-8546c5d9f8-zw744"
Sep 19 01:33:01.830435 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:01.830407    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9nb4\" (UniqueName: \"kubernetes.io/projected/e985fb28-5b5b-4515-b453-e3c0e1721e04-kube-api-access-r9nb4\") pod \"payload-pre-processing-e2e-shared-a-w3-751f6a-8546c5d9f8-zw744\" (UID: \"e985fb28-5b5b-4515-b453-e3c0e1721e04\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-751f6a-8546c5d9f8-zw744"
Sep 19 01:33:01.897950 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:01.897924    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-751f6a-8546c5d9f8-zw744"
Sep 19 01:33:02.094570 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:02.094543    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-751f6a-f5b56d759-n267t"]
Sep 19 01:33:02.097135 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:33:02.097109    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod950dbb17_8019_4b84_859b_dbc14565592b.slice/crio-9b8b78bd7023435768f9c46e49094f1a7d2b4226ca0c8f762582425ae99b7ee9 WatchSource:0}: Error finding container 9b8b78bd7023435768f9c46e49094f1a7d2b4226ca0c8f762582425ae99b7ee9: Status 404 returned error can't find the container with id 9b8b78bd7023435768f9c46e49094f1a7d2b4226ca0c8f762582425ae99b7ee9
Sep 19 01:33:02.187065 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:02.187039    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-751f6a-8546c5d9f8-zw744"]
Sep 19 01:33:02.188886 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:33:02.188844    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode985fb28_5b5b_4515_b453_e3c0e1721e04.slice/crio-29a97e33e5888c09f164582ffa4179ce7e2f414b68287fa5cc3b8b0e054e754c WatchSource:0}: Error finding container 29a97e33e5888c09f164582ffa4179ce7e2f414b68287fa5cc3b8b0e054e754c: Status 404 returned error can't find the container with id 29a97e33e5888c09f164582ffa4179ce7e2f414b68287fa5cc3b8b0e054e754c
Sep 19 01:33:02.633071 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:02.633037    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-751f6a-f5b56d759-n267t" event={"ID":"950dbb17-8019-4b84-859b-dbc14565592b","Type":"ContainerStarted","Data":"a4259819185e0b06ed90590e3541e6eb05cda95799be4614d06ff31d4d57c847"}
Sep 19 01:33:02.633238 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:02.633100    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-751f6a-f5b56d759-n267t" event={"ID":"950dbb17-8019-4b84-859b-dbc14565592b","Type":"ContainerStarted","Data":"9b8b78bd7023435768f9c46e49094f1a7d2b4226ca0c8f762582425ae99b7ee9"}
Sep 19 01:33:02.633238 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:02.633130    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-751f6a-f5b56d759-n267t"
Sep 19 01:33:02.634757 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:02.634726    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-751f6a-8546c5d9f8-zw744" event={"ID":"e985fb28-5b5b-4515-b453-e3c0e1721e04","Type":"ContainerStarted","Data":"d0697cee6805fbc76735bb84618d8f41373e0f48096e78d30d8e9f6cea96349a"}
Sep 19 01:33:02.634883 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:02.634781    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-751f6a-8546c5d9f8-zw744" event={"ID":"e985fb28-5b5b-4515-b453-e3c0e1721e04","Type":"ContainerStarted","Data":"29a97e33e5888c09f164582ffa4179ce7e2f414b68287fa5cc3b8b0e054e754c"}
Sep 19 01:33:02.634883 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:02.634872    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-751f6a-8546c5d9f8-zw744"
Sep 19 01:33:02.689325 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:02.689285    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-751f6a-8546c5d9f8-zw744" podStartSLOduration=1.6892696630000001 podStartE2EDuration="1.689269663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 01:33:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 01:33:02.687726799 +0000 UTC m=+1233.575395879" watchObservedRunningTime="2026-09-19 01:33:02.689269663 +0000 UTC m=+1233.576938739"
Sep 19 01:33:02.689770 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:02.689708    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-751f6a-f5b56d759-n267t" podStartSLOduration=1.689693953 podStartE2EDuration="1.689693953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 01:33:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 01:33:02.65684586 +0000 UTC m=+1233.544514941" watchObservedRunningTime="2026-09-19 01:33:02.689693953 +0000 UTC m=+1233.577363031"
Sep 19 01:33:06.311213 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:06.311179    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-8e708f/distinct-2-w0-8e708f-kserve-59b54cd666-tg2h7"
Sep 19 01:33:06.311631 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:06.311235    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-8e708f/distinct-2-w0-8e708f-kserve-59b54cd666-tg2h7"
Sep 19 01:33:06.324252 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:06.324227    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-8e708f/distinct-2-w0-8e708f-kserve-59b54cd666-tg2h7"
Sep 19 01:33:06.666728 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:06.666692    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-8e708f/distinct-2-w0-8e708f-kserve-59b54cd666-tg2h7"
Sep 19 01:33:13.641931 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:13.641828    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-751f6a-f5b56d759-n267t"
Sep 19 01:33:13.642280 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:13.642020    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-751f6a-8546c5d9f8-zw744"
Sep 19 01:33:18.601190 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:18.601158    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-b5fe4519/e2e-lifecycle-model-b5fe4519-kserve-5b4987998c-2kjwn"]
Sep 19 01:33:18.603385 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:18.603359    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-b5fe4519/e2e-lifecycle-model-b5fe4519-kserve-5b4987998c-2kjwn"
Sep 19 01:33:18.605486 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:33:18.605454    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"e2e-lifecycle-model-b5fe4519-kserve-self-signed-certs\" is forbidden: User \"system:node:ip-10-0-129-113.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"ai-tenant-e2e-mt-b5fe4519\": no relationship found between node 'ip-10-0-129-113.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"ai-tenant-e2e-mt-b5fe4519\"/\"e2e-lifecycle-model-b5fe4519-kserve-self-signed-certs\"" type="*v1.Secret"
Sep 19 01:33:18.605658 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:33:18.605461    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-129-113.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"ai-tenant-e2e-mt-b5fe4519\": no relationship found between node 'ip-10-0-129-113.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"ai-tenant-e2e-mt-b5fe4519\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap"
Sep 19 01:33:18.605658 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:33:18.605490    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"default-dockercfg-6c6gp\" is forbidden: User \"system:node:ip-10-0-129-113.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"ai-tenant-e2e-mt-b5fe4519\": no relationship found between node 'ip-10-0-129-113.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"ai-tenant-e2e-mt-b5fe4519\"/\"default-dockercfg-6c6gp\"" type="*v1.Secret"
Sep 19 01:33:18.605658 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:33:18.605516    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"e2e-lifecycle-model-b5fe4519-kserve-5b4987998c-2kjwn\" is forbidden: User \"system:node:ip-10-0-129-113.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"ai-tenant-e2e-mt-b5fe4519\": no relationship found between node 'ip-10-0-129-113.ec2.internal' and this object" podUID="6234610b-4569-4c7e-aabc-b955803abc90" pod="ai-tenant-e2e-mt-b5fe4519/e2e-lifecycle-model-b5fe4519-kserve-5b4987998c-2kjwn"
Sep 19 01:33:18.605834 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:33:18.605543    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-129-113.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"ai-tenant-e2e-mt-b5fe4519\": no relationship found between node 'ip-10-0-129-113.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"ai-tenant-e2e-mt-b5fe4519\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap"
Sep 19 01:33:18.628678 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:18.628658    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-b5fe4519/e2e-lifecycle-model-b5fe4519-kserve-5b4987998c-2kjwn"]
Sep 19 01:33:18.775240 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:18.775209    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq7c7\" (UniqueName: \"kubernetes.io/projected/6234610b-4569-4c7e-aabc-b955803abc90-kube-api-access-vq7c7\") pod \"e2e-lifecycle-model-b5fe4519-kserve-5b4987998c-2kjwn\" (UID: \"6234610b-4569-4c7e-aabc-b955803abc90\") " pod="ai-tenant-e2e-mt-b5fe4519/e2e-lifecycle-model-b5fe4519-kserve-5b4987998c-2kjwn"
Sep 19 01:33:18.776543 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:18.775259    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6234610b-4569-4c7e-aabc-b955803abc90-home\") pod \"e2e-lifecycle-model-b5fe4519-kserve-5b4987998c-2kjwn\" (UID: \"6234610b-4569-4c7e-aabc-b955803abc90\") " pod="ai-tenant-e2e-mt-b5fe4519/e2e-lifecycle-model-b5fe4519-kserve-5b4987998c-2kjwn"
Sep 19 01:33:18.776543 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:18.775329    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6234610b-4569-4c7e-aabc-b955803abc90-dshm\") pod \"e2e-lifecycle-model-b5fe4519-kserve-5b4987998c-2kjwn\" (UID: \"6234610b-4569-4c7e-aabc-b955803abc90\") " pod="ai-tenant-e2e-mt-b5fe4519/e2e-lifecycle-model-b5fe4519-kserve-5b4987998c-2kjwn"
Sep 19 01:33:18.776543 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:18.775430    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6234610b-4569-4c7e-aabc-b955803abc90-model-cache\") pod \"e2e-lifecycle-model-b5fe4519-kserve-5b4987998c-2kjwn\" (UID: \"6234610b-4569-4c7e-aabc-b955803abc90\") " pod="ai-tenant-e2e-mt-b5fe4519/e2e-lifecycle-model-b5fe4519-kserve-5b4987998c-2kjwn"
Sep 19 01:33:18.776543 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:18.775469    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6234610b-4569-4c7e-aabc-b955803abc90-tmp-dir\") pod \"e2e-lifecycle-model-b5fe4519-kserve-5b4987998c-2kjwn\" (UID: \"6234610b-4569-4c7e-aabc-b955803abc90\") " pod="ai-tenant-e2e-mt-b5fe4519/e2e-lifecycle-model-b5fe4519-kserve-5b4987998c-2kjwn"
Sep 19 01:33:18.776543 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:18.775544    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6234610b-4569-4c7e-aabc-b955803abc90-tls-certs\") pod \"e2e-lifecycle-model-b5fe4519-kserve-5b4987998c-2kjwn\" (UID: \"6234610b-4569-4c7e-aabc-b955803abc90\") " pod="ai-tenant-e2e-mt-b5fe4519/e2e-lifecycle-model-b5fe4519-kserve-5b4987998c-2kjwn"
Sep 19 01:33:18.876276 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:18.876228    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq7c7\" (UniqueName: \"kubernetes.io/projected/6234610b-4569-4c7e-aabc-b955803abc90-kube-api-access-vq7c7\") pod \"e2e-lifecycle-model-b5fe4519-kserve-5b4987998c-2kjwn\" (UID: \"6234610b-4569-4c7e-aabc-b955803abc90\") " pod="ai-tenant-e2e-mt-b5fe4519/e2e-lifecycle-model-b5fe4519-kserve-5b4987998c-2kjwn"
Sep 19 01:33:18.876276 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:18.876271    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6234610b-4569-4c7e-aabc-b955803abc90-home\") pod \"e2e-lifecycle-model-b5fe4519-kserve-5b4987998c-2kjwn\" (UID: \"6234610b-4569-4c7e-aabc-b955803abc90\") " pod="ai-tenant-e2e-mt-b5fe4519/e2e-lifecycle-model-b5fe4519-kserve-5b4987998c-2kjwn"
Sep 19 01:33:18.876442 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:18.876391    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6234610b-4569-4c7e-aabc-b955803abc90-dshm\") pod \"e2e-lifecycle-model-b5fe4519-kserve-5b4987998c-2kjwn\" (UID: \"6234610b-4569-4c7e-aabc-b955803abc90\") " pod="ai-tenant-e2e-mt-b5fe4519/e2e-lifecycle-model-b5fe4519-kserve-5b4987998c-2kjwn"
Sep 19 01:33:18.876498 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:18.876471    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6234610b-4569-4c7e-aabc-b955803abc90-model-cache\") pod \"e2e-lifecycle-model-b5fe4519-kserve-5b4987998c-2kjwn\" (UID: \"6234610b-4569-4c7e-aabc-b955803abc90\") " pod="ai-tenant-e2e-mt-b5fe4519/e2e-lifecycle-model-b5fe4519-kserve-5b4987998c-2kjwn"
Sep 19 01:33:18.876549 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:18.876501    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6234610b-4569-4c7e-aabc-b955803abc90-tmp-dir\") pod \"e2e-lifecycle-model-b5fe4519-kserve-5b4987998c-2kjwn\" (UID: \"6234610b-4569-4c7e-aabc-b955803abc90\") " pod="ai-tenant-e2e-mt-b5fe4519/e2e-lifecycle-model-b5fe4519-kserve-5b4987998c-2kjwn"
Sep 19 01:33:18.876603 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:18.876557    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6234610b-4569-4c7e-aabc-b955803abc90-tls-certs\") pod \"e2e-lifecycle-model-b5fe4519-kserve-5b4987998c-2kjwn\" (UID: \"6234610b-4569-4c7e-aabc-b955803abc90\") " pod="ai-tenant-e2e-mt-b5fe4519/e2e-lifecycle-model-b5fe4519-kserve-5b4987998c-2kjwn"
Sep 19 01:33:18.876603 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:18.876574    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6234610b-4569-4c7e-aabc-b955803abc90-home\") pod \"e2e-lifecycle-model-b5fe4519-kserve-5b4987998c-2kjwn\" (UID: \"6234610b-4569-4c7e-aabc-b955803abc90\") " pod="ai-tenant-e2e-mt-b5fe4519/e2e-lifecycle-model-b5fe4519-kserve-5b4987998c-2kjwn"
Sep 19 01:33:18.876781 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:18.876763    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6234610b-4569-4c7e-aabc-b955803abc90-model-cache\") pod \"e2e-lifecycle-model-b5fe4519-kserve-5b4987998c-2kjwn\" (UID: \"6234610b-4569-4c7e-aabc-b955803abc90\") " pod="ai-tenant-e2e-mt-b5fe4519/e2e-lifecycle-model-b5fe4519-kserve-5b4987998c-2kjwn"
Sep 19 01:33:18.876895 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:18.876855    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6234610b-4569-4c7e-aabc-b955803abc90-tmp-dir\") pod \"e2e-lifecycle-model-b5fe4519-kserve-5b4987998c-2kjwn\" (UID: \"6234610b-4569-4c7e-aabc-b955803abc90\") " pod="ai-tenant-e2e-mt-b5fe4519/e2e-lifecycle-model-b5fe4519-kserve-5b4987998c-2kjwn"
Sep 19 01:33:18.878540 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:18.878521    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6234610b-4569-4c7e-aabc-b955803abc90-dshm\") pod \"e2e-lifecycle-model-b5fe4519-kserve-5b4987998c-2kjwn\" (UID: \"6234610b-4569-4c7e-aabc-b955803abc90\") " pod="ai-tenant-e2e-mt-b5fe4519/e2e-lifecycle-model-b5fe4519-kserve-5b4987998c-2kjwn"
Sep 19 01:33:19.539773 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:19.539743    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-b5fe4519\"/\"kube-root-ca.crt\""
Sep 19 01:33:19.734325 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:19.734294    2581 generic.go:356] "Generic (PLEG): container finished" podID="89c7c2d8-487c-4c80-a630-dc0ffc760475" containerID="30a726cf43deddaf51c8914f3a6e84104c511c46ccd290b406d5491592e2aa3e" exitCode=0
Sep 19 01:33:19.734682 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:19.734330    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-751f6a-openshift-default-549c5bdf4d-78c97" event={"ID":"89c7c2d8-487c-4c80-a630-dc0ffc760475","Type":"ContainerDied","Data":"30a726cf43deddaf51c8914f3a6e84104c511c46ccd290b406d5491592e2aa3e"}
Sep 19 01:33:19.734682 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:19.734666    2581 scope.go:117] "RemoveContainer" containerID="30a726cf43deddaf51c8914f3a6e84104c511c46ccd290b406d5491592e2aa3e"
Sep 19 01:33:19.735703 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:19.735673    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 19 01:33:19.735796 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:19.735741    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 19 01:33:19.735796 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:19.735773    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 19 01:33:19.877579 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:33:19.877544    2581 secret.go:189] Couldn't get secret ai-tenant-e2e-mt-b5fe4519/e2e-lifecycle-model-b5fe4519-kserve-self-signed-certs: failed to sync secret cache: timed out waiting for the condition
Sep 19 01:33:19.877730 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:33:19.877647    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6234610b-4569-4c7e-aabc-b955803abc90-tls-certs podName:6234610b-4569-4c7e-aabc-b955803abc90 nodeName:}" failed. No retries permitted until 2026-09-19 01:33:20.377625547 +0000 UTC m=+1251.265294623 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/6234610b-4569-4c7e-aabc-b955803abc90-tls-certs") pod "e2e-lifecycle-model-b5fe4519-kserve-5b4987998c-2kjwn" (UID: "6234610b-4569-4c7e-aabc-b955803abc90") : failed to sync secret cache: timed out waiting for the condition
Sep 19 01:33:20.073906 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:20.073807    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-b5fe4519\"/\"default-dockercfg-6c6gp\""
Sep 19 01:33:20.113045 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:20.113015    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-b5fe4519\"/\"openshift-service-ca.crt\""
Sep 19 01:33:20.120634 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:20.120611    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq7c7\" (UniqueName: \"kubernetes.io/projected/6234610b-4569-4c7e-aabc-b955803abc90-kube-api-access-vq7c7\") pod \"e2e-lifecycle-model-b5fe4519-kserve-5b4987998c-2kjwn\" (UID: \"6234610b-4569-4c7e-aabc-b955803abc90\") " pod="ai-tenant-e2e-mt-b5fe4519/e2e-lifecycle-model-b5fe4519-kserve-5b4987998c-2kjwn"
Sep 19 01:33:20.166900 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:20.166852    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-b5fe4519\"/\"e2e-lifecycle-model-b5fe4519-kserve-self-signed-certs\""
Sep 19 01:33:20.320311 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:20.320280    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c9nsz"]
Sep 19 01:33:20.389555 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:20.389493    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6234610b-4569-4c7e-aabc-b955803abc90-tls-certs\") pod \"e2e-lifecycle-model-b5fe4519-kserve-5b4987998c-2kjwn\" (UID: \"6234610b-4569-4c7e-aabc-b955803abc90\") " pod="ai-tenant-e2e-mt-b5fe4519/e2e-lifecycle-model-b5fe4519-kserve-5b4987998c-2kjwn"
Sep 19 01:33:20.394401 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:20.394369    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6234610b-4569-4c7e-aabc-b955803abc90-tls-certs\") pod \"e2e-lifecycle-model-b5fe4519-kserve-5b4987998c-2kjwn\" (UID: \"6234610b-4569-4c7e-aabc-b955803abc90\") " pod="ai-tenant-e2e-mt-b5fe4519/e2e-lifecycle-model-b5fe4519-kserve-5b4987998c-2kjwn"
Sep 19 01:33:20.412440 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:20.412402    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-b5fe4519/e2e-lifecycle-model-b5fe4519-kserve-5b4987998c-2kjwn"
Sep 19 01:33:20.719257 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:20.719229    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-b5fe4519/e2e-lifecycle-model-b5fe4519-kserve-5b4987998c-2kjwn"]
Sep 19 01:33:20.721924 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:33:20.721900    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6234610b_4569_4c7e_aabc_b955803abc90.slice/crio-5994d52f57075a6e8ea46268c2f2ddd4a5e6a113930fba599973a66bbc87f5b6 WatchSource:0}: Error finding container 5994d52f57075a6e8ea46268c2f2ddd4a5e6a113930fba599973a66bbc87f5b6: Status 404 returned error can't find the container with id 5994d52f57075a6e8ea46268c2f2ddd4a5e6a113930fba599973a66bbc87f5b6
Sep 19 01:33:20.742799 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:20.742762    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-751f6a-openshift-default-549c5bdf4d-78c97" event={"ID":"89c7c2d8-487c-4c80-a630-dc0ffc760475","Type":"ContainerStarted","Data":"5de733c4c3746e2d9e64c623588af7cee5806b96a4c7be8e8f14542c6e0fa3df"}
Sep 19 01:33:20.744143 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:20.744082    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-b5fe4519/e2e-lifecycle-model-b5fe4519-kserve-5b4987998c-2kjwn" event={"ID":"6234610b-4569-4c7e-aabc-b955803abc90","Type":"ContainerStarted","Data":"5994d52f57075a6e8ea46268c2f2ddd4a5e6a113930fba599973a66bbc87f5b6"}
Sep 19 01:33:21.066833 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:21.066730    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-751f6a-openshift-default-549c5bdf4d-78c97"
Sep 19 01:33:21.067798 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:21.067771    2581 patch_prober.go:28] interesting pod/e2e-shared-a-w3-751f6a-openshift-default-549c5bdf4d-78c97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Sep 19 01:33:21.067954 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:21.067822    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-751f6a-openshift-default-549c5bdf4d-78c97" podUID="89c7c2d8-487c-4c80-a630-dc0ffc760475" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Sep 19 01:33:21.750821 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:21.750788    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-b5fe4519/e2e-lifecycle-model-b5fe4519-kserve-5b4987998c-2kjwn" event={"ID":"6234610b-4569-4c7e-aabc-b955803abc90","Type":"ContainerStarted","Data":"d02ed3611acf232be0212790e701aded7e591e3217a3d073d2d4427469a55c60"}
Sep 19 01:33:21.775532 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:21.775485    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-b5fe4519/e2e-lifecycle-model-b5fe4519-kserve-5b4987998c-2kjwn" podStartSLOduration=3.775471101 podStartE2EDuration="3.775471101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 01:33:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 01:33:21.772716328 +0000 UTC m=+1252.660385417" watchObservedRunningTime="2026-09-19 01:33:21.775471101 +0000 UTC m=+1252.663140179"
Sep 19 01:33:22.066598 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:22.066516    2581 patch_prober.go:28] interesting pod/e2e-shared-a-w3-751f6a-openshift-default-549c5bdf4d-78c97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Sep 19 01:33:22.066598 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:22.066568    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-751f6a-openshift-default-549c5bdf4d-78c97" podUID="89c7c2d8-487c-4c80-a630-dc0ffc760475" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Sep 19 01:33:23.066430 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:23.066398    2581 patch_prober.go:28] interesting pod/e2e-shared-a-w3-751f6a-openshift-default-549c5bdf4d-78c97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Sep 19 01:33:23.066803 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:23.066450    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-751f6a-openshift-default-549c5bdf4d-78c97" podUID="89c7c2d8-487c-4c80-a630-dc0ffc760475" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Sep 19 01:33:24.067185 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:24.067142    2581 patch_prober.go:28] interesting pod/e2e-shared-a-w3-751f6a-openshift-default-549c5bdf4d-78c97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Sep 19 01:33:24.067656 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:24.067206    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-751f6a-openshift-default-549c5bdf4d-78c97" podUID="89c7c2d8-487c-4c80-a630-dc0ffc760475" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Sep 19 01:33:24.573242 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:24.573210    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-751f6a-openshift-default-549c5bdf4d-78c97"
Sep 19 01:33:25.066700 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:25.066665    2581 patch_prober.go:28] interesting pod/e2e-shared-a-w3-751f6a-openshift-default-549c5bdf4d-78c97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Sep 19 01:33:25.066886 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:25.066723    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-751f6a-openshift-default-549c5bdf4d-78c97" podUID="89c7c2d8-487c-4c80-a630-dc0ffc760475" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Sep 19 01:33:26.088559 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:26.088525    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-751f6a-openshift-default-549c5bdf4d-78c97"
Sep 19 01:33:26.089980 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:26.089959    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-751f6a-openshift-default-549c5bdf4d-78c97"
Sep 19 01:33:30.412787 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:30.412752    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-b5fe4519/e2e-lifecycle-model-b5fe4519-kserve-5b4987998c-2kjwn"
Sep 19 01:33:30.413221 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:30.412803    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-b5fe4519/e2e-lifecycle-model-b5fe4519-kserve-5b4987998c-2kjwn"
Sep 19 01:33:30.425299 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:30.425270    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-b5fe4519/e2e-lifecycle-model-b5fe4519-kserve-5b4987998c-2kjwn"
Sep 19 01:33:30.809513 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:30.809419    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-b5fe4519/e2e-lifecycle-model-b5fe4519-kserve-5b4987998c-2kjwn"
Sep 19 01:33:31.339243 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:31.339211    2581 scope.go:117] "RemoveContainer" containerID="f85ef3e5a3717f6d76bbb9b2a0f146cba82e66b6cdcdba81612a604e0cd70fc1"
Sep 19 01:33:31.347455 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:31.347433    2581 scope.go:117] "RemoveContainer" containerID="686e541a9465d8245ef26bda49170aea5a79b70c7c8a63946c36d0a4779fdb32"
Sep 19 01:33:33.336427 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:33.336397    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-bebef4-6f75dbf44f-s242z"]
Sep 19 01:33:33.339991 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:33.339974    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bebef4-6f75dbf44f-s242z"
Sep 19 01:33:33.342711 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:33.342694    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-bebef4\""
Sep 19 01:33:33.343391 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:33.343366    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-bebef4-dockercfg-s4ppb\""
Sep 19 01:33:33.351216 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:33.351195    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-bebef4-6f75dbf44f-s242z"]
Sep 19 01:33:33.406628 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:33.406587    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kksdl\" (UniqueName: \"kubernetes.io/projected/bc8ef285-2554-4f8d-a4b4-cb76af7ccec8-kube-api-access-kksdl\") pod \"payload-processing-e2e-shared-b-w3-bebef4-6f75dbf44f-s242z\" (UID: \"bc8ef285-2554-4f8d-a4b4-cb76af7ccec8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bebef4-6f75dbf44f-s242z"
Sep 19 01:33:33.406805 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:33.406705    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc8ef285-2554-4f8d-a4b4-cb76af7ccec8-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-bebef4-6f75dbf44f-s242z\" (UID: \"bc8ef285-2554-4f8d-a4b4-cb76af7ccec8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bebef4-6f75dbf44f-s242z"
Sep 19 01:33:33.507491 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:33.507454    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kksdl\" (UniqueName: \"kubernetes.io/projected/bc8ef285-2554-4f8d-a4b4-cb76af7ccec8-kube-api-access-kksdl\") pod \"payload-processing-e2e-shared-b-w3-bebef4-6f75dbf44f-s242z\" (UID: \"bc8ef285-2554-4f8d-a4b4-cb76af7ccec8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bebef4-6f75dbf44f-s242z"
Sep 19 01:33:33.507688 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:33.507526    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc8ef285-2554-4f8d-a4b4-cb76af7ccec8-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-bebef4-6f75dbf44f-s242z\" (UID: \"bc8ef285-2554-4f8d-a4b4-cb76af7ccec8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bebef4-6f75dbf44f-s242z"
Sep 19 01:33:33.508178 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:33.508159    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc8ef285-2554-4f8d-a4b4-cb76af7ccec8-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-bebef4-6f75dbf44f-s242z\" (UID: \"bc8ef285-2554-4f8d-a4b4-cb76af7ccec8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bebef4-6f75dbf44f-s242z"
Sep 19 01:33:33.516580 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:33.516555    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kksdl\" (UniqueName: \"kubernetes.io/projected/bc8ef285-2554-4f8d-a4b4-cb76af7ccec8-kube-api-access-kksdl\") pod \"payload-processing-e2e-shared-b-w3-bebef4-6f75dbf44f-s242z\" (UID: \"bc8ef285-2554-4f8d-a4b4-cb76af7ccec8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bebef4-6f75dbf44f-s242z"
Sep 19 01:33:33.650509 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:33.650484    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bebef4-6f75dbf44f-s242z"
Sep 19 01:33:33.924011 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:33.923987    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-bebef4-6f75dbf44f-s242z"]
Sep 19 01:33:33.926026 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:33:33.926000    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc8ef285_2554_4f8d_a4b4_cb76af7ccec8.slice/crio-60a4d279fba2187d693c90ff41cd7e7b9cb4e1c15e6dcc8c0848aee4fc46e063 WatchSource:0}: Error finding container 60a4d279fba2187d693c90ff41cd7e7b9cb4e1c15e6dcc8c0848aee4fc46e063: Status 404 returned error can't find the container with id 60a4d279fba2187d693c90ff41cd7e7b9cb4e1c15e6dcc8c0848aee4fc46e063
Sep 19 01:33:34.821513 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:34.821478    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bebef4-6f75dbf44f-s242z" event={"ID":"bc8ef285-2554-4f8d-a4b4-cb76af7ccec8","Type":"ContainerStarted","Data":"ddf8559f9210b71fbe3c611e8fd742a59b8e13efc98b6092428da98c7c7d3bb9"}
Sep 19 01:33:34.822088 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:34.821536    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bebef4-6f75dbf44f-s242z" event={"ID":"bc8ef285-2554-4f8d-a4b4-cb76af7ccec8","Type":"ContainerStarted","Data":"60a4d279fba2187d693c90ff41cd7e7b9cb4e1c15e6dcc8c0848aee4fc46e063"}
Sep 19 01:33:34.822088 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:34.821573    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bebef4-6f75dbf44f-s242z"
Sep 19 01:33:34.839595 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:34.839543    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bebef4-6f75dbf44f-s242z" podStartSLOduration=1.839524667 podStartE2EDuration="1.839524667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 01:33:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 01:33:34.839287758 +0000 UTC m=+1265.726956863" watchObservedRunningTime="2026-09-19 01:33:34.839524667 +0000 UTC m=+1265.727193745"
Sep 19 01:33:36.702844 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:36.702812    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c9nsz"]
Sep 19 01:33:45.830307 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:45.830274    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bebef4-6f75dbf44f-s242z"
Sep 19 01:33:56.323350 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:33:56.323309    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c9nsz"]
Sep 19 01:34:01.157576 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:34:01.157544    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b5fe4519-5bgbm"]
Sep 19 01:34:01.159953 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:34:01.159936    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b5fe4519-5bgbm"
Sep 19 01:34:01.162451 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:34:01.162432    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 19 01:34:01.162603 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:34:01.162584    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s7pvt\""
Sep 19 01:34:01.170409 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:34:01.170382    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b5fe4519-5bgbm"]
Sep 19 01:34:01.240423 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:34:01.240398    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccc4411e-7cea-4f5e-93e2-3baffa34e2ff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-b5fe4519-5bgbm\" (UID: \"ccc4411e-7cea-4f5e-93e2-3baffa34e2ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b5fe4519-5bgbm"
Sep 19 01:34:01.341387 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:34:01.341359    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccc4411e-7cea-4f5e-93e2-3baffa34e2ff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-b5fe4519-5bgbm\" (UID: \"ccc4411e-7cea-4f5e-93e2-3baffa34e2ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b5fe4519-5bgbm"
Sep 19 01:34:01.342018 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:34:01.342002    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccc4411e-7cea-4f5e-93e2-3baffa34e2ff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-b5fe4519-5bgbm\" (UID: \"ccc4411e-7cea-4f5e-93e2-3baffa34e2ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b5fe4519-5bgbm"
Sep 19 01:34:01.470407 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:34:01.470346    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b5fe4519-5bgbm"
Sep 19 01:34:01.741378 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:34:01.741303    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b5fe4519-5bgbm"]
Sep 19 01:34:01.744494 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:34:01.744464    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccc4411e_7cea_4f5e_93e2_3baffa34e2ff.slice/crio-a20b372ffb7e7dab2d3f7d67c918f97265ec052310ab26294f5e301207cd5b47 WatchSource:0}: Error finding container a20b372ffb7e7dab2d3f7d67c918f97265ec052310ab26294f5e301207cd5b47: Status 404 returned error can't find the container with id a20b372ffb7e7dab2d3f7d67c918f97265ec052310ab26294f5e301207cd5b47
Sep 19 01:34:01.961812 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:34:01.961775    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b5fe4519-5bgbm" event={"ID":"ccc4411e-7cea-4f5e-93e2-3baffa34e2ff","Type":"ContainerStarted","Data":"a20b372ffb7e7dab2d3f7d67c918f97265ec052310ab26294f5e301207cd5b47"}
Sep 19 01:34:02.968594 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:34:02.968561    2581 generic.go:356] "Generic (PLEG): container finished" podID="ccc4411e-7cea-4f5e-93e2-3baffa34e2ff" containerID="562a61cccc510877b523ab935b17c05760393a428974c83e1693df8e4f831ac6" exitCode=0
Sep 19 01:34:02.969048 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:34:02.968603    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b5fe4519-5bgbm" event={"ID":"ccc4411e-7cea-4f5e-93e2-3baffa34e2ff","Type":"ContainerDied","Data":"562a61cccc510877b523ab935b17c05760393a428974c83e1693df8e4f831ac6"}
Sep 19 01:34:04.103778 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:34:04.103752    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b5fe4519-5bgbm"
Sep 19 01:34:04.265258 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:34:04.265194    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccc4411e-7cea-4f5e-93e2-3baffa34e2ff-maas-api-service-ca\") pod \"ccc4411e-7cea-4f5e-93e2-3baffa34e2ff\" (UID: \"ccc4411e-7cea-4f5e-93e2-3baffa34e2ff\") "
Sep 19 01:34:04.265504 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:34:04.265483    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccc4411e-7cea-4f5e-93e2-3baffa34e2ff-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ccc4411e-7cea-4f5e-93e2-3baffa34e2ff" (UID: "ccc4411e-7cea-4f5e-93e2-3baffa34e2ff"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 01:34:04.366702 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:34:04.366673    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccc4411e-7cea-4f5e-93e2-3baffa34e2ff-maas-api-service-ca\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:34:04.980124 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:34:04.980089    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b5fe4519-5bgbm"
Sep 19 01:34:04.980124 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:34:04.980117    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b5fe4519-5bgbm" event={"ID":"ccc4411e-7cea-4f5e-93e2-3baffa34e2ff","Type":"ContainerDied","Data":"a20b372ffb7e7dab2d3f7d67c918f97265ec052310ab26294f5e301207cd5b47"}
Sep 19 01:34:04.980375 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:34:04.980170    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a20b372ffb7e7dab2d3f7d67c918f97265ec052310ab26294f5e301207cd5b47"
Sep 19 01:34:17.011150 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:34:17.011117    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b5fe4519-b4d97c6f8-n7pdx"]
Sep 19 01:34:17.042439 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:34:17.042394    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-b5fe4519-b4d97c6f8-n7pdx" podUID="271c7dde-6e52-47ab-ba47-7c8aaac9c549" containerName="payload-pre-processing" containerID="cri-o://563793a68b90da1b8fccd0c9cc081547e65437e6b2f0562ed9c68fba3bdc74c7" gracePeriod=30
Sep 19 01:34:17.300218 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:34:17.300194    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b5fe4519-b4d97c6f8-n7pdx"
Sep 19 01:34:17.395048 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:34:17.395014    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b5fe4519-5bgbm"]
Sep 19 01:34:17.400205 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:34:17.400182    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b5fe4519-5bgbm"]
Sep 19 01:34:17.474319 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:34:17.474298    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/271c7dde-6e52-47ab-ba47-7c8aaac9c549-plugins-config-volume\") pod \"271c7dde-6e52-47ab-ba47-7c8aaac9c549\" (UID: \"271c7dde-6e52-47ab-ba47-7c8aaac9c549\") "
Sep 19 01:34:17.474440 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:34:17.474381    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zltlg\" (UniqueName: \"kubernetes.io/projected/271c7dde-6e52-47ab-ba47-7c8aaac9c549-kube-api-access-zltlg\") pod \"271c7dde-6e52-47ab-ba47-7c8aaac9c549\" (UID: \"271c7dde-6e52-47ab-ba47-7c8aaac9c549\") "
Sep 19 01:34:17.474623 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:34:17.474604    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/271c7dde-6e52-47ab-ba47-7c8aaac9c549-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "271c7dde-6e52-47ab-ba47-7c8aaac9c549" (UID: "271c7dde-6e52-47ab-ba47-7c8aaac9c549"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 01:34:17.476596 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:34:17.476571    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/271c7dde-6e52-47ab-ba47-7c8aaac9c549-kube-api-access-zltlg" (OuterVolumeSpecName: "kube-api-access-zltlg") pod "271c7dde-6e52-47ab-ba47-7c8aaac9c549" (UID: "271c7dde-6e52-47ab-ba47-7c8aaac9c549"). InnerVolumeSpecName "kube-api-access-zltlg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 01:34:17.574944 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:34:17.574887    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zltlg\" (UniqueName: \"kubernetes.io/projected/271c7dde-6e52-47ab-ba47-7c8aaac9c549-kube-api-access-zltlg\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:34:17.574944 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:34:17.574909    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/271c7dde-6e52-47ab-ba47-7c8aaac9c549-plugins-config-volume\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:34:18.048308 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:34:18.048270    2581 generic.go:356] "Generic (PLEG): container finished" podID="271c7dde-6e52-47ab-ba47-7c8aaac9c549" containerID="563793a68b90da1b8fccd0c9cc081547e65437e6b2f0562ed9c68fba3bdc74c7" exitCode=0
Sep 19 01:34:18.048738 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:34:18.048320    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b5fe4519-b4d97c6f8-n7pdx"
Sep 19 01:34:18.048738 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:34:18.048359    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b5fe4519-b4d97c6f8-n7pdx" event={"ID":"271c7dde-6e52-47ab-ba47-7c8aaac9c549","Type":"ContainerDied","Data":"563793a68b90da1b8fccd0c9cc081547e65437e6b2f0562ed9c68fba3bdc74c7"}
Sep 19 01:34:18.048738 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:34:18.048410    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b5fe4519-b4d97c6f8-n7pdx" event={"ID":"271c7dde-6e52-47ab-ba47-7c8aaac9c549","Type":"ContainerDied","Data":"7874432e5ddcc65d35dfb56221dc4a15fd92794d765fd94075bb3bf57d194b53"}
Sep 19 01:34:18.048738 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:34:18.048434    2581 scope.go:117] "RemoveContainer" containerID="563793a68b90da1b8fccd0c9cc081547e65437e6b2f0562ed9c68fba3bdc74c7"
Sep 19 01:34:18.057373 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:34:18.057353    2581 scope.go:117] "RemoveContainer" containerID="563793a68b90da1b8fccd0c9cc081547e65437e6b2f0562ed9c68fba3bdc74c7"
Sep 19 01:34:18.057622 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:34:18.057607    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"563793a68b90da1b8fccd0c9cc081547e65437e6b2f0562ed9c68fba3bdc74c7\": container with ID starting with 563793a68b90da1b8fccd0c9cc081547e65437e6b2f0562ed9c68fba3bdc74c7 not found: ID does not exist" containerID="563793a68b90da1b8fccd0c9cc081547e65437e6b2f0562ed9c68fba3bdc74c7"
Sep 19 01:34:18.057666 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:34:18.057629    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"563793a68b90da1b8fccd0c9cc081547e65437e6b2f0562ed9c68fba3bdc74c7"} err="failed to get container status \"563793a68b90da1b8fccd0c9cc081547e65437e6b2f0562ed9c68fba3bdc74c7\": rpc error: code = NotFound desc = could not find container \"563793a68b90da1b8fccd0c9cc081547e65437e6b2f0562ed9c68fba3bdc74c7\": container with ID starting with 563793a68b90da1b8fccd0c9cc081547e65437e6b2f0562ed9c68fba3bdc74c7 not found: ID does not exist"
Sep 19 01:34:18.094426 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:34:18.094402    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b5fe4519-b4d97c6f8-n7pdx"]
Sep 19 01:34:18.110327 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:34:18.110304    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b5fe4519-b4d97c6f8-n7pdx"]
Sep 19 01:34:19.272196 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:34:19.272167    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="271c7dde-6e52-47ab-ba47-7c8aaac9c549" path="/var/lib/kubelet/pods/271c7dde-6e52-47ab-ba47-7c8aaac9c549/volumes"
Sep 19 01:34:19.272553 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:34:19.272512    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ccc4411e-7cea-4f5e-93e2-3baffa34e2ff" path="/var/lib/kubelet/pods/ccc4411e-7cea-4f5e-93e2-3baffa34e2ff/volumes"
Sep 19 01:34:21.757219 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:34:21.757184    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c9nsz"]
Sep 19 01:34:25.994549 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:34:25.994509    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-b5fe4519/e2e-lifecycle-model-b5fe4519-kserve-5b4987998c-2kjwn"]
Sep 19 01:34:31.376483 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:34:31.376452    2581 scope.go:117] "RemoveContainer" containerID="4dab58502dc990e25489c186901be18f0d00dadce0d01b65ef98517f873aec2a"
Sep 19 01:34:31.385295 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:34:31.385274    2581 scope.go:117] "RemoveContainer" containerID="18d8479030ec325675d61010119209028edc2df02040a304effc386aae6e1142"
Sep 19 01:34:41.014046 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:34:41.013959    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-b5fe4519/e2e-lifecycle-model-b5fe4519-kserve-5b4987998c-2kjwn" podUID="6234610b-4569-4c7e-aabc-b955803abc90" containerName="main" containerID="cri-o://d02ed3611acf232be0212790e701aded7e591e3217a3d073d2d4427469a55c60" gracePeriod=45
Sep 19 01:34:41.171000 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:34:41.170947    2581 generic.go:356] "Generic (PLEG): container finished" podID="6234610b-4569-4c7e-aabc-b955803abc90" containerID="d02ed3611acf232be0212790e701aded7e591e3217a3d073d2d4427469a55c60" exitCode=0
Sep 19 01:34:41.171144 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:34:41.171002    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-b5fe4519/e2e-lifecycle-model-b5fe4519-kserve-5b4987998c-2kjwn" event={"ID":"6234610b-4569-4c7e-aabc-b955803abc90","Type":"ContainerDied","Data":"d02ed3611acf232be0212790e701aded7e591e3217a3d073d2d4427469a55c60"}
Sep 19 01:34:41.264512 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:34:41.264460    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-b5fe4519/e2e-lifecycle-model-b5fe4519-kserve-5b4987998c-2kjwn"
Sep 19 01:34:41.366020 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:34:41.365990    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6234610b-4569-4c7e-aabc-b955803abc90-home\") pod \"6234610b-4569-4c7e-aabc-b955803abc90\" (UID: \"6234610b-4569-4c7e-aabc-b955803abc90\") "
Sep 19 01:34:41.366020 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:34:41.366025    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6234610b-4569-4c7e-aabc-b955803abc90-model-cache\") pod \"6234610b-4569-4c7e-aabc-b955803abc90\" (UID: \"6234610b-4569-4c7e-aabc-b955803abc90\") "
Sep 19 01:34:41.366242 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:34:41.366048    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6234610b-4569-4c7e-aabc-b955803abc90-tmp-dir\") pod \"6234610b-4569-4c7e-aabc-b955803abc90\" (UID: \"6234610b-4569-4c7e-aabc-b955803abc90\") "
Sep 19 01:34:41.366242 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:34:41.366086    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6234610b-4569-4c7e-aabc-b955803abc90-tls-certs\") pod \"6234610b-4569-4c7e-aabc-b955803abc90\" (UID: \"6234610b-4569-4c7e-aabc-b955803abc90\") "
Sep 19 01:34:41.366242 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:34:41.366133    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6234610b-4569-4c7e-aabc-b955803abc90-dshm\") pod \"6234610b-4569-4c7e-aabc-b955803abc90\" (UID: \"6234610b-4569-4c7e-aabc-b955803abc90\") "
Sep 19 01:34:41.366415 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:34:41.366253    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vq7c7\" (UniqueName: \"kubernetes.io/projected/6234610b-4569-4c7e-aabc-b955803abc90-kube-api-access-vq7c7\") pod \"6234610b-4569-4c7e-aabc-b955803abc90\" (UID: \"6234610b-4569-4c7e-aabc-b955803abc90\") "
Sep 19 01:34:41.366415 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:34:41.366266    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6234610b-4569-4c7e-aabc-b955803abc90-model-cache" (OuterVolumeSpecName: "model-cache") pod "6234610b-4569-4c7e-aabc-b955803abc90" (UID: "6234610b-4569-4c7e-aabc-b955803abc90"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 01:34:41.366415 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:34:41.366293    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6234610b-4569-4c7e-aabc-b955803abc90-home" (OuterVolumeSpecName: "home") pod "6234610b-4569-4c7e-aabc-b955803abc90" (UID: "6234610b-4569-4c7e-aabc-b955803abc90"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 01:34:41.366415 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:34:41.366309    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6234610b-4569-4c7e-aabc-b955803abc90-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6234610b-4569-4c7e-aabc-b955803abc90" (UID: "6234610b-4569-4c7e-aabc-b955803abc90"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 01:34:41.366639 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:34:41.366524    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6234610b-4569-4c7e-aabc-b955803abc90-home\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:34:41.366639 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:34:41.366542    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6234610b-4569-4c7e-aabc-b955803abc90-model-cache\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:34:41.366639 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:34:41.366557    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6234610b-4569-4c7e-aabc-b955803abc90-tmp-dir\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:34:41.368732 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:34:41.368704    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6234610b-4569-4c7e-aabc-b955803abc90-dshm" (OuterVolumeSpecName: "dshm") pod "6234610b-4569-4c7e-aabc-b955803abc90" (UID: "6234610b-4569-4c7e-aabc-b955803abc90"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 01:34:41.368892 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:34:41.368876    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6234610b-4569-4c7e-aabc-b955803abc90-kube-api-access-vq7c7" (OuterVolumeSpecName: "kube-api-access-vq7c7") pod "6234610b-4569-4c7e-aabc-b955803abc90" (UID: "6234610b-4569-4c7e-aabc-b955803abc90"). InnerVolumeSpecName "kube-api-access-vq7c7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 01:34:41.368892 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:34:41.368874    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6234610b-4569-4c7e-aabc-b955803abc90-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6234610b-4569-4c7e-aabc-b955803abc90" (UID: "6234610b-4569-4c7e-aabc-b955803abc90"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 01:34:41.467541 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:34:41.467512    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6234610b-4569-4c7e-aabc-b955803abc90-tls-certs\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:34:41.467541 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:34:41.467534    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6234610b-4569-4c7e-aabc-b955803abc90-dshm\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:34:41.467541 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:34:41.467543    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vq7c7\" (UniqueName: \"kubernetes.io/projected/6234610b-4569-4c7e-aabc-b955803abc90-kube-api-access-vq7c7\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:34:42.177000 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:34:42.176976    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-b5fe4519/e2e-lifecycle-model-b5fe4519-kserve-5b4987998c-2kjwn"
Sep 19 01:34:42.177465 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:34:42.177013    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-b5fe4519/e2e-lifecycle-model-b5fe4519-kserve-5b4987998c-2kjwn" event={"ID":"6234610b-4569-4c7e-aabc-b955803abc90","Type":"ContainerDied","Data":"5994d52f57075a6e8ea46268c2f2ddd4a5e6a113930fba599973a66bbc87f5b6"}
Sep 19 01:34:42.177465 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:34:42.177069    2581 scope.go:117] "RemoveContainer" containerID="d02ed3611acf232be0212790e701aded7e591e3217a3d073d2d4427469a55c60"
Sep 19 01:34:42.199218 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:34:42.199194    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-b5fe4519/e2e-lifecycle-model-b5fe4519-kserve-5b4987998c-2kjwn"]
Sep 19 01:34:42.206800 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:34:42.206775    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-b5fe4519/e2e-lifecycle-model-b5fe4519-kserve-5b4987998c-2kjwn"]
Sep 19 01:34:43.271725 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:34:43.271695    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6234610b-4569-4c7e-aabc-b955803abc90" path="/var/lib/kubelet/pods/6234610b-4569-4c7e-aabc-b955803abc90/volumes"
Sep 19 01:34:45.595973 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:34:45.595938    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c9nsz"]
Sep 19 01:35:12.997302 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:35:12.997269    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-8e708f/embedding-w0-8e708f-kserve-c6b4956b6-bkg7s"]
Sep 19 01:35:12.997699 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:35:12.997619    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="271c7dde-6e52-47ab-ba47-7c8aaac9c549" containerName="payload-pre-processing"
Sep 19 01:35:12.997699 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:35:12.997630    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="271c7dde-6e52-47ab-ba47-7c8aaac9c549" containerName="payload-pre-processing"
Sep 19 01:35:12.997699 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:35:12.997638    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6234610b-4569-4c7e-aabc-b955803abc90" containerName="main"
Sep 19 01:35:12.997699 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:35:12.997643    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6234610b-4569-4c7e-aabc-b955803abc90" containerName="main"
Sep 19 01:35:12.997699 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:35:12.997653    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccc4411e-7cea-4f5e-93e2-3baffa34e2ff" containerName="revoke-keys"
Sep 19 01:35:12.997699 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:35:12.997659    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccc4411e-7cea-4f5e-93e2-3baffa34e2ff" containerName="revoke-keys"
Sep 19 01:35:12.997913 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:35:12.997718    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="271c7dde-6e52-47ab-ba47-7c8aaac9c549" containerName="payload-pre-processing"
Sep 19 01:35:12.997913 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:35:12.997727    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6234610b-4569-4c7e-aabc-b955803abc90" containerName="main"
Sep 19 01:35:12.997913 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:35:12.997745    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ccc4411e-7cea-4f5e-93e2-3baffa34e2ff" containerName="revoke-keys"
Sep 19 01:35:12.999670 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:35:12.999655    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-8e708f/embedding-w0-8e708f-kserve-c6b4956b6-bkg7s"
Sep 19 01:35:13.001961 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:35:13.001940    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-8e708f\"/\"embedding-w0-8e708f-kserve-self-signed-certs\""
Sep 19 01:35:13.015838 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:35:13.015817    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-8e708f/embedding-w0-8e708f-kserve-c6b4956b6-bkg7s"]
Sep 19 01:35:13.128683 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:35:13.128655    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dr86l\" (UniqueName: \"kubernetes.io/projected/aaed025d-9218-4949-b6b3-80c4f2a16699-kube-api-access-dr86l\") pod \"embedding-w0-8e708f-kserve-c6b4956b6-bkg7s\" (UID: \"aaed025d-9218-4949-b6b3-80c4f2a16699\") " pod="e2e-models-e2e-worker-w0-8e708f/embedding-w0-8e708f-kserve-c6b4956b6-bkg7s"
Sep 19 01:35:13.128791 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:35:13.128685    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aaed025d-9218-4949-b6b3-80c4f2a16699-dshm\") pod \"embedding-w0-8e708f-kserve-c6b4956b6-bkg7s\" (UID: \"aaed025d-9218-4949-b6b3-80c4f2a16699\") " pod="e2e-models-e2e-worker-w0-8e708f/embedding-w0-8e708f-kserve-c6b4956b6-bkg7s"
Sep 19 01:35:13.128791 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:35:13.128712    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aaed025d-9218-4949-b6b3-80c4f2a16699-model-cache\") pod \"embedding-w0-8e708f-kserve-c6b4956b6-bkg7s\" (UID: \"aaed025d-9218-4949-b6b3-80c4f2a16699\") " pod="e2e-models-e2e-worker-w0-8e708f/embedding-w0-8e708f-kserve-c6b4956b6-bkg7s"
Sep 19 01:35:13.128791 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:35:13.128785    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aaed025d-9218-4949-b6b3-80c4f2a16699-tmp-dir\") pod \"embedding-w0-8e708f-kserve-c6b4956b6-bkg7s\" (UID: \"aaed025d-9218-4949-b6b3-80c4f2a16699\") " pod="e2e-models-e2e-worker-w0-8e708f/embedding-w0-8e708f-kserve-c6b4956b6-bkg7s"
Sep 19 01:35:13.128926 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:35:13.128815    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aaed025d-9218-4949-b6b3-80c4f2a16699-tls-certs\") pod \"embedding-w0-8e708f-kserve-c6b4956b6-bkg7s\" (UID: \"aaed025d-9218-4949-b6b3-80c4f2a16699\") " pod="e2e-models-e2e-worker-w0-8e708f/embedding-w0-8e708f-kserve-c6b4956b6-bkg7s"
Sep 19 01:35:13.128926 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:35:13.128905    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aaed025d-9218-4949-b6b3-80c4f2a16699-home\") pod \"embedding-w0-8e708f-kserve-c6b4956b6-bkg7s\" (UID: \"aaed025d-9218-4949-b6b3-80c4f2a16699\") " pod="e2e-models-e2e-worker-w0-8e708f/embedding-w0-8e708f-kserve-c6b4956b6-bkg7s"
Sep 19 01:35:13.229645 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:35:13.229603    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dr86l\" (UniqueName: \"kubernetes.io/projected/aaed025d-9218-4949-b6b3-80c4f2a16699-kube-api-access-dr86l\") pod \"embedding-w0-8e708f-kserve-c6b4956b6-bkg7s\" (UID: \"aaed025d-9218-4949-b6b3-80c4f2a16699\") " pod="e2e-models-e2e-worker-w0-8e708f/embedding-w0-8e708f-kserve-c6b4956b6-bkg7s"
Sep 19 01:35:13.229645 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:35:13.229647    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aaed025d-9218-4949-b6b3-80c4f2a16699-dshm\") pod \"embedding-w0-8e708f-kserve-c6b4956b6-bkg7s\" (UID: \"aaed025d-9218-4949-b6b3-80c4f2a16699\") " pod="e2e-models-e2e-worker-w0-8e708f/embedding-w0-8e708f-kserve-c6b4956b6-bkg7s"
Sep 19 01:35:13.229838 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:35:13.229676    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aaed025d-9218-4949-b6b3-80c4f2a16699-model-cache\") pod \"embedding-w0-8e708f-kserve-c6b4956b6-bkg7s\" (UID: \"aaed025d-9218-4949-b6b3-80c4f2a16699\") " pod="e2e-models-e2e-worker-w0-8e708f/embedding-w0-8e708f-kserve-c6b4956b6-bkg7s"
Sep 19 01:35:13.229838 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:35:13.229706    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aaed025d-9218-4949-b6b3-80c4f2a16699-tmp-dir\") pod \"embedding-w0-8e708f-kserve-c6b4956b6-bkg7s\" (UID: \"aaed025d-9218-4949-b6b3-80c4f2a16699\") " pod="e2e-models-e2e-worker-w0-8e708f/embedding-w0-8e708f-kserve-c6b4956b6-bkg7s"
Sep 19 01:35:13.229838 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:35:13.229731    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aaed025d-9218-4949-b6b3-80c4f2a16699-tls-certs\") pod \"embedding-w0-8e708f-kserve-c6b4956b6-bkg7s\" (UID: \"aaed025d-9218-4949-b6b3-80c4f2a16699\") " pod="e2e-models-e2e-worker-w0-8e708f/embedding-w0-8e708f-kserve-c6b4956b6-bkg7s"
Sep 19 01:35:13.229838 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:35:13.229756    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aaed025d-9218-4949-b6b3-80c4f2a16699-home\") pod \"embedding-w0-8e708f-kserve-c6b4956b6-bkg7s\" (UID: \"aaed025d-9218-4949-b6b3-80c4f2a16699\") " pod="e2e-models-e2e-worker-w0-8e708f/embedding-w0-8e708f-kserve-c6b4956b6-bkg7s"
Sep 19 01:35:13.230167 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:35:13.230138    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aaed025d-9218-4949-b6b3-80c4f2a16699-tmp-dir\") pod \"embedding-w0-8e708f-kserve-c6b4956b6-bkg7s\" (UID: \"aaed025d-9218-4949-b6b3-80c4f2a16699\") " pod="e2e-models-e2e-worker-w0-8e708f/embedding-w0-8e708f-kserve-c6b4956b6-bkg7s"
Sep 19 01:35:13.230249 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:35:13.230163    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aaed025d-9218-4949-b6b3-80c4f2a16699-home\") pod \"embedding-w0-8e708f-kserve-c6b4956b6-bkg7s\" (UID: \"aaed025d-9218-4949-b6b3-80c4f2a16699\") " pod="e2e-models-e2e-worker-w0-8e708f/embedding-w0-8e708f-kserve-c6b4956b6-bkg7s"
Sep 19 01:35:13.230249 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:35:13.230189    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aaed025d-9218-4949-b6b3-80c4f2a16699-model-cache\") pod \"embedding-w0-8e708f-kserve-c6b4956b6-bkg7s\" (UID: \"aaed025d-9218-4949-b6b3-80c4f2a16699\") " pod="e2e-models-e2e-worker-w0-8e708f/embedding-w0-8e708f-kserve-c6b4956b6-bkg7s"
Sep 19 01:35:13.232163 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:35:13.232131    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aaed025d-9218-4949-b6b3-80c4f2a16699-dshm\") pod \"embedding-w0-8e708f-kserve-c6b4956b6-bkg7s\" (UID: \"aaed025d-9218-4949-b6b3-80c4f2a16699\") " pod="e2e-models-e2e-worker-w0-8e708f/embedding-w0-8e708f-kserve-c6b4956b6-bkg7s"
Sep 19 01:35:13.232414 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:35:13.232394    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aaed025d-9218-4949-b6b3-80c4f2a16699-tls-certs\") pod \"embedding-w0-8e708f-kserve-c6b4956b6-bkg7s\" (UID: \"aaed025d-9218-4949-b6b3-80c4f2a16699\") " pod="e2e-models-e2e-worker-w0-8e708f/embedding-w0-8e708f-kserve-c6b4956b6-bkg7s"
Sep 19 01:35:13.237449 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:35:13.237428    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dr86l\" (UniqueName: \"kubernetes.io/projected/aaed025d-9218-4949-b6b3-80c4f2a16699-kube-api-access-dr86l\") pod \"embedding-w0-8e708f-kserve-c6b4956b6-bkg7s\" (UID: \"aaed025d-9218-4949-b6b3-80c4f2a16699\") " pod="e2e-models-e2e-worker-w0-8e708f/embedding-w0-8e708f-kserve-c6b4956b6-bkg7s"
Sep 19 01:35:13.308932 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:35:13.308873    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-8e708f/embedding-w0-8e708f-kserve-c6b4956b6-bkg7s"
Sep 19 01:35:13.581393 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:35:13.581371    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-8e708f/embedding-w0-8e708f-kserve-c6b4956b6-bkg7s"]
Sep 19 01:35:13.583606 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:35:13.583579    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaaed025d_9218_4949_b6b3_80c4f2a16699.slice/crio-525d1bc9bf56f746e9acb27315d30cb52e8e42eefb3435d24c50cedf49296724 WatchSource:0}: Error finding container 525d1bc9bf56f746e9acb27315d30cb52e8e42eefb3435d24c50cedf49296724: Status 404 returned error can't find the container with id 525d1bc9bf56f746e9acb27315d30cb52e8e42eefb3435d24c50cedf49296724
Sep 19 01:35:14.339386 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:35:14.339349    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-8e708f/embedding-w0-8e708f-kserve-c6b4956b6-bkg7s" event={"ID":"aaed025d-9218-4949-b6b3-80c4f2a16699","Type":"ContainerStarted","Data":"07ec0ad7c9841b43d6b4af7a0cd8ae6bcb2f52515612c4881ae0bcf7facb7c2d"}
Sep 19 01:35:14.339806 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:35:14.339394    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-8e708f/embedding-w0-8e708f-kserve-c6b4956b6-bkg7s" event={"ID":"aaed025d-9218-4949-b6b3-80c4f2a16699","Type":"ContainerStarted","Data":"525d1bc9bf56f746e9acb27315d30cb52e8e42eefb3435d24c50cedf49296724"}
Sep 19 01:35:14.359190 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:35:14.359144    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-8e708f/embedding-w0-8e708f-kserve-c6b4956b6-bkg7s" podStartSLOduration=2.359128427 podStartE2EDuration="2.359128427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 01:35:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 01:35:14.356979988 +0000 UTC m=+1365.244649079" watchObservedRunningTime="2026-09-19 01:35:14.359128427 +0000 UTC m=+1365.246797557"
Sep 19 01:35:14.687619 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:35:14.687584    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c9nsz"]
Sep 19 01:35:23.309507 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:35:23.309470    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-8e708f/embedding-w0-8e708f-kserve-c6b4956b6-bkg7s"
Sep 19 01:35:23.309507 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:35:23.309519    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-8e708f/embedding-w0-8e708f-kserve-c6b4956b6-bkg7s"
Sep 19 01:35:23.322025 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:35:23.321999    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-8e708f/embedding-w0-8e708f-kserve-c6b4956b6-bkg7s"
Sep 19 01:35:23.392494 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:35:23.392470    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-8e708f/embedding-w0-8e708f-kserve-c6b4956b6-bkg7s"
Sep 19 01:35:31.410608 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:35:31.410571    2581 scope.go:117] "RemoveContainer" containerID="e6df035147b35c8c248c3ecb2dfcd70e1691e4df378fcd5d0700a5e01eab7db5"
Sep 19 01:35:34.119369 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:35:34.119327    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b5fe4519-openshift-default-84cfc86584-vxfnt"]
Sep 19 01:35:34.119999 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:35:34.119536    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-b5fe4519-openshift-default-84cfc86584-vxfnt" podUID="62b1c519-4609-4138-a75c-7bcc23dcbf10" containerName="istio-proxy" containerID="cri-o://d1ed080b26198bd2b779f38ab755e6a358a750a0bd85da0be6ca2e2a07e39e3d" gracePeriod=30
Sep 19 01:35:36.546106 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:35:36.546074    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c9nsz"]
Sep 19 01:35:39.362547 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:35:39.362523    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b5fe4519-openshift-default-84cfc86584-vxfnt"
Sep 19 01:35:39.456278 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:35:39.456193    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62b1c519-4609-4138-a75c-7bcc23dcbf10-istio-data\") pod \"62b1c519-4609-4138-a75c-7bcc23dcbf10\" (UID: \"62b1c519-4609-4138-a75c-7bcc23dcbf10\") "
Sep 19 01:35:39.456278 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:35:39.456252    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kpkkb\" (UniqueName: \"kubernetes.io/projected/62b1c519-4609-4138-a75c-7bcc23dcbf10-kube-api-access-kpkkb\") pod \"62b1c519-4609-4138-a75c-7bcc23dcbf10\" (UID: \"62b1c519-4609-4138-a75c-7bcc23dcbf10\") "
Sep 19 01:35:39.456493 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:35:39.456282    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62b1c519-4609-4138-a75c-7bcc23dcbf10-workload-socket\") pod \"62b1c519-4609-4138-a75c-7bcc23dcbf10\" (UID: \"62b1c519-4609-4138-a75c-7bcc23dcbf10\") "
Sep 19 01:35:39.456493 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:35:39.456303    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62b1c519-4609-4138-a75c-7bcc23dcbf10-istiod-ca-cert\") pod \"62b1c519-4609-4138-a75c-7bcc23dcbf10\" (UID: \"62b1c519-4609-4138-a75c-7bcc23dcbf10\") "
Sep 19 01:35:39.456493 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:35:39.456341    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62b1c519-4609-4138-a75c-7bcc23dcbf10-istio-envoy\") pod \"62b1c519-4609-4138-a75c-7bcc23dcbf10\" (UID: \"62b1c519-4609-4138-a75c-7bcc23dcbf10\") "
Sep 19 01:35:39.456493 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:35:39.456392    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62b1c519-4609-4138-a75c-7bcc23dcbf10-workload-certs\") pod \"62b1c519-4609-4138-a75c-7bcc23dcbf10\" (UID: \"62b1c519-4609-4138-a75c-7bcc23dcbf10\") "
Sep 19 01:35:39.456493 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:35:39.456413    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62b1c519-4609-4138-a75c-7bcc23dcbf10-credential-socket\") pod \"62b1c519-4609-4138-a75c-7bcc23dcbf10\" (UID: \"62b1c519-4609-4138-a75c-7bcc23dcbf10\") "
Sep 19 01:35:39.456493 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:35:39.456439    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62b1c519-4609-4138-a75c-7bcc23dcbf10-istio-token\") pod \"62b1c519-4609-4138-a75c-7bcc23dcbf10\" (UID: \"62b1c519-4609-4138-a75c-7bcc23dcbf10\") "
Sep 19 01:35:39.456493 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:35:39.456463    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62b1c519-4609-4138-a75c-7bcc23dcbf10-istio-podinfo\") pod \"62b1c519-4609-4138-a75c-7bcc23dcbf10\" (UID: \"62b1c519-4609-4138-a75c-7bcc23dcbf10\") "
Sep 19 01:35:39.456889 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:35:39.456497    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62b1c519-4609-4138-a75c-7bcc23dcbf10-istio-data" (OuterVolumeSpecName: "istio-data") pod "62b1c519-4609-4138-a75c-7bcc23dcbf10" (UID: "62b1c519-4609-4138-a75c-7bcc23dcbf10"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 01:35:39.456889 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:35:39.456666    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62b1c519-4609-4138-a75c-7bcc23dcbf10-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "62b1c519-4609-4138-a75c-7bcc23dcbf10" (UID: "62b1c519-4609-4138-a75c-7bcc23dcbf10"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 01:35:39.456889 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:35:39.456689    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62b1c519-4609-4138-a75c-7bcc23dcbf10-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "62b1c519-4609-4138-a75c-7bcc23dcbf10" (UID: "62b1c519-4609-4138-a75c-7bcc23dcbf10"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 01:35:39.456889 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:35:39.456836    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62b1c519-4609-4138-a75c-7bcc23dcbf10-workload-certs\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:35:39.456889 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:35:39.456884    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62b1c519-4609-4138-a75c-7bcc23dcbf10-credential-socket\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:35:39.457355 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:35:39.456901    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62b1c519-4609-4138-a75c-7bcc23dcbf10-istio-data\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:35:39.457355 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:35:39.457088    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62b1c519-4609-4138-a75c-7bcc23dcbf10-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "62b1c519-4609-4138-a75c-7bcc23dcbf10" (UID: "62b1c519-4609-4138-a75c-7bcc23dcbf10"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 01:35:39.457355 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:35:39.457133    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62b1c519-4609-4138-a75c-7bcc23dcbf10-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "62b1c519-4609-4138-a75c-7bcc23dcbf10" (UID: "62b1c519-4609-4138-a75c-7bcc23dcbf10"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 01:35:39.458817 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:35:39.458790    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62b1c519-4609-4138-a75c-7bcc23dcbf10-kube-api-access-kpkkb" (OuterVolumeSpecName: "kube-api-access-kpkkb") pod "62b1c519-4609-4138-a75c-7bcc23dcbf10" (UID: "62b1c519-4609-4138-a75c-7bcc23dcbf10"). InnerVolumeSpecName "kube-api-access-kpkkb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 01:35:39.458948 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:35:39.458930    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62b1c519-4609-4138-a75c-7bcc23dcbf10-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "62b1c519-4609-4138-a75c-7bcc23dcbf10" (UID: "62b1c519-4609-4138-a75c-7bcc23dcbf10"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 01:35:39.459029 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:35:39.459015    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62b1c519-4609-4138-a75c-7bcc23dcbf10-istio-token" (OuterVolumeSpecName: "istio-token") pod "62b1c519-4609-4138-a75c-7bcc23dcbf10" (UID: "62b1c519-4609-4138-a75c-7bcc23dcbf10"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 01:35:39.459470 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:35:39.459444    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/62b1c519-4609-4138-a75c-7bcc23dcbf10-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "62b1c519-4609-4138-a75c-7bcc23dcbf10" (UID: "62b1c519-4609-4138-a75c-7bcc23dcbf10"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 19 01:35:39.464516 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:35:39.464495    2581 generic.go:356] "Generic (PLEG): container finished" podID="62b1c519-4609-4138-a75c-7bcc23dcbf10" containerID="d1ed080b26198bd2b779f38ab755e6a358a750a0bd85da0be6ca2e2a07e39e3d" exitCode=0
Sep 19 01:35:39.464607 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:35:39.464549    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b5fe4519-openshift-default-84cfc86584-vxfnt"
Sep 19 01:35:39.464607 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:35:39.464586    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b5fe4519-openshift-default-84cfc86584-vxfnt" event={"ID":"62b1c519-4609-4138-a75c-7bcc23dcbf10","Type":"ContainerDied","Data":"d1ed080b26198bd2b779f38ab755e6a358a750a0bd85da0be6ca2e2a07e39e3d"}
Sep 19 01:35:39.464681 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:35:39.464627    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b5fe4519-openshift-default-84cfc86584-vxfnt" event={"ID":"62b1c519-4609-4138-a75c-7bcc23dcbf10","Type":"ContainerDied","Data":"5490179ec2ae77798af857c4f80403bd4a77b8eb0c297424281de9cf88ccf1eb"}
Sep 19 01:35:39.464681 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:35:39.464648    2581 scope.go:117] "RemoveContainer" containerID="d1ed080b26198bd2b779f38ab755e6a358a750a0bd85da0be6ca2e2a07e39e3d"
Sep 19 01:35:39.482263 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:35:39.482244    2581 scope.go:117] "RemoveContainer" containerID="df1658114676d73ee724993ad81890e6cc11b4550ff0bd12d3b61329b8229945"
Sep 19 01:35:39.490142 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:35:39.490120    2581 scope.go:117] "RemoveContainer" containerID="d1ed080b26198bd2b779f38ab755e6a358a750a0bd85da0be6ca2e2a07e39e3d"
Sep 19 01:35:39.490387 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:35:39.490370    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1ed080b26198bd2b779f38ab755e6a358a750a0bd85da0be6ca2e2a07e39e3d\": container with ID starting with d1ed080b26198bd2b779f38ab755e6a358a750a0bd85da0be6ca2e2a07e39e3d not found: ID does not exist" containerID="d1ed080b26198bd2b779f38ab755e6a358a750a0bd85da0be6ca2e2a07e39e3d"
Sep 19 01:35:39.490466 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:35:39.490398    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1ed080b26198bd2b779f38ab755e6a358a750a0bd85da0be6ca2e2a07e39e3d"} err="failed to get container status \"d1ed080b26198bd2b779f38ab755e6a358a750a0bd85da0be6ca2e2a07e39e3d\": rpc error: code = NotFound desc = could not find container \"d1ed080b26198bd2b779f38ab755e6a358a750a0bd85da0be6ca2e2a07e39e3d\": container with ID starting with d1ed080b26198bd2b779f38ab755e6a358a750a0bd85da0be6ca2e2a07e39e3d not found: ID does not exist"
Sep 19 01:35:39.490466 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:35:39.490420    2581 scope.go:117] "RemoveContainer" containerID="df1658114676d73ee724993ad81890e6cc11b4550ff0bd12d3b61329b8229945"
Sep 19 01:35:39.490720 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:35:39.490700    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df1658114676d73ee724993ad81890e6cc11b4550ff0bd12d3b61329b8229945\": container with ID starting with df1658114676d73ee724993ad81890e6cc11b4550ff0bd12d3b61329b8229945 not found: ID does not exist" containerID="df1658114676d73ee724993ad81890e6cc11b4550ff0bd12d3b61329b8229945"
Sep 19 01:35:39.490761 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:35:39.490728    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df1658114676d73ee724993ad81890e6cc11b4550ff0bd12d3b61329b8229945"} err="failed to get container status \"df1658114676d73ee724993ad81890e6cc11b4550ff0bd12d3b61329b8229945\": rpc error: code = NotFound desc = could not find container \"df1658114676d73ee724993ad81890e6cc11b4550ff0bd12d3b61329b8229945\": container with ID starting with df1658114676d73ee724993ad81890e6cc11b4550ff0bd12d3b61329b8229945 not found: ID does not exist"
Sep 19 01:35:39.499154 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:35:39.499132    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b5fe4519-openshift-default-84cfc86584-vxfnt"]
Sep 19 01:35:39.502663 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:35:39.502643    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-b5fe4519-openshift-default-84cfc86584-vxfnt"]
Sep 19 01:35:39.557957 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:35:39.557933    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62b1c519-4609-4138-a75c-7bcc23dcbf10-istio-token\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:35:39.557957 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:35:39.557954    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62b1c519-4609-4138-a75c-7bcc23dcbf10-istio-podinfo\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:35:39.558081 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:35:39.557963    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpkkb\" (UniqueName: \"kubernetes.io/projected/62b1c519-4609-4138-a75c-7bcc23dcbf10-kube-api-access-kpkkb\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:35:39.558081 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:35:39.557973    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62b1c519-4609-4138-a75c-7bcc23dcbf10-workload-socket\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:35:39.558081 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:35:39.557983    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62b1c519-4609-4138-a75c-7bcc23dcbf10-istiod-ca-cert\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:35:39.558081 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:35:39.557990    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62b1c519-4609-4138-a75c-7bcc23dcbf10-istio-envoy\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:35:41.272350 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:35:41.272316    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62b1c519-4609-4138-a75c-7bcc23dcbf10" path="/var/lib/kubelet/pods/62b1c519-4609-4138-a75c-7bcc23dcbf10/volumes"
Sep 19 01:36:05.592378 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:05.592331    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c9nsz"]
Sep 19 01:36:12.059914 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:12.059820    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-8e708f-rk8qk"]
Sep 19 01:36:12.060276 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:12.060212    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62b1c519-4609-4138-a75c-7bcc23dcbf10" containerName="istio-proxy"
Sep 19 01:36:12.060276 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:12.060227    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="62b1c519-4609-4138-a75c-7bcc23dcbf10" containerName="istio-proxy"
Sep 19 01:36:12.060354 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:12.060296    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="62b1c519-4609-4138-a75c-7bcc23dcbf10" containerName="istio-proxy"
Sep 19 01:36:12.063434 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:12.063416    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-8e708f-rk8qk"
Sep 19 01:36:12.066126 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:36:12.066090    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"maas-api-revoke-keys-e2e-worker-w0-8e708f-rk8qk\" is forbidden: User \"system:node:ip-10-0-129-113.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-129-113.ec2.internal' and this object" podUID="8454b703-9ec7-4ad3-b43a-c10494457bb6" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-8e708f-rk8qk"
Sep 19 01:36:12.066246 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:36:12.066117    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-129-113.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-129-113.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap"
Sep 19 01:36:12.066971 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:12.066955    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s7pvt\""
Sep 19 01:36:12.083292 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:12.083267    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-8e708f-rk8qk"]
Sep 19 01:36:12.225734 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:12.225715    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8454b703-9ec7-4ad3-b43a-c10494457bb6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-8e708f-rk8qk\" (UID: \"8454b703-9ec7-4ad3-b43a-c10494457bb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-8e708f-rk8qk"
Sep 19 01:36:12.326673 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:12.326613    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8454b703-9ec7-4ad3-b43a-c10494457bb6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-8e708f-rk8qk\" (UID: \"8454b703-9ec7-4ad3-b43a-c10494457bb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-8e708f-rk8qk"
Sep 19 01:36:13.292632 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:13.292601    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 19 01:36:13.298084 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:13.298062    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8454b703-9ec7-4ad3-b43a-c10494457bb6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-8e708f-rk8qk\" (UID: \"8454b703-9ec7-4ad3-b43a-c10494457bb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-8e708f-rk8qk"
Sep 19 01:36:13.573884 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:13.573780    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-8e708f-rk8qk"
Sep 19 01:36:13.849993 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:13.849970    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-8e708f-rk8qk"]
Sep 19 01:36:13.852414 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:36:13.852387    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8454b703_9ec7_4ad3_b43a_c10494457bb6.slice/crio-3fb4fe5735e82ac1158511fc2a34d75c3c90894e4abd9c0015420dbe57a98fb0 WatchSource:0}: Error finding container 3fb4fe5735e82ac1158511fc2a34d75c3c90894e4abd9c0015420dbe57a98fb0: Status 404 returned error can't find the container with id 3fb4fe5735e82ac1158511fc2a34d75c3c90894e4abd9c0015420dbe57a98fb0
Sep 19 01:36:14.642387 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:14.642357    2581 generic.go:356] "Generic (PLEG): container finished" podID="8454b703-9ec7-4ad3-b43a-c10494457bb6" containerID="1b9cd47f5bf044527ffcd790a9d01b644c8b8287d3245ea2d15985f95d9c4a8d" exitCode=0
Sep 19 01:36:14.642709 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:14.642451    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-8e708f-rk8qk" event={"ID":"8454b703-9ec7-4ad3-b43a-c10494457bb6","Type":"ContainerDied","Data":"1b9cd47f5bf044527ffcd790a9d01b644c8b8287d3245ea2d15985f95d9c4a8d"}
Sep 19 01:36:14.642709 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:14.642507    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-8e708f-rk8qk" event={"ID":"8454b703-9ec7-4ad3-b43a-c10494457bb6","Type":"ContainerStarted","Data":"3fb4fe5735e82ac1158511fc2a34d75c3c90894e4abd9c0015420dbe57a98fb0"}
Sep 19 01:36:15.780377 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:15.780355    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-8e708f-rk8qk"
Sep 19 01:36:15.855785 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:15.855764    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8454b703-9ec7-4ad3-b43a-c10494457bb6-maas-api-service-ca\") pod \"8454b703-9ec7-4ad3-b43a-c10494457bb6\" (UID: \"8454b703-9ec7-4ad3-b43a-c10494457bb6\") "
Sep 19 01:36:15.856092 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:15.856070    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8454b703-9ec7-4ad3-b43a-c10494457bb6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8454b703-9ec7-4ad3-b43a-c10494457bb6" (UID: "8454b703-9ec7-4ad3-b43a-c10494457bb6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 01:36:15.956750 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:15.956686    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8454b703-9ec7-4ad3-b43a-c10494457bb6-maas-api-service-ca\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:36:16.653895 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:16.653850    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-8e708f-rk8qk"
Sep 19 01:36:16.653895 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:16.653869    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-8e708f-rk8qk" event={"ID":"8454b703-9ec7-4ad3-b43a-c10494457bb6","Type":"ContainerDied","Data":"3fb4fe5735e82ac1158511fc2a34d75c3c90894e4abd9c0015420dbe57a98fb0"}
Sep 19 01:36:16.654115 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:16.653917    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3fb4fe5735e82ac1158511fc2a34d75c3c90894e4abd9c0015420dbe57a98fb0"
Sep 19 01:36:16.964345 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:16.964268    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-8e708f/distinct-2-w0-8e708f-kserve-59b54cd666-tg2h7"]
Sep 19 01:36:16.979202 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:16.979176    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-8e708f/distinct-w0-8e708f-kserve-6f58868fbf-mfmbr"]
Sep 19 01:36:17.006123 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:17.006090    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-8e708f/embedding-w0-8e708f-kserve-c6b4956b6-bkg7s"]
Sep 19 01:36:17.063809 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:17.063777    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-8e708f/simulator-w0-8e708f-kserve-5f9cd6679f-6ncxb"]
Sep 19 01:36:27.766542 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:27.766509    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8e708f-29829690-2sn9s"]
Sep 19 01:36:27.770463 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:27.770438    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8e708f-29829690-2sn9s"]
Sep 19 01:36:27.804257 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:27.804226    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-8e708f-75fb8965d6-j4nrt"]
Sep 19 01:36:27.804528 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:27.804472    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-8e708f-75fb8965d6-j4nrt" podUID="c44fe035-0180-4ec3-b581-4a412e349571" containerName="payload-pre-processing" containerID="cri-o://63c20c8888f98b8d6f5102a149aec03c7cb4bae285ff1d349c085927999b8a19" gracePeriod=30
Sep 19 01:36:28.028035 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:28.027951    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-8e708f-rk8qk"]
Sep 19 01:36:28.034690 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:28.034665    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-8e708f-rk8qk"]
Sep 19 01:36:28.051385 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:28.051366    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-8e708f-75fb8965d6-j4nrt"
Sep 19 01:36:28.154204 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:28.154172    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4dkgc\" (UniqueName: \"kubernetes.io/projected/c44fe035-0180-4ec3-b581-4a412e349571-kube-api-access-4dkgc\") pod \"c44fe035-0180-4ec3-b581-4a412e349571\" (UID: \"c44fe035-0180-4ec3-b581-4a412e349571\") "
Sep 19 01:36:28.154405 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:28.154249    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c44fe035-0180-4ec3-b581-4a412e349571-plugins-config-volume\") pod \"c44fe035-0180-4ec3-b581-4a412e349571\" (UID: \"c44fe035-0180-4ec3-b581-4a412e349571\") "
Sep 19 01:36:28.154616 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:28.154592    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c44fe035-0180-4ec3-b581-4a412e349571-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c44fe035-0180-4ec3-b581-4a412e349571" (UID: "c44fe035-0180-4ec3-b581-4a412e349571"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 01:36:28.156461 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:28.156441    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c44fe035-0180-4ec3-b581-4a412e349571-kube-api-access-4dkgc" (OuterVolumeSpecName: "kube-api-access-4dkgc") pod "c44fe035-0180-4ec3-b581-4a412e349571" (UID: "c44fe035-0180-4ec3-b581-4a412e349571"). InnerVolumeSpecName "kube-api-access-4dkgc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 01:36:28.255178 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:28.255151    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c44fe035-0180-4ec3-b581-4a412e349571-plugins-config-volume\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:36:28.255178 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:28.255174    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dkgc\" (UniqueName: \"kubernetes.io/projected/c44fe035-0180-4ec3-b581-4a412e349571-kube-api-access-4dkgc\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:36:28.714771 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:28.714730    2581 generic.go:356] "Generic (PLEG): container finished" podID="c44fe035-0180-4ec3-b581-4a412e349571" containerID="63c20c8888f98b8d6f5102a149aec03c7cb4bae285ff1d349c085927999b8a19" exitCode=0
Sep 19 01:36:28.714946 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:28.714781    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-8e708f-75fb8965d6-j4nrt"
Sep 19 01:36:28.714946 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:28.714816    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-8e708f-75fb8965d6-j4nrt" event={"ID":"c44fe035-0180-4ec3-b581-4a412e349571","Type":"ContainerDied","Data":"63c20c8888f98b8d6f5102a149aec03c7cb4bae285ff1d349c085927999b8a19"}
Sep 19 01:36:28.714946 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:28.714889    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-8e708f-75fb8965d6-j4nrt" event={"ID":"c44fe035-0180-4ec3-b581-4a412e349571","Type":"ContainerDied","Data":"a32e02bb55007eb820cb4d91aa3c517581d0c5d6f059ede092df23abd9be9ad9"}
Sep 19 01:36:28.714946 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:28.714913    2581 scope.go:117] "RemoveContainer" containerID="63c20c8888f98b8d6f5102a149aec03c7cb4bae285ff1d349c085927999b8a19"
Sep 19 01:36:28.724513 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:28.724496    2581 scope.go:117] "RemoveContainer" containerID="63c20c8888f98b8d6f5102a149aec03c7cb4bae285ff1d349c085927999b8a19"
Sep 19 01:36:28.724769 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:36:28.724751    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63c20c8888f98b8d6f5102a149aec03c7cb4bae285ff1d349c085927999b8a19\": container with ID starting with 63c20c8888f98b8d6f5102a149aec03c7cb4bae285ff1d349c085927999b8a19 not found: ID does not exist" containerID="63c20c8888f98b8d6f5102a149aec03c7cb4bae285ff1d349c085927999b8a19"
Sep 19 01:36:28.724813 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:28.724776    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63c20c8888f98b8d6f5102a149aec03c7cb4bae285ff1d349c085927999b8a19"} err="failed to get container status \"63c20c8888f98b8d6f5102a149aec03c7cb4bae285ff1d349c085927999b8a19\": rpc error: code = NotFound desc = could not find container \"63c20c8888f98b8d6f5102a149aec03c7cb4bae285ff1d349c085927999b8a19\": container with ID starting with 63c20c8888f98b8d6f5102a149aec03c7cb4bae285ff1d349c085927999b8a19 not found: ID does not exist"
Sep 19 01:36:28.737441 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:28.737414    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-8e708f-75fb8965d6-j4nrt"]
Sep 19 01:36:28.742050 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:28.742029    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-8e708f-75fb8965d6-j4nrt"]
Sep 19 01:36:29.271841 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:29.271807    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25527fb0-589a-44d6-9445-013a4a27f92b" path="/var/lib/kubelet/pods/25527fb0-589a-44d6-9445-013a4a27f92b/volumes"
Sep 19 01:36:29.272248 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:29.272125    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8454b703-9ec7-4ad3-b43a-c10494457bb6" path="/var/lib/kubelet/pods/8454b703-9ec7-4ad3-b43a-c10494457bb6/volumes"
Sep 19 01:36:29.272449 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:29.272435    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c44fe035-0180-4ec3-b581-4a412e349571" path="/var/lib/kubelet/pods/c44fe035-0180-4ec3-b581-4a412e349571/volumes"
Sep 19 01:36:29.621128 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:29.621101    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c9nsz"]
Sep 19 01:36:31.433433 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:31.433403    2581 scope.go:117] "RemoveContainer" containerID="e949de4ab1cf21962484bf39ea72b18866a747ff9e680971b8e898795af9301b"
Sep 19 01:36:31.441569 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:31.441550    2581 scope.go:117] "RemoveContainer" containerID="20fe871e3a06b4387d5d3fc354165cd0b8ddc62277c26b1ccc5b2d7235d3dbcb"
Sep 19 01:36:31.622407 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:31.622377    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-a7b600-openshift-default-c88cb89b8-jxm2q"]
Sep 19 01:36:31.622687 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:31.622676    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8454b703-9ec7-4ad3-b43a-c10494457bb6" containerName="revoke-keys"
Sep 19 01:36:31.622731 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:31.622689    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8454b703-9ec7-4ad3-b43a-c10494457bb6" containerName="revoke-keys"
Sep 19 01:36:31.622731 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:31.622702    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c44fe035-0180-4ec3-b581-4a412e349571" containerName="payload-pre-processing"
Sep 19 01:36:31.622731 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:31.622708    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c44fe035-0180-4ec3-b581-4a412e349571" containerName="payload-pre-processing"
Sep 19 01:36:31.622731 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:31.622721    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62b1c519-4609-4138-a75c-7bcc23dcbf10" containerName="istio-proxy"
Sep 19 01:36:31.622731 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:31.622726    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="62b1c519-4609-4138-a75c-7bcc23dcbf10" containerName="istio-proxy"
Sep 19 01:36:31.622910 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:31.622782    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8454b703-9ec7-4ad3-b43a-c10494457bb6" containerName="revoke-keys"
Sep 19 01:36:31.622910 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:31.622791    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c44fe035-0180-4ec3-b581-4a412e349571" containerName="payload-pre-processing"
Sep 19 01:36:31.622910 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:31.622799    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="62b1c519-4609-4138-a75c-7bcc23dcbf10" containerName="istio-proxy"
Sep 19 01:36:31.627061 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:31.627046    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a7b600-openshift-default-c88cb89b8-jxm2q"
Sep 19 01:36:31.629422 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:31.629403    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-a7b600-openshift-default-dockercfg-6s8j5\""
Sep 19 01:36:31.638929 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:31.638906    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a7b600-openshift-default-c88cb89b8-jxm2q"]
Sep 19 01:36:31.783802 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:31.783719    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/461f7ac7-05b6-412d-8d17-6d0a46f17fc2-istio-envoy\") pod \"e2e-api-a-a7b600-openshift-default-c88cb89b8-jxm2q\" (UID: \"461f7ac7-05b6-412d-8d17-6d0a46f17fc2\") " pod="openshift-ingress/e2e-api-a-a7b600-openshift-default-c88cb89b8-jxm2q"
Sep 19 01:36:31.783802 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:31.783774    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/461f7ac7-05b6-412d-8d17-6d0a46f17fc2-credential-socket\") pod \"e2e-api-a-a7b600-openshift-default-c88cb89b8-jxm2q\" (UID: \"461f7ac7-05b6-412d-8d17-6d0a46f17fc2\") " pod="openshift-ingress/e2e-api-a-a7b600-openshift-default-c88cb89b8-jxm2q"
Sep 19 01:36:31.784057 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:31.783829    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/461f7ac7-05b6-412d-8d17-6d0a46f17fc2-istio-data\") pod \"e2e-api-a-a7b600-openshift-default-c88cb89b8-jxm2q\" (UID: \"461f7ac7-05b6-412d-8d17-6d0a46f17fc2\") " pod="openshift-ingress/e2e-api-a-a7b600-openshift-default-c88cb89b8-jxm2q"
Sep 19 01:36:31.784057 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:31.783885    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhxf6\" (UniqueName: \"kubernetes.io/projected/461f7ac7-05b6-412d-8d17-6d0a46f17fc2-kube-api-access-nhxf6\") pod \"e2e-api-a-a7b600-openshift-default-c88cb89b8-jxm2q\" (UID: \"461f7ac7-05b6-412d-8d17-6d0a46f17fc2\") " pod="openshift-ingress/e2e-api-a-a7b600-openshift-default-c88cb89b8-jxm2q"
Sep 19 01:36:31.784057 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:31.783919    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/461f7ac7-05b6-412d-8d17-6d0a46f17fc2-istiod-ca-cert\") pod \"e2e-api-a-a7b600-openshift-default-c88cb89b8-jxm2q\" (UID: \"461f7ac7-05b6-412d-8d17-6d0a46f17fc2\") " pod="openshift-ingress/e2e-api-a-a7b600-openshift-default-c88cb89b8-jxm2q"
Sep 19 01:36:31.784057 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:31.783952    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/461f7ac7-05b6-412d-8d17-6d0a46f17fc2-workload-socket\") pod \"e2e-api-a-a7b600-openshift-default-c88cb89b8-jxm2q\" (UID: \"461f7ac7-05b6-412d-8d17-6d0a46f17fc2\") " pod="openshift-ingress/e2e-api-a-a7b600-openshift-default-c88cb89b8-jxm2q"
Sep 19 01:36:31.784057 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:31.784014    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/461f7ac7-05b6-412d-8d17-6d0a46f17fc2-istio-token\") pod \"e2e-api-a-a7b600-openshift-default-c88cb89b8-jxm2q\" (UID: \"461f7ac7-05b6-412d-8d17-6d0a46f17fc2\") " pod="openshift-ingress/e2e-api-a-a7b600-openshift-default-c88cb89b8-jxm2q"
Sep 19 01:36:31.784057 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:31.784056    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/461f7ac7-05b6-412d-8d17-6d0a46f17fc2-workload-certs\") pod \"e2e-api-a-a7b600-openshift-default-c88cb89b8-jxm2q\" (UID: \"461f7ac7-05b6-412d-8d17-6d0a46f17fc2\") " pod="openshift-ingress/e2e-api-a-a7b600-openshift-default-c88cb89b8-jxm2q"
Sep 19 01:36:31.784277 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:31.784087    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/461f7ac7-05b6-412d-8d17-6d0a46f17fc2-istio-podinfo\") pod \"e2e-api-a-a7b600-openshift-default-c88cb89b8-jxm2q\" (UID: \"461f7ac7-05b6-412d-8d17-6d0a46f17fc2\") " pod="openshift-ingress/e2e-api-a-a7b600-openshift-default-c88cb89b8-jxm2q"
Sep 19 01:36:31.884608 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:31.884578    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/461f7ac7-05b6-412d-8d17-6d0a46f17fc2-workload-certs\") pod \"e2e-api-a-a7b600-openshift-default-c88cb89b8-jxm2q\" (UID: \"461f7ac7-05b6-412d-8d17-6d0a46f17fc2\") " pod="openshift-ingress/e2e-api-a-a7b600-openshift-default-c88cb89b8-jxm2q"
Sep 19 01:36:31.884726 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:31.884612    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/461f7ac7-05b6-412d-8d17-6d0a46f17fc2-istio-podinfo\") pod \"e2e-api-a-a7b600-openshift-default-c88cb89b8-jxm2q\" (UID: \"461f7ac7-05b6-412d-8d17-6d0a46f17fc2\") " pod="openshift-ingress/e2e-api-a-a7b600-openshift-default-c88cb89b8-jxm2q"
Sep 19 01:36:31.884726 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:31.884648    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/461f7ac7-05b6-412d-8d17-6d0a46f17fc2-istio-envoy\") pod \"e2e-api-a-a7b600-openshift-default-c88cb89b8-jxm2q\" (UID: \"461f7ac7-05b6-412d-8d17-6d0a46f17fc2\") " pod="openshift-ingress/e2e-api-a-a7b600-openshift-default-c88cb89b8-jxm2q"
Sep 19 01:36:31.884726 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:31.884689    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/461f7ac7-05b6-412d-8d17-6d0a46f17fc2-credential-socket\") pod \"e2e-api-a-a7b600-openshift-default-c88cb89b8-jxm2q\" (UID: \"461f7ac7-05b6-412d-8d17-6d0a46f17fc2\") " pod="openshift-ingress/e2e-api-a-a7b600-openshift-default-c88cb89b8-jxm2q"
Sep 19 01:36:31.884726 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:31.884716    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/461f7ac7-05b6-412d-8d17-6d0a46f17fc2-istio-data\") pod \"e2e-api-a-a7b600-openshift-default-c88cb89b8-jxm2q\" (UID: \"461f7ac7-05b6-412d-8d17-6d0a46f17fc2\") " pod="openshift-ingress/e2e-api-a-a7b600-openshift-default-c88cb89b8-jxm2q"
Sep 19 01:36:31.884955 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:31.884745    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhxf6\" (UniqueName: \"kubernetes.io/projected/461f7ac7-05b6-412d-8d17-6d0a46f17fc2-kube-api-access-nhxf6\") pod \"e2e-api-a-a7b600-openshift-default-c88cb89b8-jxm2q\" (UID: \"461f7ac7-05b6-412d-8d17-6d0a46f17fc2\") " pod="openshift-ingress/e2e-api-a-a7b600-openshift-default-c88cb89b8-jxm2q"
Sep 19 01:36:31.884955 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:31.884775    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/461f7ac7-05b6-412d-8d17-6d0a46f17fc2-istiod-ca-cert\") pod \"e2e-api-a-a7b600-openshift-default-c88cb89b8-jxm2q\" (UID: \"461f7ac7-05b6-412d-8d17-6d0a46f17fc2\") " pod="openshift-ingress/e2e-api-a-a7b600-openshift-default-c88cb89b8-jxm2q"
Sep 19 01:36:31.884955 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:31.884807    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/461f7ac7-05b6-412d-8d17-6d0a46f17fc2-workload-socket\") pod \"e2e-api-a-a7b600-openshift-default-c88cb89b8-jxm2q\" (UID: \"461f7ac7-05b6-412d-8d17-6d0a46f17fc2\") " pod="openshift-ingress/e2e-api-a-a7b600-openshift-default-c88cb89b8-jxm2q"
Sep 19 01:36:31.884955 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:31.884843    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/461f7ac7-05b6-412d-8d17-6d0a46f17fc2-istio-token\") pod \"e2e-api-a-a7b600-openshift-default-c88cb89b8-jxm2q\" (UID: \"461f7ac7-05b6-412d-8d17-6d0a46f17fc2\") " pod="openshift-ingress/e2e-api-a-a7b600-openshift-default-c88cb89b8-jxm2q"
Sep 19 01:36:31.885161 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:31.885007    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/461f7ac7-05b6-412d-8d17-6d0a46f17fc2-workload-certs\") pod \"e2e-api-a-a7b600-openshift-default-c88cb89b8-jxm2q\" (UID: \"461f7ac7-05b6-412d-8d17-6d0a46f17fc2\") " pod="openshift-ingress/e2e-api-a-a7b600-openshift-default-c88cb89b8-jxm2q"
Sep 19 01:36:31.885161 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:31.885132    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/461f7ac7-05b6-412d-8d17-6d0a46f17fc2-credential-socket\") pod \"e2e-api-a-a7b600-openshift-default-c88cb89b8-jxm2q\" (UID: \"461f7ac7-05b6-412d-8d17-6d0a46f17fc2\") " pod="openshift-ingress/e2e-api-a-a7b600-openshift-default-c88cb89b8-jxm2q"
Sep 19 01:36:31.885371 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:31.885349    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/461f7ac7-05b6-412d-8d17-6d0a46f17fc2-istio-data\") pod \"e2e-api-a-a7b600-openshift-default-c88cb89b8-jxm2q\" (UID: \"461f7ac7-05b6-412d-8d17-6d0a46f17fc2\") " pod="openshift-ingress/e2e-api-a-a7b600-openshift-default-c88cb89b8-jxm2q"
Sep 19 01:36:31.885470 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:31.885346    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/461f7ac7-05b6-412d-8d17-6d0a46f17fc2-workload-socket\") pod \"e2e-api-a-a7b600-openshift-default-c88cb89b8-jxm2q\" (UID: \"461f7ac7-05b6-412d-8d17-6d0a46f17fc2\") " pod="openshift-ingress/e2e-api-a-a7b600-openshift-default-c88cb89b8-jxm2q"
Sep 19 01:36:31.885517 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:31.885482    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/461f7ac7-05b6-412d-8d17-6d0a46f17fc2-istiod-ca-cert\") pod \"e2e-api-a-a7b600-openshift-default-c88cb89b8-jxm2q\" (UID: \"461f7ac7-05b6-412d-8d17-6d0a46f17fc2\") " pod="openshift-ingress/e2e-api-a-a7b600-openshift-default-c88cb89b8-jxm2q"
Sep 19 01:36:31.887180 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:31.887160    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/461f7ac7-05b6-412d-8d17-6d0a46f17fc2-istio-envoy\") pod \"e2e-api-a-a7b600-openshift-default-c88cb89b8-jxm2q\" (UID: \"461f7ac7-05b6-412d-8d17-6d0a46f17fc2\") " pod="openshift-ingress/e2e-api-a-a7b600-openshift-default-c88cb89b8-jxm2q"
Sep 19 01:36:31.887414 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:31.887384    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/461f7ac7-05b6-412d-8d17-6d0a46f17fc2-istio-podinfo\") pod \"e2e-api-a-a7b600-openshift-default-c88cb89b8-jxm2q\" (UID: \"461f7ac7-05b6-412d-8d17-6d0a46f17fc2\") " pod="openshift-ingress/e2e-api-a-a7b600-openshift-default-c88cb89b8-jxm2q"
Sep 19 01:36:31.895430 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:31.895411    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/461f7ac7-05b6-412d-8d17-6d0a46f17fc2-istio-token\") pod \"e2e-api-a-a7b600-openshift-default-c88cb89b8-jxm2q\" (UID: \"461f7ac7-05b6-412d-8d17-6d0a46f17fc2\") " pod="openshift-ingress/e2e-api-a-a7b600-openshift-default-c88cb89b8-jxm2q"
Sep 19 01:36:31.895671 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:31.895652    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhxf6\" (UniqueName: \"kubernetes.io/projected/461f7ac7-05b6-412d-8d17-6d0a46f17fc2-kube-api-access-nhxf6\") pod \"e2e-api-a-a7b600-openshift-default-c88cb89b8-jxm2q\" (UID: \"461f7ac7-05b6-412d-8d17-6d0a46f17fc2\") " pod="openshift-ingress/e2e-api-a-a7b600-openshift-default-c88cb89b8-jxm2q"
Sep 19 01:36:31.937273 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:31.937248    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a7b600-openshift-default-c88cb89b8-jxm2q"
Sep 19 01:36:31.984106 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:31.984029    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-8e708f/distinct-2-w0-8e708f-kserve-59b54cd666-tg2h7" podUID="9430ca55-a991-45b3-840c-9fc6fe68d4d6" containerName="main" containerID="cri-o://e6b1ad4704cec6fa600617e539c79d9ac3186e3942d205c25630d6c277e2931a" gracePeriod=45
Sep 19 01:36:31.999556 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:31.999391    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-8e708f/distinct-w0-8e708f-kserve-6f58868fbf-mfmbr" podUID="99de318b-4f35-4f13-8a91-099ad5aaa66a" containerName="main" containerID="cri-o://77472ae496c2dfc75e7d36ffaf6ee6941a6f4d3fbd82a6295f9502b34edb4869" gracePeriod=45
Sep 19 01:36:32.023819 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.023772    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-8e708f/embedding-w0-8e708f-kserve-c6b4956b6-bkg7s" podUID="aaed025d-9218-4949-b6b3-80c4f2a16699" containerName="main" containerID="cri-o://07ec0ad7c9841b43d6b4af7a0cd8ae6bcb2f52515612c4881ae0bcf7facb7c2d" gracePeriod=45
Sep 19 01:36:32.082008 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.081978    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-8e708f/simulator-w0-8e708f-kserve-5f9cd6679f-6ncxb" podUID="6fdc64e9-7d4e-40a9-a8b4-a243f5ef5df8" containerName="main" containerID="cri-o://549a712dae5e7fb5f0e9e85ab8108a06db6fe0cd53778bfa8711c7337b9f50b3" gracePeriod=45
Sep 19 01:36:32.223230 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.223201    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a7b600-openshift-default-c88cb89b8-jxm2q"]
Sep 19 01:36:32.226781 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:36:32.226749    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod461f7ac7_05b6_412d_8d17_6d0a46f17fc2.slice/crio-76a246d6f9dce647b92a6cfe88dc72a5259a0c18e2d9dad164d1f0f07c1ce322 WatchSource:0}: Error finding container 76a246d6f9dce647b92a6cfe88dc72a5259a0c18e2d9dad164d1f0f07c1ce322: Status 404 returned error can't find the container with id 76a246d6f9dce647b92a6cfe88dc72a5259a0c18e2d9dad164d1f0f07c1ce322
Sep 19 01:36:32.229761 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.229722    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 19 01:36:32.229913 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.229838    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 19 01:36:32.229999 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.229922    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 19 01:36:32.352407 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.352283    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-8e708f/distinct-w0-8e708f-kserve-6f58868fbf-mfmbr"
Sep 19 01:36:32.363538 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.363509    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-8e708f/distinct-2-w0-8e708f-kserve-59b54cd666-tg2h7"
Sep 19 01:36:32.415373 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.415349    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-8e708f/embedding-w0-8e708f-kserve-c6b4956b6-bkg7s"
Sep 19 01:36:32.421018 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.420978    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-8e708f/simulator-w0-8e708f-kserve-5f9cd6679f-6ncxb"
Sep 19 01:36:32.491393 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.491358    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9430ca55-a991-45b3-840c-9fc6fe68d4d6-tmp-dir\") pod \"9430ca55-a991-45b3-840c-9fc6fe68d4d6\" (UID: \"9430ca55-a991-45b3-840c-9fc6fe68d4d6\") "
Sep 19 01:36:32.491849 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.491418    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n7bnb\" (UniqueName: \"kubernetes.io/projected/9430ca55-a991-45b3-840c-9fc6fe68d4d6-kube-api-access-n7bnb\") pod \"9430ca55-a991-45b3-840c-9fc6fe68d4d6\" (UID: \"9430ca55-a991-45b3-840c-9fc6fe68d4d6\") "
Sep 19 01:36:32.491849 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.491497    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99de318b-4f35-4f13-8a91-099ad5aaa66a-home\") pod \"99de318b-4f35-4f13-8a91-099ad5aaa66a\" (UID: \"99de318b-4f35-4f13-8a91-099ad5aaa66a\") "
Sep 19 01:36:32.491849 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.491529    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99de318b-4f35-4f13-8a91-099ad5aaa66a-dshm\") pod \"99de318b-4f35-4f13-8a91-099ad5aaa66a\" (UID: \"99de318b-4f35-4f13-8a91-099ad5aaa66a\") "
Sep 19 01:36:32.491849 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.491561    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dbglr\" (UniqueName: \"kubernetes.io/projected/99de318b-4f35-4f13-8a91-099ad5aaa66a-kube-api-access-dbglr\") pod \"99de318b-4f35-4f13-8a91-099ad5aaa66a\" (UID: \"99de318b-4f35-4f13-8a91-099ad5aaa66a\") "
Sep 19 01:36:32.491849 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.491592    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9430ca55-a991-45b3-840c-9fc6fe68d4d6-model-cache\") pod \"9430ca55-a991-45b3-840c-9fc6fe68d4d6\" (UID: \"9430ca55-a991-45b3-840c-9fc6fe68d4d6\") "
Sep 19 01:36:32.491849 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.491623    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99de318b-4f35-4f13-8a91-099ad5aaa66a-model-cache\") pod \"99de318b-4f35-4f13-8a91-099ad5aaa66a\" (UID: \"99de318b-4f35-4f13-8a91-099ad5aaa66a\") "
Sep 19 01:36:32.491849 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.491654    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99de318b-4f35-4f13-8a91-099ad5aaa66a-tls-certs\") pod \"99de318b-4f35-4f13-8a91-099ad5aaa66a\" (UID: \"99de318b-4f35-4f13-8a91-099ad5aaa66a\") "
Sep 19 01:36:32.491849 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.491687    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9430ca55-a991-45b3-840c-9fc6fe68d4d6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9430ca55-a991-45b3-840c-9fc6fe68d4d6" (UID: "9430ca55-a991-45b3-840c-9fc6fe68d4d6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 01:36:32.491849 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.491708    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9430ca55-a991-45b3-840c-9fc6fe68d4d6-dshm\") pod \"9430ca55-a991-45b3-840c-9fc6fe68d4d6\" (UID: \"9430ca55-a991-45b3-840c-9fc6fe68d4d6\") "
Sep 19 01:36:32.491849 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.491756    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9430ca55-a991-45b3-840c-9fc6fe68d4d6-home\") pod \"9430ca55-a991-45b3-840c-9fc6fe68d4d6\" (UID: \"9430ca55-a991-45b3-840c-9fc6fe68d4d6\") "
Sep 19 01:36:32.491849 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.491795    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99de318b-4f35-4f13-8a91-099ad5aaa66a-tmp-dir\") pod \"99de318b-4f35-4f13-8a91-099ad5aaa66a\" (UID: \"99de318b-4f35-4f13-8a91-099ad5aaa66a\") "
Sep 19 01:36:32.491849 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.491827    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9430ca55-a991-45b3-840c-9fc6fe68d4d6-tls-certs\") pod \"9430ca55-a991-45b3-840c-9fc6fe68d4d6\" (UID: \"9430ca55-a991-45b3-840c-9fc6fe68d4d6\") "
Sep 19 01:36:32.492451 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.492167    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9430ca55-a991-45b3-840c-9fc6fe68d4d6-tmp-dir\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:36:32.492550 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.492524    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99de318b-4f35-4f13-8a91-099ad5aaa66a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "99de318b-4f35-4f13-8a91-099ad5aaa66a" (UID: "99de318b-4f35-4f13-8a91-099ad5aaa66a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 01:36:32.492850 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.492823    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99de318b-4f35-4f13-8a91-099ad5aaa66a-home" (OuterVolumeSpecName: "home") pod "99de318b-4f35-4f13-8a91-099ad5aaa66a" (UID: "99de318b-4f35-4f13-8a91-099ad5aaa66a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 01:36:32.493483 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.493087    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9430ca55-a991-45b3-840c-9fc6fe68d4d6-model-cache" (OuterVolumeSpecName: "model-cache") pod "9430ca55-a991-45b3-840c-9fc6fe68d4d6" (UID: "9430ca55-a991-45b3-840c-9fc6fe68d4d6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 01:36:32.493483 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.493272    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99de318b-4f35-4f13-8a91-099ad5aaa66a-model-cache" (OuterVolumeSpecName: "model-cache") pod "99de318b-4f35-4f13-8a91-099ad5aaa66a" (UID: "99de318b-4f35-4f13-8a91-099ad5aaa66a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 01:36:32.494177 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.494156    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9430ca55-a991-45b3-840c-9fc6fe68d4d6-home" (OuterVolumeSpecName: "home") pod "9430ca55-a991-45b3-840c-9fc6fe68d4d6" (UID: "9430ca55-a991-45b3-840c-9fc6fe68d4d6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 01:36:32.494672 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.494638    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9430ca55-a991-45b3-840c-9fc6fe68d4d6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9430ca55-a991-45b3-840c-9fc6fe68d4d6" (UID: "9430ca55-a991-45b3-840c-9fc6fe68d4d6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 01:36:32.495552 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.495514    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99de318b-4f35-4f13-8a91-099ad5aaa66a-kube-api-access-dbglr" (OuterVolumeSpecName: "kube-api-access-dbglr") pod "99de318b-4f35-4f13-8a91-099ad5aaa66a" (UID: "99de318b-4f35-4f13-8a91-099ad5aaa66a"). InnerVolumeSpecName "kube-api-access-dbglr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 01:36:32.495655 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.495587    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9430ca55-a991-45b3-840c-9fc6fe68d4d6-kube-api-access-n7bnb" (OuterVolumeSpecName: "kube-api-access-n7bnb") pod "9430ca55-a991-45b3-840c-9fc6fe68d4d6" (UID: "9430ca55-a991-45b3-840c-9fc6fe68d4d6"). InnerVolumeSpecName "kube-api-access-n7bnb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 01:36:32.497028 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.497002    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99de318b-4f35-4f13-8a91-099ad5aaa66a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "99de318b-4f35-4f13-8a91-099ad5aaa66a" (UID: "99de318b-4f35-4f13-8a91-099ad5aaa66a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 01:36:32.497289 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.497265    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9430ca55-a991-45b3-840c-9fc6fe68d4d6-dshm" (OuterVolumeSpecName: "dshm") pod "9430ca55-a991-45b3-840c-9fc6fe68d4d6" (UID: "9430ca55-a991-45b3-840c-9fc6fe68d4d6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 01:36:32.497376 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.497265    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99de318b-4f35-4f13-8a91-099ad5aaa66a-dshm" (OuterVolumeSpecName: "dshm") pod "99de318b-4f35-4f13-8a91-099ad5aaa66a" (UID: "99de318b-4f35-4f13-8a91-099ad5aaa66a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 01:36:32.592631 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.592602    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fdc64e9-7d4e-40a9-a8b4-a243f5ef5df8-tls-certs\") pod \"6fdc64e9-7d4e-40a9-a8b4-a243f5ef5df8\" (UID: \"6fdc64e9-7d4e-40a9-a8b4-a243f5ef5df8\") "
Sep 19 01:36:32.592974 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.592647    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aaed025d-9218-4949-b6b3-80c4f2a16699-tmp-dir\") pod \"aaed025d-9218-4949-b6b3-80c4f2a16699\" (UID: \"aaed025d-9218-4949-b6b3-80c4f2a16699\") "
Sep 19 01:36:32.592974 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.592710    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aaed025d-9218-4949-b6b3-80c4f2a16699-home\") pod \"aaed025d-9218-4949-b6b3-80c4f2a16699\" (UID: \"aaed025d-9218-4949-b6b3-80c4f2a16699\") "
Sep 19 01:36:32.592974 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.592763    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6fdc64e9-7d4e-40a9-a8b4-a243f5ef5df8-home\") pod \"6fdc64e9-7d4e-40a9-a8b4-a243f5ef5df8\" (UID: \"6fdc64e9-7d4e-40a9-a8b4-a243f5ef5df8\") "
Sep 19 01:36:32.592974 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.592826    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sb8vx\" (UniqueName: \"kubernetes.io/projected/6fdc64e9-7d4e-40a9-a8b4-a243f5ef5df8-kube-api-access-sb8vx\") pod \"6fdc64e9-7d4e-40a9-a8b4-a243f5ef5df8\" (UID: \"6fdc64e9-7d4e-40a9-a8b4-a243f5ef5df8\") "
Sep 19 01:36:32.592974 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.592889    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dr86l\" (UniqueName: \"kubernetes.io/projected/aaed025d-9218-4949-b6b3-80c4f2a16699-kube-api-access-dr86l\") pod \"aaed025d-9218-4949-b6b3-80c4f2a16699\" (UID: \"aaed025d-9218-4949-b6b3-80c4f2a16699\") "
Sep 19 01:36:32.592974 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.592910    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaed025d-9218-4949-b6b3-80c4f2a16699-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "aaed025d-9218-4949-b6b3-80c4f2a16699" (UID: "aaed025d-9218-4949-b6b3-80c4f2a16699"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 01:36:32.592974 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.592926    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aaed025d-9218-4949-b6b3-80c4f2a16699-tls-certs\") pod \"aaed025d-9218-4949-b6b3-80c4f2a16699\" (UID: \"aaed025d-9218-4949-b6b3-80c4f2a16699\") "
Sep 19 01:36:32.593505 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.592977    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaed025d-9218-4949-b6b3-80c4f2a16699-home" (OuterVolumeSpecName: "home") pod "aaed025d-9218-4949-b6b3-80c4f2a16699" (UID: "aaed025d-9218-4949-b6b3-80c4f2a16699"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 01:36:32.593505 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.592991    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fdc64e9-7d4e-40a9-a8b4-a243f5ef5df8-home" (OuterVolumeSpecName: "home") pod "6fdc64e9-7d4e-40a9-a8b4-a243f5ef5df8" (UID: "6fdc64e9-7d4e-40a9-a8b4-a243f5ef5df8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 01:36:32.593505 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.592986    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6fdc64e9-7d4e-40a9-a8b4-a243f5ef5df8-model-cache\") pod \"6fdc64e9-7d4e-40a9-a8b4-a243f5ef5df8\" (UID: \"6fdc64e9-7d4e-40a9-a8b4-a243f5ef5df8\") "
Sep 19 01:36:32.593505 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.593052    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6fdc64e9-7d4e-40a9-a8b4-a243f5ef5df8-tmp-dir\") pod \"6fdc64e9-7d4e-40a9-a8b4-a243f5ef5df8\" (UID: \"6fdc64e9-7d4e-40a9-a8b4-a243f5ef5df8\") "
Sep 19 01:36:32.593505 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.593130    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aaed025d-9218-4949-b6b3-80c4f2a16699-dshm\") pod \"aaed025d-9218-4949-b6b3-80c4f2a16699\" (UID: \"aaed025d-9218-4949-b6b3-80c4f2a16699\") "
Sep 19 01:36:32.593505 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.593153    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fdc64e9-7d4e-40a9-a8b4-a243f5ef5df8-model-cache" (OuterVolumeSpecName: "model-cache") pod "6fdc64e9-7d4e-40a9-a8b4-a243f5ef5df8" (UID: "6fdc64e9-7d4e-40a9-a8b4-a243f5ef5df8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 01:36:32.593505 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.593171    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6fdc64e9-7d4e-40a9-a8b4-a243f5ef5df8-dshm\") pod \"6fdc64e9-7d4e-40a9-a8b4-a243f5ef5df8\" (UID: \"6fdc64e9-7d4e-40a9-a8b4-a243f5ef5df8\") "
Sep 19 01:36:32.593505 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.593196    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aaed025d-9218-4949-b6b3-80c4f2a16699-model-cache\") pod \"aaed025d-9218-4949-b6b3-80c4f2a16699\" (UID: \"aaed025d-9218-4949-b6b3-80c4f2a16699\") "
Sep 19 01:36:32.593505 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.593294    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fdc64e9-7d4e-40a9-a8b4-a243f5ef5df8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6fdc64e9-7d4e-40a9-a8b4-a243f5ef5df8" (UID: "6fdc64e9-7d4e-40a9-a8b4-a243f5ef5df8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 01:36:32.594001 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.593514    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99de318b-4f35-4f13-8a91-099ad5aaa66a-model-cache\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:36:32.594001 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.593529    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99de318b-4f35-4f13-8a91-099ad5aaa66a-tls-certs\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:36:32.594001 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.593541    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9430ca55-a991-45b3-840c-9fc6fe68d4d6-dshm\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:36:32.594001 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.593552    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aaed025d-9218-4949-b6b3-80c4f2a16699-tmp-dir\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:36:32.594001 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.593564    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aaed025d-9218-4949-b6b3-80c4f2a16699-home\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:36:32.594001 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.593575    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6fdc64e9-7d4e-40a9-a8b4-a243f5ef5df8-home\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:36:32.594001 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.593586    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9430ca55-a991-45b3-840c-9fc6fe68d4d6-home\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:36:32.594001 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.593597    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99de318b-4f35-4f13-8a91-099ad5aaa66a-tmp-dir\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:36:32.594001 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.593609    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9430ca55-a991-45b3-840c-9fc6fe68d4d6-tls-certs\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:36:32.594001 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.593620    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6fdc64e9-7d4e-40a9-a8b4-a243f5ef5df8-model-cache\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:36:32.594001 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.593631    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6fdc64e9-7d4e-40a9-a8b4-a243f5ef5df8-tmp-dir\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:36:32.594001 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.593644    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7bnb\" (UniqueName: \"kubernetes.io/projected/9430ca55-a991-45b3-840c-9fc6fe68d4d6-kube-api-access-n7bnb\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:36:32.594001 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.593656    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99de318b-4f35-4f13-8a91-099ad5aaa66a-home\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:36:32.594001 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.593667    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99de318b-4f35-4f13-8a91-099ad5aaa66a-dshm\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:36:32.594001 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.593679    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbglr\" (UniqueName: \"kubernetes.io/projected/99de318b-4f35-4f13-8a91-099ad5aaa66a-kube-api-access-dbglr\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:36:32.594001 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.593693    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9430ca55-a991-45b3-840c-9fc6fe68d4d6-model-cache\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:36:32.594001 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.593840    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaed025d-9218-4949-b6b3-80c4f2a16699-model-cache" (OuterVolumeSpecName: "model-cache") pod "aaed025d-9218-4949-b6b3-80c4f2a16699" (UID: "aaed025d-9218-4949-b6b3-80c4f2a16699"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 01:36:32.595797 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.595749    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fdc64e9-7d4e-40a9-a8b4-a243f5ef5df8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6fdc64e9-7d4e-40a9-a8b4-a243f5ef5df8" (UID: "6fdc64e9-7d4e-40a9-a8b4-a243f5ef5df8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 01:36:32.595919 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.595809    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aaed025d-9218-4949-b6b3-80c4f2a16699-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "aaed025d-9218-4949-b6b3-80c4f2a16699" (UID: "aaed025d-9218-4949-b6b3-80c4f2a16699"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 01:36:32.595919 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.595852    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaed025d-9218-4949-b6b3-80c4f2a16699-kube-api-access-dr86l" (OuterVolumeSpecName: "kube-api-access-dr86l") pod "aaed025d-9218-4949-b6b3-80c4f2a16699" (UID: "aaed025d-9218-4949-b6b3-80c4f2a16699"). InnerVolumeSpecName "kube-api-access-dr86l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 01:36:32.596033 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.596008    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fdc64e9-7d4e-40a9-a8b4-a243f5ef5df8-dshm" (OuterVolumeSpecName: "dshm") pod "6fdc64e9-7d4e-40a9-a8b4-a243f5ef5df8" (UID: "6fdc64e9-7d4e-40a9-a8b4-a243f5ef5df8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 01:36:32.596158 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.596140    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaed025d-9218-4949-b6b3-80c4f2a16699-dshm" (OuterVolumeSpecName: "dshm") pod "aaed025d-9218-4949-b6b3-80c4f2a16699" (UID: "aaed025d-9218-4949-b6b3-80c4f2a16699"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 01:36:32.596817 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.596798    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fdc64e9-7d4e-40a9-a8b4-a243f5ef5df8-kube-api-access-sb8vx" (OuterVolumeSpecName: "kube-api-access-sb8vx") pod "6fdc64e9-7d4e-40a9-a8b4-a243f5ef5df8" (UID: "6fdc64e9-7d4e-40a9-a8b4-a243f5ef5df8"). InnerVolumeSpecName "kube-api-access-sb8vx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 01:36:32.694660 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.694639    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aaed025d-9218-4949-b6b3-80c4f2a16699-dshm\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:36:32.694660 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.694658    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6fdc64e9-7d4e-40a9-a8b4-a243f5ef5df8-dshm\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:36:32.694778 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.694666    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aaed025d-9218-4949-b6b3-80c4f2a16699-model-cache\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:36:32.694778 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.694676    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fdc64e9-7d4e-40a9-a8b4-a243f5ef5df8-tls-certs\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:36:32.694778 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.694685    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sb8vx\" (UniqueName: \"kubernetes.io/projected/6fdc64e9-7d4e-40a9-a8b4-a243f5ef5df8-kube-api-access-sb8vx\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:36:32.694778 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.694694    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dr86l\" (UniqueName: \"kubernetes.io/projected/aaed025d-9218-4949-b6b3-80c4f2a16699-kube-api-access-dr86l\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:36:32.694778 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.694701    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aaed025d-9218-4949-b6b3-80c4f2a16699-tls-certs\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:36:32.738722 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.738689    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a7b600-openshift-default-c88cb89b8-jxm2q" event={"ID":"461f7ac7-05b6-412d-8d17-6d0a46f17fc2","Type":"ContainerStarted","Data":"8f96814090cda0bb2759796fee43ab4933b836f613ab9364bfb62328725e9e74"}
Sep 19 01:36:32.738825 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.738742    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a7b600-openshift-default-c88cb89b8-jxm2q" event={"ID":"461f7ac7-05b6-412d-8d17-6d0a46f17fc2","Type":"ContainerStarted","Data":"76a246d6f9dce647b92a6cfe88dc72a5259a0c18e2d9dad164d1f0f07c1ce322"}
Sep 19 01:36:32.740108 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.740085    2581 generic.go:356] "Generic (PLEG): container finished" podID="6fdc64e9-7d4e-40a9-a8b4-a243f5ef5df8" containerID="549a712dae5e7fb5f0e9e85ab8108a06db6fe0cd53778bfa8711c7337b9f50b3" exitCode=0
Sep 19 01:36:32.740216 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.740130    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-8e708f/simulator-w0-8e708f-kserve-5f9cd6679f-6ncxb"
Sep 19 01:36:32.740216 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.740146    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-8e708f/simulator-w0-8e708f-kserve-5f9cd6679f-6ncxb" event={"ID":"6fdc64e9-7d4e-40a9-a8b4-a243f5ef5df8","Type":"ContainerDied","Data":"549a712dae5e7fb5f0e9e85ab8108a06db6fe0cd53778bfa8711c7337b9f50b3"}
Sep 19 01:36:32.740216 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.740185    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-8e708f/simulator-w0-8e708f-kserve-5f9cd6679f-6ncxb" event={"ID":"6fdc64e9-7d4e-40a9-a8b4-a243f5ef5df8","Type":"ContainerDied","Data":"02dbf6609d85109b1c9c40d5827ad1b5dd2bcc9d5bda554c6888426fc6963c65"}
Sep 19 01:36:32.740216 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.740206    2581 scope.go:117] "RemoveContainer" containerID="549a712dae5e7fb5f0e9e85ab8108a06db6fe0cd53778bfa8711c7337b9f50b3"
Sep 19 01:36:32.741585 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.741539    2581 generic.go:356] "Generic (PLEG): container finished" podID="9430ca55-a991-45b3-840c-9fc6fe68d4d6" containerID="e6b1ad4704cec6fa600617e539c79d9ac3186e3942d205c25630d6c277e2931a" exitCode=0
Sep 19 01:36:32.741651 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.741595    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-8e708f/distinct-2-w0-8e708f-kserve-59b54cd666-tg2h7"
Sep 19 01:36:32.741651 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.741631    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-8e708f/distinct-2-w0-8e708f-kserve-59b54cd666-tg2h7" event={"ID":"9430ca55-a991-45b3-840c-9fc6fe68d4d6","Type":"ContainerDied","Data":"e6b1ad4704cec6fa600617e539c79d9ac3186e3942d205c25630d6c277e2931a"}
Sep 19 01:36:32.741764 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.741661    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-8e708f/distinct-2-w0-8e708f-kserve-59b54cd666-tg2h7" event={"ID":"9430ca55-a991-45b3-840c-9fc6fe68d4d6","Type":"ContainerDied","Data":"aa0bab60b71c3d906c5bfc5a3e44576fc01b64b78d8c11f1ddbf4d8c2d16e2b1"}
Sep 19 01:36:32.743376 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.743356    2581 generic.go:356] "Generic (PLEG): container finished" podID="aaed025d-9218-4949-b6b3-80c4f2a16699" containerID="07ec0ad7c9841b43d6b4af7a0cd8ae6bcb2f52515612c4881ae0bcf7facb7c2d" exitCode=0
Sep 19 01:36:32.743491 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.743415    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-8e708f/embedding-w0-8e708f-kserve-c6b4956b6-bkg7s"
Sep 19 01:36:32.743491 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.743444    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-8e708f/embedding-w0-8e708f-kserve-c6b4956b6-bkg7s" event={"ID":"aaed025d-9218-4949-b6b3-80c4f2a16699","Type":"ContainerDied","Data":"07ec0ad7c9841b43d6b4af7a0cd8ae6bcb2f52515612c4881ae0bcf7facb7c2d"}
Sep 19 01:36:32.743491 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.743471    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-8e708f/embedding-w0-8e708f-kserve-c6b4956b6-bkg7s" event={"ID":"aaed025d-9218-4949-b6b3-80c4f2a16699","Type":"ContainerDied","Data":"525d1bc9bf56f746e9acb27315d30cb52e8e42eefb3435d24c50cedf49296724"}
Sep 19 01:36:32.745057 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.745032    2581 generic.go:356] "Generic (PLEG): container finished" podID="99de318b-4f35-4f13-8a91-099ad5aaa66a" containerID="77472ae496c2dfc75e7d36ffaf6ee6941a6f4d3fbd82a6295f9502b34edb4869" exitCode=0
Sep 19 01:36:32.745149 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.745122    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-8e708f/distinct-w0-8e708f-kserve-6f58868fbf-mfmbr" event={"ID":"99de318b-4f35-4f13-8a91-099ad5aaa66a","Type":"ContainerDied","Data":"77472ae496c2dfc75e7d36ffaf6ee6941a6f4d3fbd82a6295f9502b34edb4869"}
Sep 19 01:36:32.745205 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.745146    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-8e708f/distinct-w0-8e708f-kserve-6f58868fbf-mfmbr" event={"ID":"99de318b-4f35-4f13-8a91-099ad5aaa66a","Type":"ContainerDied","Data":"a8bd3de27fa25c904d5b4078c5347b69b9e24e6d8a0547dd27cbb8d1a43c84c6"}
Sep 19 01:36:32.745205 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.745195    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-8e708f/distinct-w0-8e708f-kserve-6f58868fbf-mfmbr"
Sep 19 01:36:32.753160 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.753139    2581 scope.go:117] "RemoveContainer" containerID="549a712dae5e7fb5f0e9e85ab8108a06db6fe0cd53778bfa8711c7337b9f50b3"
Sep 19 01:36:32.753993 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:36:32.753966    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"549a712dae5e7fb5f0e9e85ab8108a06db6fe0cd53778bfa8711c7337b9f50b3\": container with ID starting with 549a712dae5e7fb5f0e9e85ab8108a06db6fe0cd53778bfa8711c7337b9f50b3 not found: ID does not exist" containerID="549a712dae5e7fb5f0e9e85ab8108a06db6fe0cd53778bfa8711c7337b9f50b3"
Sep 19 01:36:32.754154 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.754128    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"549a712dae5e7fb5f0e9e85ab8108a06db6fe0cd53778bfa8711c7337b9f50b3"} err="failed to get container status \"549a712dae5e7fb5f0e9e85ab8108a06db6fe0cd53778bfa8711c7337b9f50b3\": rpc error: code = NotFound desc = could not find container \"549a712dae5e7fb5f0e9e85ab8108a06db6fe0cd53778bfa8711c7337b9f50b3\": container with ID starting with 549a712dae5e7fb5f0e9e85ab8108a06db6fe0cd53778bfa8711c7337b9f50b3 not found: ID does not exist"
Sep 19 01:36:32.754265 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.754251    2581 scope.go:117] "RemoveContainer" containerID="e6b1ad4704cec6fa600617e539c79d9ac3186e3942d205c25630d6c277e2931a"
Sep 19 01:36:32.763414 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.763351    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-a7b600-openshift-default-c88cb89b8-jxm2q" podStartSLOduration=1.7633395630000002 podStartE2EDuration="1.763339563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 01:36:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 01:36:32.761436366 +0000 UTC m=+1443.649105442" watchObservedRunningTime="2026-09-19 01:36:32.763339563 +0000 UTC m=+1443.651008642"
Sep 19 01:36:32.766952 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.766926    2581 scope.go:117] "RemoveContainer" containerID="e6b1ad4704cec6fa600617e539c79d9ac3186e3942d205c25630d6c277e2931a"
Sep 19 01:36:32.767260 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:36:32.767241    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6b1ad4704cec6fa600617e539c79d9ac3186e3942d205c25630d6c277e2931a\": container with ID starting with e6b1ad4704cec6fa600617e539c79d9ac3186e3942d205c25630d6c277e2931a not found: ID does not exist" containerID="e6b1ad4704cec6fa600617e539c79d9ac3186e3942d205c25630d6c277e2931a"
Sep 19 01:36:32.767333 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.767267    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6b1ad4704cec6fa600617e539c79d9ac3186e3942d205c25630d6c277e2931a"} err="failed to get container status \"e6b1ad4704cec6fa600617e539c79d9ac3186e3942d205c25630d6c277e2931a\": rpc error: code = NotFound desc = could not find container \"e6b1ad4704cec6fa600617e539c79d9ac3186e3942d205c25630d6c277e2931a\": container with ID starting with e6b1ad4704cec6fa600617e539c79d9ac3186e3942d205c25630d6c277e2931a not found: ID does not exist"
Sep 19 01:36:32.767333 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.767283    2581 scope.go:117] "RemoveContainer" containerID="07ec0ad7c9841b43d6b4af7a0cd8ae6bcb2f52515612c4881ae0bcf7facb7c2d"
Sep 19 01:36:32.775093 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.775073    2581 scope.go:117] "RemoveContainer" containerID="07ec0ad7c9841b43d6b4af7a0cd8ae6bcb2f52515612c4881ae0bcf7facb7c2d"
Sep 19 01:36:32.775359 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:36:32.775341    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07ec0ad7c9841b43d6b4af7a0cd8ae6bcb2f52515612c4881ae0bcf7facb7c2d\": container with ID starting with 07ec0ad7c9841b43d6b4af7a0cd8ae6bcb2f52515612c4881ae0bcf7facb7c2d not found: ID does not exist" containerID="07ec0ad7c9841b43d6b4af7a0cd8ae6bcb2f52515612c4881ae0bcf7facb7c2d"
Sep 19 01:36:32.775420 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.775365    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07ec0ad7c9841b43d6b4af7a0cd8ae6bcb2f52515612c4881ae0bcf7facb7c2d"} err="failed to get container status \"07ec0ad7c9841b43d6b4af7a0cd8ae6bcb2f52515612c4881ae0bcf7facb7c2d\": rpc error: code = NotFound desc = could not find container \"07ec0ad7c9841b43d6b4af7a0cd8ae6bcb2f52515612c4881ae0bcf7facb7c2d\": container with ID starting with 07ec0ad7c9841b43d6b4af7a0cd8ae6bcb2f52515612c4881ae0bcf7facb7c2d not found: ID does not exist"
Sep 19 01:36:32.775420 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.775380    2581 scope.go:117] "RemoveContainer" containerID="77472ae496c2dfc75e7d36ffaf6ee6941a6f4d3fbd82a6295f9502b34edb4869"
Sep 19 01:36:32.782777 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.782754    2581 scope.go:117] "RemoveContainer" containerID="77472ae496c2dfc75e7d36ffaf6ee6941a6f4d3fbd82a6295f9502b34edb4869"
Sep 19 01:36:32.783070 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:36:32.783052    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77472ae496c2dfc75e7d36ffaf6ee6941a6f4d3fbd82a6295f9502b34edb4869\": container with ID starting with 77472ae496c2dfc75e7d36ffaf6ee6941a6f4d3fbd82a6295f9502b34edb4869 not found: ID does not exist" containerID="77472ae496c2dfc75e7d36ffaf6ee6941a6f4d3fbd82a6295f9502b34edb4869"
Sep 19 01:36:32.783134 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.783085    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77472ae496c2dfc75e7d36ffaf6ee6941a6f4d3fbd82a6295f9502b34edb4869"} err="failed to get container status \"77472ae496c2dfc75e7d36ffaf6ee6941a6f4d3fbd82a6295f9502b34edb4869\": rpc error: code = NotFound desc = could not find container \"77472ae496c2dfc75e7d36ffaf6ee6941a6f4d3fbd82a6295f9502b34edb4869\": container with ID starting with 77472ae496c2dfc75e7d36ffaf6ee6941a6f4d3fbd82a6295f9502b34edb4869 not found: ID does not exist"
Sep 19 01:36:32.790392 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.790372    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-8e708f/simulator-w0-8e708f-kserve-5f9cd6679f-6ncxb"]
Sep 19 01:36:32.792597 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.792576    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-8e708f/simulator-w0-8e708f-kserve-5f9cd6679f-6ncxb"]
Sep 19 01:36:32.804106 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.804085    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-8e708f/distinct-w0-8e708f-kserve-6f58868fbf-mfmbr"]
Sep 19 01:36:32.814251 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.814232    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-8e708f/distinct-w0-8e708f-kserve-6f58868fbf-mfmbr"]
Sep 19 01:36:32.825478 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.825450    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-8e708f/embedding-w0-8e708f-kserve-c6b4956b6-bkg7s"]
Sep 19 01:36:32.832958 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.832939    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-8e708f/embedding-w0-8e708f-kserve-c6b4956b6-bkg7s"]
Sep 19 01:36:32.845206 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.845153    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-8e708f/distinct-2-w0-8e708f-kserve-59b54cd666-tg2h7"]
Sep 19 01:36:32.848637 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.848618    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-8e708f/distinct-2-w0-8e708f-kserve-59b54cd666-tg2h7"]
Sep 19 01:36:32.937927 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:32.937906    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-a7b600-openshift-default-c88cb89b8-jxm2q"
Sep 19 01:36:33.272025 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:33.271948    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6fdc64e9-7d4e-40a9-a8b4-a243f5ef5df8" path="/var/lib/kubelet/pods/6fdc64e9-7d4e-40a9-a8b4-a243f5ef5df8/volumes"
Sep 19 01:36:33.272397 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:33.272375    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9430ca55-a991-45b3-840c-9fc6fe68d4d6" path="/var/lib/kubelet/pods/9430ca55-a991-45b3-840c-9fc6fe68d4d6/volumes"
Sep 19 01:36:33.272794 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:33.272776    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99de318b-4f35-4f13-8a91-099ad5aaa66a" path="/var/lib/kubelet/pods/99de318b-4f35-4f13-8a91-099ad5aaa66a/volumes"
Sep 19 01:36:33.273215 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:33.273197    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aaed025d-9218-4949-b6b3-80c4f2a16699" path="/var/lib/kubelet/pods/aaed025d-9218-4949-b6b3-80c4f2a16699/volumes"
Sep 19 01:36:33.942960 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:33.942933    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-a7b600-openshift-default-c88cb89b8-jxm2q"
Sep 19 01:36:34.758629 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:34.758602    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-a7b600-openshift-default-c88cb89b8-jxm2q"
Sep 19 01:36:34.759529 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:34.759507    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-a7b600-openshift-default-c88cb89b8-jxm2q"
Sep 19 01:36:39.088108 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:39.088074    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a7b600-79995f4d78-clb6d"]
Sep 19 01:36:39.088567 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:39.088420    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99de318b-4f35-4f13-8a91-099ad5aaa66a" containerName="main"
Sep 19 01:36:39.088567 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:39.088431    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="99de318b-4f35-4f13-8a91-099ad5aaa66a" containerName="main"
Sep 19 01:36:39.088567 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:39.088444    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9430ca55-a991-45b3-840c-9fc6fe68d4d6" containerName="main"
Sep 19 01:36:39.088567 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:39.088450    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9430ca55-a991-45b3-840c-9fc6fe68d4d6" containerName="main"
Sep 19 01:36:39.088567 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:39.088460    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fdc64e9-7d4e-40a9-a8b4-a243f5ef5df8" containerName="main"
Sep 19 01:36:39.088567 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:39.088465    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fdc64e9-7d4e-40a9-a8b4-a243f5ef5df8" containerName="main"
Sep 19 01:36:39.088567 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:39.088476    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aaed025d-9218-4949-b6b3-80c4f2a16699" containerName="main"
Sep 19 01:36:39.088567 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:39.088481    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="aaed025d-9218-4949-b6b3-80c4f2a16699" containerName="main"
Sep 19 01:36:39.088567 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:39.088527    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9430ca55-a991-45b3-840c-9fc6fe68d4d6" containerName="main"
Sep 19 01:36:39.088567 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:39.088534    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="aaed025d-9218-4949-b6b3-80c4f2a16699" containerName="main"
Sep 19 01:36:39.088567 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:39.088541    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fdc64e9-7d4e-40a9-a8b4-a243f5ef5df8" containerName="main"
Sep 19 01:36:39.088567 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:39.088547    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="99de318b-4f35-4f13-8a91-099ad5aaa66a" containerName="main"
Sep 19 01:36:39.092844 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:39.092827    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a7b600-79995f4d78-clb6d"
Sep 19 01:36:39.103032 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:39.103016    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-a7b600\""
Sep 19 01:36:39.103251 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:39.103234    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-a7b600-dockercfg-gfvm8\""
Sep 19 01:36:39.112028 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:39.112006    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a7b600-79995f4d78-clb6d"]
Sep 19 01:36:39.119900 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:39.119878    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a7b600-d967c6dc-h4hw6"]
Sep 19 01:36:39.122839 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:39.122822    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a7b600-d967c6dc-h4hw6"
Sep 19 01:36:39.151999 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:39.151978    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c961b0a-4f34-4aee-9807-69d4e1f0e20b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-a7b600-d967c6dc-h4hw6\" (UID: \"2c961b0a-4f34-4aee-9807-69d4e1f0e20b\") " pod="openshift-ingress/payload-processing-e2e-api-a-a7b600-d967c6dc-h4hw6"
Sep 19 01:36:39.152096 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:39.152026    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wk7hl\" (UniqueName: \"kubernetes.io/projected/b34285f9-074c-4800-aab6-cda644d32a5e-kube-api-access-wk7hl\") pod \"payload-pre-processing-e2e-api-a-a7b600-79995f4d78-clb6d\" (UID: \"b34285f9-074c-4800-aab6-cda644d32a5e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a7b600-79995f4d78-clb6d"
Sep 19 01:36:39.152096 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:39.152057    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b34285f9-074c-4800-aab6-cda644d32a5e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a7b600-79995f4d78-clb6d\" (UID: \"b34285f9-074c-4800-aab6-cda644d32a5e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a7b600-79995f4d78-clb6d"
Sep 19 01:36:39.152193 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:39.152105    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwzth\" (UniqueName: \"kubernetes.io/projected/2c961b0a-4f34-4aee-9807-69d4e1f0e20b-kube-api-access-qwzth\") pod \"payload-processing-e2e-api-a-a7b600-d967c6dc-h4hw6\" (UID: \"2c961b0a-4f34-4aee-9807-69d4e1f0e20b\") " pod="openshift-ingress/payload-processing-e2e-api-a-a7b600-d967c6dc-h4hw6"
Sep 19 01:36:39.156504 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:39.156484    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a7b600-d967c6dc-h4hw6"]
Sep 19 01:36:39.253480 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:39.253446    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wk7hl\" (UniqueName: \"kubernetes.io/projected/b34285f9-074c-4800-aab6-cda644d32a5e-kube-api-access-wk7hl\") pod \"payload-pre-processing-e2e-api-a-a7b600-79995f4d78-clb6d\" (UID: \"b34285f9-074c-4800-aab6-cda644d32a5e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a7b600-79995f4d78-clb6d"
Sep 19 01:36:39.253680 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:39.253532    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b34285f9-074c-4800-aab6-cda644d32a5e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a7b600-79995f4d78-clb6d\" (UID: \"b34285f9-074c-4800-aab6-cda644d32a5e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a7b600-79995f4d78-clb6d"
Sep 19 01:36:39.253680 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:39.253574    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwzth\" (UniqueName: \"kubernetes.io/projected/2c961b0a-4f34-4aee-9807-69d4e1f0e20b-kube-api-access-qwzth\") pod \"payload-processing-e2e-api-a-a7b600-d967c6dc-h4hw6\" (UID: \"2c961b0a-4f34-4aee-9807-69d4e1f0e20b\") " pod="openshift-ingress/payload-processing-e2e-api-a-a7b600-d967c6dc-h4hw6"
Sep 19 01:36:39.253680 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:39.253614    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c961b0a-4f34-4aee-9807-69d4e1f0e20b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-a7b600-d967c6dc-h4hw6\" (UID: \"2c961b0a-4f34-4aee-9807-69d4e1f0e20b\") " pod="openshift-ingress/payload-processing-e2e-api-a-a7b600-d967c6dc-h4hw6"
Sep 19 01:36:39.254335 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:39.254310    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b34285f9-074c-4800-aab6-cda644d32a5e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a7b600-79995f4d78-clb6d\" (UID: \"b34285f9-074c-4800-aab6-cda644d32a5e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a7b600-79995f4d78-clb6d"
Sep 19 01:36:39.254438 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:39.254321    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c961b0a-4f34-4aee-9807-69d4e1f0e20b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-a7b600-d967c6dc-h4hw6\" (UID: \"2c961b0a-4f34-4aee-9807-69d4e1f0e20b\") " pod="openshift-ingress/payload-processing-e2e-api-a-a7b600-d967c6dc-h4hw6"
Sep 19 01:36:39.280956 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:39.280934    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwzth\" (UniqueName: \"kubernetes.io/projected/2c961b0a-4f34-4aee-9807-69d4e1f0e20b-kube-api-access-qwzth\") pod \"payload-processing-e2e-api-a-a7b600-d967c6dc-h4hw6\" (UID: \"2c961b0a-4f34-4aee-9807-69d4e1f0e20b\") " pod="openshift-ingress/payload-processing-e2e-api-a-a7b600-d967c6dc-h4hw6"
Sep 19 01:36:39.281080 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:39.281050    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wk7hl\" (UniqueName: \"kubernetes.io/projected/b34285f9-074c-4800-aab6-cda644d32a5e-kube-api-access-wk7hl\") pod \"payload-pre-processing-e2e-api-a-a7b600-79995f4d78-clb6d\" (UID: \"b34285f9-074c-4800-aab6-cda644d32a5e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a7b600-79995f4d78-clb6d"
Sep 19 01:36:39.402014 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:39.401988    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a7b600-79995f4d78-clb6d"
Sep 19 01:36:39.431899 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:39.431856    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a7b600-d967c6dc-h4hw6"
Sep 19 01:36:39.679696 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:39.679664    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a7b600-79995f4d78-clb6d"]
Sep 19 01:36:39.680721 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:36:39.680701    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb34285f9_074c_4800_aab6_cda644d32a5e.slice/crio-f0c4e4eea14d318cb94b55866aaa3bf074c4522d23fb1646ac92b5a4c9a9ec30 WatchSource:0}: Error finding container f0c4e4eea14d318cb94b55866aaa3bf074c4522d23fb1646ac92b5a4c9a9ec30: Status 404 returned error can't find the container with id f0c4e4eea14d318cb94b55866aaa3bf074c4522d23fb1646ac92b5a4c9a9ec30
Sep 19 01:36:39.716808 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:39.716786    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a7b600-d967c6dc-h4hw6"]
Sep 19 01:36:39.718364 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:36:39.718333    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c961b0a_4f34_4aee_9807_69d4e1f0e20b.slice/crio-9b1ab25183bacd6c32e160a197ceaebde6b526a80e9ff77ae75ea0887db252bb WatchSource:0}: Error finding container 9b1ab25183bacd6c32e160a197ceaebde6b526a80e9ff77ae75ea0887db252bb: Status 404 returned error can't find the container with id 9b1ab25183bacd6c32e160a197ceaebde6b526a80e9ff77ae75ea0887db252bb
Sep 19 01:36:39.784309 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:39.784278    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a7b600-79995f4d78-clb6d" event={"ID":"b34285f9-074c-4800-aab6-cda644d32a5e","Type":"ContainerStarted","Data":"c7107dbe02ed2c69a4b4ac5fe5a16468b816f50d8e6c0e1d676bcd32e5cc1e9e"}
Sep 19 01:36:39.784447 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:39.784332    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a7b600-79995f4d78-clb6d" event={"ID":"b34285f9-074c-4800-aab6-cda644d32a5e","Type":"ContainerStarted","Data":"f0c4e4eea14d318cb94b55866aaa3bf074c4522d23fb1646ac92b5a4c9a9ec30"}
Sep 19 01:36:39.784447 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:39.784363    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a7b600-79995f4d78-clb6d"
Sep 19 01:36:39.786220 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:39.786192    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a7b600-d967c6dc-h4hw6" event={"ID":"2c961b0a-4f34-4aee-9807-69d4e1f0e20b","Type":"ContainerStarted","Data":"d70e7bf9bfc87967c4f576a586881ae901995e73ca6b54ecbebc4c43c25e97cb"}
Sep 19 01:36:39.786326 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:39.786247    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-a7b600-d967c6dc-h4hw6"
Sep 19 01:36:39.786326 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:39.786272    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a7b600-d967c6dc-h4hw6" event={"ID":"2c961b0a-4f34-4aee-9807-69d4e1f0e20b","Type":"ContainerStarted","Data":"9b1ab25183bacd6c32e160a197ceaebde6b526a80e9ff77ae75ea0887db252bb"}
Sep 19 01:36:39.802773 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:39.802733    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a7b600-79995f4d78-clb6d" podStartSLOduration=0.802719616 podStartE2EDuration="802.719616ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 01:36:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 01:36:39.80099153 +0000 UTC m=+1450.688660638" watchObservedRunningTime="2026-09-19 01:36:39.802719616 +0000 UTC m=+1450.690388693"
Sep 19 01:36:39.821814 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:39.821763    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-a7b600-d967c6dc-h4hw6" podStartSLOduration=0.821743789 podStartE2EDuration="821.743789ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 01:36:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 01:36:39.819085324 +0000 UTC m=+1450.706754400" watchObservedRunningTime="2026-09-19 01:36:39.821743789 +0000 UTC m=+1450.709412868"
Sep 19 01:36:50.795054 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:50.795020    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-a7b600-d967c6dc-h4hw6"
Sep 19 01:36:50.795444 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:50.795087    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a7b600-79995f4d78-clb6d"
Sep 19 01:36:51.102883 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:36:51.102834    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c9nsz"]
Sep 19 01:37:01.913878 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:01.913840    2581 generic.go:356] "Generic (PLEG): container finished" podID="461f7ac7-05b6-412d-8d17-6d0a46f17fc2" containerID="8f96814090cda0bb2759796fee43ab4933b836f613ab9364bfb62328725e9e74" exitCode=0
Sep 19 01:37:01.914422 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:01.913921    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a7b600-openshift-default-c88cb89b8-jxm2q" event={"ID":"461f7ac7-05b6-412d-8d17-6d0a46f17fc2","Type":"ContainerDied","Data":"8f96814090cda0bb2759796fee43ab4933b836f613ab9364bfb62328725e9e74"}
Sep 19 01:37:01.914503 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:01.914425    2581 scope.go:117] "RemoveContainer" containerID="8f96814090cda0bb2759796fee43ab4933b836f613ab9364bfb62328725e9e74"
Sep 19 01:37:01.915659 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:01.915634    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 19 01:37:01.915724 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:01.915704    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 19 01:37:01.915759 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:01.915732    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 19 01:37:02.921580 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:02.921541    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a7b600-openshift-default-c88cb89b8-jxm2q" event={"ID":"461f7ac7-05b6-412d-8d17-6d0a46f17fc2","Type":"ContainerStarted","Data":"c27a8eef84c402a6b26c9fb1906c11b8c31c155acd96bbd8baa8fb07635a6da7"}
Sep 19 01:37:02.938305 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:02.938277    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-a7b600-openshift-default-c88cb89b8-jxm2q"
Sep 19 01:37:02.939472 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:02.939440    2581 patch_prober.go:28] interesting pod/e2e-api-a-a7b600-openshift-default-c88cb89b8-jxm2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Sep 19 01:37:02.939605 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:02.939501    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a7b600-openshift-default-c88cb89b8-jxm2q" podUID="461f7ac7-05b6-412d-8d17-6d0a46f17fc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Sep 19 01:37:03.938174 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:03.938146    2581 patch_prober.go:28] interesting pod/e2e-api-a-a7b600-openshift-default-c88cb89b8-jxm2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Sep 19 01:37:03.938570 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:03.938200    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a7b600-openshift-default-c88cb89b8-jxm2q" podUID="461f7ac7-05b6-412d-8d17-6d0a46f17fc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Sep 19 01:37:04.758807 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:04.758762    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-a7b600-openshift-default-c88cb89b8-jxm2q"
Sep 19 01:37:04.938147 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:04.938112    2581 patch_prober.go:28] interesting pod/e2e-api-a-a7b600-openshift-default-c88cb89b8-jxm2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Sep 19 01:37:04.938316 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:04.938163    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a7b600-openshift-default-c88cb89b8-jxm2q" podUID="461f7ac7-05b6-412d-8d17-6d0a46f17fc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Sep 19 01:37:05.850981 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:05.850950    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-29d76c-openshift-default-5c894f7888-9hd9h"]
Sep 19 01:37:05.853511 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:05.853492    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-29d76c-openshift-default-5c894f7888-9hd9h"
Sep 19 01:37:05.855665 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:05.855643    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-29d76c-openshift-default-dockercfg-wtq88\""
Sep 19 01:37:05.868958 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:05.868935    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-29d76c-openshift-default-5c894f7888-9hd9h"]
Sep 19 01:37:05.937967 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:05.937934    2581 patch_prober.go:28] interesting pod/e2e-api-a-a7b600-openshift-default-c88cb89b8-jxm2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Sep 19 01:37:05.938119 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:05.937985    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a7b600-openshift-default-c88cb89b8-jxm2q" podUID="461f7ac7-05b6-412d-8d17-6d0a46f17fc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Sep 19 01:37:06.005671 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:06.005635    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b6722d3-0ee2-4f66-b026-7a343ac7d2a5-workload-socket\") pod \"e2e-api-b-29d76c-openshift-default-5c894f7888-9hd9h\" (UID: \"3b6722d3-0ee2-4f66-b026-7a343ac7d2a5\") " pod="openshift-ingress/e2e-api-b-29d76c-openshift-default-5c894f7888-9hd9h"
Sep 19 01:37:06.005671 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:06.005675    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b6722d3-0ee2-4f66-b026-7a343ac7d2a5-istio-envoy\") pod \"e2e-api-b-29d76c-openshift-default-5c894f7888-9hd9h\" (UID: \"3b6722d3-0ee2-4f66-b026-7a343ac7d2a5\") " pod="openshift-ingress/e2e-api-b-29d76c-openshift-default-5c894f7888-9hd9h"
Sep 19 01:37:06.006124 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:06.005718    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b6722d3-0ee2-4f66-b026-7a343ac7d2a5-workload-certs\") pod \"e2e-api-b-29d76c-openshift-default-5c894f7888-9hd9h\" (UID: \"3b6722d3-0ee2-4f66-b026-7a343ac7d2a5\") " pod="openshift-ingress/e2e-api-b-29d76c-openshift-default-5c894f7888-9hd9h"
Sep 19 01:37:06.006124 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:06.005760    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b6722d3-0ee2-4f66-b026-7a343ac7d2a5-istio-podinfo\") pod \"e2e-api-b-29d76c-openshift-default-5c894f7888-9hd9h\" (UID: \"3b6722d3-0ee2-4f66-b026-7a343ac7d2a5\") " pod="openshift-ingress/e2e-api-b-29d76c-openshift-default-5c894f7888-9hd9h"
Sep 19 01:37:06.006124 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:06.005777    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b6722d3-0ee2-4f66-b026-7a343ac7d2a5-istio-data\") pod \"e2e-api-b-29d76c-openshift-default-5c894f7888-9hd9h\" (UID: \"3b6722d3-0ee2-4f66-b026-7a343ac7d2a5\") " pod="openshift-ingress/e2e-api-b-29d76c-openshift-default-5c894f7888-9hd9h"
Sep 19 01:37:06.006124 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:06.005793    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b6722d3-0ee2-4f66-b026-7a343ac7d2a5-istiod-ca-cert\") pod \"e2e-api-b-29d76c-openshift-default-5c894f7888-9hd9h\" (UID: \"3b6722d3-0ee2-4f66-b026-7a343ac7d2a5\") " pod="openshift-ingress/e2e-api-b-29d76c-openshift-default-5c894f7888-9hd9h"
Sep 19 01:37:06.006124 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:06.005883    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l88br\" (UniqueName: \"kubernetes.io/projected/3b6722d3-0ee2-4f66-b026-7a343ac7d2a5-kube-api-access-l88br\") pod \"e2e-api-b-29d76c-openshift-default-5c894f7888-9hd9h\" (UID: \"3b6722d3-0ee2-4f66-b026-7a343ac7d2a5\") " pod="openshift-ingress/e2e-api-b-29d76c-openshift-default-5c894f7888-9hd9h"
Sep 19 01:37:06.006124 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:06.005931    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b6722d3-0ee2-4f66-b026-7a343ac7d2a5-istio-token\") pod \"e2e-api-b-29d76c-openshift-default-5c894f7888-9hd9h\" (UID: \"3b6722d3-0ee2-4f66-b026-7a343ac7d2a5\") " pod="openshift-ingress/e2e-api-b-29d76c-openshift-default-5c894f7888-9hd9h"
Sep 19 01:37:06.006124 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:06.005957    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b6722d3-0ee2-4f66-b026-7a343ac7d2a5-credential-socket\") pod \"e2e-api-b-29d76c-openshift-default-5c894f7888-9hd9h\" (UID: \"3b6722d3-0ee2-4f66-b026-7a343ac7d2a5\") " pod="openshift-ingress/e2e-api-b-29d76c-openshift-default-5c894f7888-9hd9h"
Sep 19 01:37:06.107239 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:06.107156    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l88br\" (UniqueName: \"kubernetes.io/projected/3b6722d3-0ee2-4f66-b026-7a343ac7d2a5-kube-api-access-l88br\") pod \"e2e-api-b-29d76c-openshift-default-5c894f7888-9hd9h\" (UID: \"3b6722d3-0ee2-4f66-b026-7a343ac7d2a5\") " pod="openshift-ingress/e2e-api-b-29d76c-openshift-default-5c894f7888-9hd9h"
Sep 19 01:37:06.107239 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:06.107203    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b6722d3-0ee2-4f66-b026-7a343ac7d2a5-istio-token\") pod \"e2e-api-b-29d76c-openshift-default-5c894f7888-9hd9h\" (UID: \"3b6722d3-0ee2-4f66-b026-7a343ac7d2a5\") " pod="openshift-ingress/e2e-api-b-29d76c-openshift-default-5c894f7888-9hd9h"
Sep 19 01:37:06.107239 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:06.107221    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b6722d3-0ee2-4f66-b026-7a343ac7d2a5-credential-socket\") pod \"e2e-api-b-29d76c-openshift-default-5c894f7888-9hd9h\" (UID: \"3b6722d3-0ee2-4f66-b026-7a343ac7d2a5\") " pod="openshift-ingress/e2e-api-b-29d76c-openshift-default-5c894f7888-9hd9h"
Sep 19 01:37:06.107239 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:06.107241    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b6722d3-0ee2-4f66-b026-7a343ac7d2a5-workload-socket\") pod \"e2e-api-b-29d76c-openshift-default-5c894f7888-9hd9h\" (UID: \"3b6722d3-0ee2-4f66-b026-7a343ac7d2a5\") " pod="openshift-ingress/e2e-api-b-29d76c-openshift-default-5c894f7888-9hd9h"
Sep 19 01:37:06.107573 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:06.107274    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b6722d3-0ee2-4f66-b026-7a343ac7d2a5-istio-envoy\") pod \"e2e-api-b-29d76c-openshift-default-5c894f7888-9hd9h\" (UID: \"3b6722d3-0ee2-4f66-b026-7a343ac7d2a5\") " pod="openshift-ingress/e2e-api-b-29d76c-openshift-default-5c894f7888-9hd9h"
Sep 19 01:37:06.107573 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:06.107314    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b6722d3-0ee2-4f66-b026-7a343ac7d2a5-workload-certs\") pod \"e2e-api-b-29d76c-openshift-default-5c894f7888-9hd9h\" (UID: \"3b6722d3-0ee2-4f66-b026-7a343ac7d2a5\") " pod="openshift-ingress/e2e-api-b-29d76c-openshift-default-5c894f7888-9hd9h"
Sep 19 01:37:06.107573 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:06.107333    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b6722d3-0ee2-4f66-b026-7a343ac7d2a5-istio-podinfo\") pod \"e2e-api-b-29d76c-openshift-default-5c894f7888-9hd9h\" (UID: \"3b6722d3-0ee2-4f66-b026-7a343ac7d2a5\") " pod="openshift-ingress/e2e-api-b-29d76c-openshift-default-5c894f7888-9hd9h"
Sep 19 01:37:06.107573 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:06.107348    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b6722d3-0ee2-4f66-b026-7a343ac7d2a5-istio-data\") pod \"e2e-api-b-29d76c-openshift-default-5c894f7888-9hd9h\" (UID: \"3b6722d3-0ee2-4f66-b026-7a343ac7d2a5\") " pod="openshift-ingress/e2e-api-b-29d76c-openshift-default-5c894f7888-9hd9h"
Sep 19 01:37:06.107573 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:06.107363    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b6722d3-0ee2-4f66-b026-7a343ac7d2a5-istiod-ca-cert\") pod \"e2e-api-b-29d76c-openshift-default-5c894f7888-9hd9h\" (UID: \"3b6722d3-0ee2-4f66-b026-7a343ac7d2a5\") " pod="openshift-ingress/e2e-api-b-29d76c-openshift-default-5c894f7888-9hd9h"
Sep 19 01:37:06.107877 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:06.107686    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b6722d3-0ee2-4f66-b026-7a343ac7d2a5-workload-socket\") pod \"e2e-api-b-29d76c-openshift-default-5c894f7888-9hd9h\" (UID: \"3b6722d3-0ee2-4f66-b026-7a343ac7d2a5\") " pod="openshift-ingress/e2e-api-b-29d76c-openshift-default-5c894f7888-9hd9h"
Sep 19 01:37:06.107877 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:06.107735    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b6722d3-0ee2-4f66-b026-7a343ac7d2a5-istio-data\") pod \"e2e-api-b-29d76c-openshift-default-5c894f7888-9hd9h\" (UID: \"3b6722d3-0ee2-4f66-b026-7a343ac7d2a5\") " pod="openshift-ingress/e2e-api-b-29d76c-openshift-default-5c894f7888-9hd9h"
Sep 19 01:37:06.107877 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:06.107840    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b6722d3-0ee2-4f66-b026-7a343ac7d2a5-workload-certs\") pod \"e2e-api-b-29d76c-openshift-default-5c894f7888-9hd9h\" (UID: \"3b6722d3-0ee2-4f66-b026-7a343ac7d2a5\") " pod="openshift-ingress/e2e-api-b-29d76c-openshift-default-5c894f7888-9hd9h"
Sep 19 01:37:06.108051 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:06.108017    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b6722d3-0ee2-4f66-b026-7a343ac7d2a5-credential-socket\") pod \"e2e-api-b-29d76c-openshift-default-5c894f7888-9hd9h\" (UID: \"3b6722d3-0ee2-4f66-b026-7a343ac7d2a5\") " pod="openshift-ingress/e2e-api-b-29d76c-openshift-default-5c894f7888-9hd9h"
Sep 19 01:37:06.108225 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:06.108204    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b6722d3-0ee2-4f66-b026-7a343ac7d2a5-istiod-ca-cert\") pod \"e2e-api-b-29d76c-openshift-default-5c894f7888-9hd9h\" (UID: \"3b6722d3-0ee2-4f66-b026-7a343ac7d2a5\") " pod="openshift-ingress/e2e-api-b-29d76c-openshift-default-5c894f7888-9hd9h"
Sep 19 01:37:06.109901 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:06.109877    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b6722d3-0ee2-4f66-b026-7a343ac7d2a5-istio-envoy\") pod \"e2e-api-b-29d76c-openshift-default-5c894f7888-9hd9h\" (UID: \"3b6722d3-0ee2-4f66-b026-7a343ac7d2a5\") " pod="openshift-ingress/e2e-api-b-29d76c-openshift-default-5c894f7888-9hd9h"
Sep 19 01:37:06.110026 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:06.110009    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b6722d3-0ee2-4f66-b026-7a343ac7d2a5-istio-podinfo\") pod \"e2e-api-b-29d76c-openshift-default-5c894f7888-9hd9h\" (UID: \"3b6722d3-0ee2-4f66-b026-7a343ac7d2a5\") " pod="openshift-ingress/e2e-api-b-29d76c-openshift-default-5c894f7888-9hd9h"
Sep 19 01:37:06.116231 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:06.116205    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b6722d3-0ee2-4f66-b026-7a343ac7d2a5-istio-token\") pod \"e2e-api-b-29d76c-openshift-default-5c894f7888-9hd9h\" (UID: \"3b6722d3-0ee2-4f66-b026-7a343ac7d2a5\") " pod="openshift-ingress/e2e-api-b-29d76c-openshift-default-5c894f7888-9hd9h"
Sep 19 01:37:06.116231 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:06.116214    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l88br\" (UniqueName: \"kubernetes.io/projected/3b6722d3-0ee2-4f66-b026-7a343ac7d2a5-kube-api-access-l88br\") pod \"e2e-api-b-29d76c-openshift-default-5c894f7888-9hd9h\" (UID: \"3b6722d3-0ee2-4f66-b026-7a343ac7d2a5\") " pod="openshift-ingress/e2e-api-b-29d76c-openshift-default-5c894f7888-9hd9h"
Sep 19 01:37:06.162987 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:06.162965    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-29d76c-openshift-default-5c894f7888-9hd9h"
Sep 19 01:37:06.462081 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:06.462004    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-29d76c-openshift-default-5c894f7888-9hd9h"]
Sep 19 01:37:06.465594 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:37:06.465569    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b6722d3_0ee2_4f66_b026_7a343ac7d2a5.slice/crio-e347f256a3b0e47a7f2989739bb9b9f03d6fa6e6005f3a47dafd49d3f6918dc7 WatchSource:0}: Error finding container e347f256a3b0e47a7f2989739bb9b9f03d6fa6e6005f3a47dafd49d3f6918dc7: Status 404 returned error can't find the container with id e347f256a3b0e47a7f2989739bb9b9f03d6fa6e6005f3a47dafd49d3f6918dc7
Sep 19 01:37:06.474287 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:06.474232    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 19 01:37:06.474404 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:06.474376    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 19 01:37:06.474557 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:06.474460    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 19 01:37:06.938129 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:06.938094    2581 patch_prober.go:28] interesting pod/e2e-api-a-a7b600-openshift-default-c88cb89b8-jxm2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Sep 19 01:37:06.938333 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:06.938142    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a7b600-openshift-default-c88cb89b8-jxm2q" podUID="461f7ac7-05b6-412d-8d17-6d0a46f17fc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Sep 19 01:37:06.943875 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:06.943835    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-29d76c-openshift-default-5c894f7888-9hd9h" event={"ID":"3b6722d3-0ee2-4f66-b026-7a343ac7d2a5","Type":"ContainerStarted","Data":"df07e1960f4778ff59a3ddeceb435131064a5a86457952be9c3cec8ee2f99e9e"}
Sep 19 01:37:06.944010 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:06.943928    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-29d76c-openshift-default-5c894f7888-9hd9h" event={"ID":"3b6722d3-0ee2-4f66-b026-7a343ac7d2a5","Type":"ContainerStarted","Data":"e347f256a3b0e47a7f2989739bb9b9f03d6fa6e6005f3a47dafd49d3f6918dc7"}
Sep 19 01:37:06.981468 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:06.981407    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-29d76c-openshift-default-5c894f7888-9hd9h" podStartSLOduration=1.9813884339999999 podStartE2EDuration="1.981388434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 01:37:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 01:37:06.979674167 +0000 UTC m=+1477.867343267" watchObservedRunningTime="2026-09-19 01:37:06.981388434 +0000 UTC m=+1477.869057511"
Sep 19 01:37:07.163846 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:07.163812    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-29d76c-openshift-default-5c894f7888-9hd9h"
Sep 19 01:37:07.169473 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:07.169448    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-29d76c-openshift-default-5c894f7888-9hd9h"
Sep 19 01:37:07.938030 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:07.937998    2581 patch_prober.go:28] interesting pod/e2e-api-a-a7b600-openshift-default-c88cb89b8-jxm2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Sep 19 01:37:07.938181 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:07.938048    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a7b600-openshift-default-c88cb89b8-jxm2q" podUID="461f7ac7-05b6-412d-8d17-6d0a46f17fc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Sep 19 01:37:07.949963 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:07.949937    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-29d76c-openshift-default-5c894f7888-9hd9h"
Sep 19 01:37:07.950678 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:07.950661    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-29d76c-openshift-default-5c894f7888-9hd9h"
Sep 19 01:37:08.942010 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:08.941980    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-a7b600-openshift-default-c88cb89b8-jxm2q"
Sep 19 01:37:08.942982 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:08.942962    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-a7b600-openshift-default-c88cb89b8-jxm2q"
Sep 19 01:37:10.894434 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:10.894398    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c9nsz"]
Sep 19 01:37:29.186498 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:29.186463    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c9nsz"]
Sep 19 01:37:31.068515 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:31.068487    2581 generic.go:356] "Generic (PLEG): container finished" podID="3b6722d3-0ee2-4f66-b026-7a343ac7d2a5" containerID="df07e1960f4778ff59a3ddeceb435131064a5a86457952be9c3cec8ee2f99e9e" exitCode=0
Sep 19 01:37:31.068962 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:31.068567    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-29d76c-openshift-default-5c894f7888-9hd9h" event={"ID":"3b6722d3-0ee2-4f66-b026-7a343ac7d2a5","Type":"ContainerDied","Data":"df07e1960f4778ff59a3ddeceb435131064a5a86457952be9c3cec8ee2f99e9e"}
Sep 19 01:37:31.069021 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:31.068993    2581 scope.go:117] "RemoveContainer" containerID="df07e1960f4778ff59a3ddeceb435131064a5a86457952be9c3cec8ee2f99e9e"
Sep 19 01:37:31.070189 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:31.070165    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 19 01:37:31.070272 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:31.070229    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 19 01:37:31.070272 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:31.070262    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 19 01:37:31.265479 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:31.265451    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x785d_f759a12f-5b6f-4924-96f2-dd5e00fc1348/ovn-acl-logging/0.log"
Sep 19 01:37:31.269093 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:31.269069    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x785d_f759a12f-5b6f-4924-96f2-dd5e00fc1348/ovn-acl-logging/0.log"
Sep 19 01:37:31.472116 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:31.472086    2581 scope.go:117] "RemoveContainer" containerID="6db9a10f219875cb499e9e790bf18bf0bd9cce4b5db16899647bc75a3ca24b49"
Sep 19 01:37:31.480437 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:31.480412    2581 scope.go:117] "RemoveContainer" containerID="b7b726aafd2f4427c203c2b4a3a2ff54062601b74d31874d0227576fae2f715c"
Sep 19 01:37:32.075559 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:32.075519    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-29d76c-openshift-default-5c894f7888-9hd9h" event={"ID":"3b6722d3-0ee2-4f66-b026-7a343ac7d2a5","Type":"ContainerStarted","Data":"bf57a99a5fadeae25b08dea3df686e7e9638a3f5bd4ce3c89b62841c5c3e8e61"}
Sep 19 01:37:32.163619 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:32.163586    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-29d76c-openshift-default-5c894f7888-9hd9h"
Sep 19 01:37:32.163819 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:32.163798    2581 patch_prober.go:28] interesting pod/e2e-api-b-29d76c-openshift-default-5c894f7888-9hd9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Sep 19 01:37:32.163896 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:32.163845    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-29d76c-openshift-default-5c894f7888-9hd9h" podUID="3b6722d3-0ee2-4f66-b026-7a343ac7d2a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Sep 19 01:37:33.163850 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:33.163814    2581 patch_prober.go:28] interesting pod/e2e-api-b-29d76c-openshift-default-5c894f7888-9hd9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Sep 19 01:37:33.164270 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:33.163883    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-29d76c-openshift-default-5c894f7888-9hd9h" podUID="3b6722d3-0ee2-4f66-b026-7a343ac7d2a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Sep 19 01:37:34.163925 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:34.163884    2581 patch_prober.go:28] interesting pod/e2e-api-b-29d76c-openshift-default-5c894f7888-9hd9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Sep 19 01:37:34.164417 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:34.163952    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-29d76c-openshift-default-5c894f7888-9hd9h" podUID="3b6722d3-0ee2-4f66-b026-7a343ac7d2a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Sep 19 01:37:35.163615 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:35.163582    2581 patch_prober.go:28] interesting pod/e2e-api-b-29d76c-openshift-default-5c894f7888-9hd9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Sep 19 01:37:35.163791 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:35.163632    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-29d76c-openshift-default-5c894f7888-9hd9h" podUID="3b6722d3-0ee2-4f66-b026-7a343ac7d2a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Sep 19 01:37:36.163576 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:36.163546    2581 patch_prober.go:28] interesting pod/e2e-api-b-29d76c-openshift-default-5c894f7888-9hd9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Sep 19 01:37:36.164002 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:36.163599    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-29d76c-openshift-default-5c894f7888-9hd9h" podUID="3b6722d3-0ee2-4f66-b026-7a343ac7d2a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Sep 19 01:37:37.167909 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:37.167878    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-29d76c-openshift-default-5c894f7888-9hd9h"
Sep 19 01:37:37.168374 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:37.168152    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-29d76c-openshift-default-5c894f7888-9hd9h"
Sep 19 01:37:37.168877 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:37.168842    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-29d76c-openshift-default-5c894f7888-9hd9h"
Sep 19 01:37:42.169144 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:42.169060    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a7b600-kmsm4"]
Sep 19 01:37:42.172873 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:42.172840    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a7b600-kmsm4"
Sep 19 01:37:42.175111 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:42.175092    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 19 01:37:42.175245 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:42.175211    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s7pvt\""
Sep 19 01:37:42.181765 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:42.181743    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a7b600-kmsm4"]
Sep 19 01:37:42.317554 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:42.317526    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9ee52ef-f51a-4efd-b715-daa77b8e20a0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a7b600-kmsm4\" (UID: \"a9ee52ef-f51a-4efd-b715-daa77b8e20a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a7b600-kmsm4"
Sep 19 01:37:42.418051 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:42.418020    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9ee52ef-f51a-4efd-b715-daa77b8e20a0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a7b600-kmsm4\" (UID: \"a9ee52ef-f51a-4efd-b715-daa77b8e20a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a7b600-kmsm4"
Sep 19 01:37:42.418626 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:42.418606    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9ee52ef-f51a-4efd-b715-daa77b8e20a0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a7b600-kmsm4\" (UID: \"a9ee52ef-f51a-4efd-b715-daa77b8e20a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a7b600-kmsm4"
Sep 19 01:37:42.482996 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:42.482946    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a7b600-kmsm4"
Sep 19 01:37:42.754722 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:42.754676    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a7b600-kmsm4"]
Sep 19 01:37:42.756896 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:37:42.756854    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9ee52ef_f51a_4efd_b715_daa77b8e20a0.slice/crio-e73b38ea6a7909cede3dd9e0b84a0694056c1e62ed0ddcc91721043804a66af1 WatchSource:0}: Error finding container e73b38ea6a7909cede3dd9e0b84a0694056c1e62ed0ddcc91721043804a66af1: Status 404 returned error can't find the container with id e73b38ea6a7909cede3dd9e0b84a0694056c1e62ed0ddcc91721043804a66af1
Sep 19 01:37:43.137911 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:43.137873    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a7b600-kmsm4" event={"ID":"a9ee52ef-f51a-4efd-b715-daa77b8e20a0","Type":"ContainerStarted","Data":"b3f3a7eb0280e1b198dc842cfbea7bdf2fa9914fba1ee4fc1b939e2615bee5ec"}
Sep 19 01:37:43.138063 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:43.137942    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a7b600-kmsm4" event={"ID":"a9ee52ef-f51a-4efd-b715-daa77b8e20a0","Type":"ContainerStarted","Data":"e73b38ea6a7909cede3dd9e0b84a0694056c1e62ed0ddcc91721043804a66af1"}
Sep 19 01:37:43.153668 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:43.153615    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a7b600-kmsm4" podStartSLOduration=1.153596478 podStartE2EDuration="1.153596478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 01:37:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 01:37:43.152621287 +0000 UTC m=+1514.040290376" watchObservedRunningTime="2026-09-19 01:37:43.153596478 +0000 UTC m=+1514.041265556"
Sep 19 01:37:44.144523 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:44.144488    2581 generic.go:356] "Generic (PLEG): container finished" podID="a9ee52ef-f51a-4efd-b715-daa77b8e20a0" containerID="b3f3a7eb0280e1b198dc842cfbea7bdf2fa9914fba1ee4fc1b939e2615bee5ec" exitCode=0
Sep 19 01:37:44.144944 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:44.144571    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a7b600-kmsm4" event={"ID":"a9ee52ef-f51a-4efd-b715-daa77b8e20a0","Type":"ContainerDied","Data":"b3f3a7eb0280e1b198dc842cfbea7bdf2fa9914fba1ee4fc1b939e2615bee5ec"}
Sep 19 01:37:45.285366 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:45.285338    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a7b600-kmsm4"
Sep 19 01:37:45.344143 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:45.344109    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9ee52ef-f51a-4efd-b715-daa77b8e20a0-maas-api-service-ca\") pod \"a9ee52ef-f51a-4efd-b715-daa77b8e20a0\" (UID: \"a9ee52ef-f51a-4efd-b715-daa77b8e20a0\") "
Sep 19 01:37:45.344457 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:45.344430    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9ee52ef-f51a-4efd-b715-daa77b8e20a0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a9ee52ef-f51a-4efd-b715-daa77b8e20a0" (UID: "a9ee52ef-f51a-4efd-b715-daa77b8e20a0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 01:37:45.445370 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:45.445297    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9ee52ef-f51a-4efd-b715-daa77b8e20a0-maas-api-service-ca\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:37:46.156750 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:46.156718    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a7b600-kmsm4"
Sep 19 01:37:46.156750 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:46.156741    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a7b600-kmsm4" event={"ID":"a9ee52ef-f51a-4efd-b715-daa77b8e20a0","Type":"ContainerDied","Data":"e73b38ea6a7909cede3dd9e0b84a0694056c1e62ed0ddcc91721043804a66af1"}
Sep 19 01:37:46.157065 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:46.156785    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e73b38ea6a7909cede3dd9e0b84a0694056c1e62ed0ddcc91721043804a66af1"
Sep 19 01:37:48.088799 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:48.088763    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c9nsz"]
Sep 19 01:37:58.280575 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:58.280541    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a7b600-79995f4d78-clb6d"]
Sep 19 01:37:58.281037 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:58.280725    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a7b600-79995f4d78-clb6d" podUID="b34285f9-074c-4800-aab6-cda644d32a5e" containerName="payload-pre-processing" containerID="cri-o://c7107dbe02ed2c69a4b4ac5fe5a16468b816f50d8e6c0e1d676bcd32e5cc1e9e" gracePeriod=30
Sep 19 01:37:58.284600 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:58.284577    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a7b600-d967c6dc-h4hw6"]
Sep 19 01:37:58.531616 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:58.531549    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a7b600-79995f4d78-clb6d"
Sep 19 01:37:58.663179 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:58.663133    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a7b600-kmsm4"]
Sep 19 01:37:58.663497 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:58.663474    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wk7hl\" (UniqueName: \"kubernetes.io/projected/b34285f9-074c-4800-aab6-cda644d32a5e-kube-api-access-wk7hl\") pod \"b34285f9-074c-4800-aab6-cda644d32a5e\" (UID: \"b34285f9-074c-4800-aab6-cda644d32a5e\") "
Sep 19 01:37:58.663616 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:58.663603    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b34285f9-074c-4800-aab6-cda644d32a5e-plugins-config-volume\") pod \"b34285f9-074c-4800-aab6-cda644d32a5e\" (UID: \"b34285f9-074c-4800-aab6-cda644d32a5e\") "
Sep 19 01:37:58.664046 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:58.664009    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b34285f9-074c-4800-aab6-cda644d32a5e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b34285f9-074c-4800-aab6-cda644d32a5e" (UID: "b34285f9-074c-4800-aab6-cda644d32a5e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 01:37:58.666208 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:58.666184    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b34285f9-074c-4800-aab6-cda644d32a5e-kube-api-access-wk7hl" (OuterVolumeSpecName: "kube-api-access-wk7hl") pod "b34285f9-074c-4800-aab6-cda644d32a5e" (UID: "b34285f9-074c-4800-aab6-cda644d32a5e"). InnerVolumeSpecName "kube-api-access-wk7hl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 01:37:58.667153 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:58.667133    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a7b600-kmsm4"]
Sep 19 01:37:58.764754 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:58.764709    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b34285f9-074c-4800-aab6-cda644d32a5e-plugins-config-volume\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:37:58.764942 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:58.764757    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wk7hl\" (UniqueName: \"kubernetes.io/projected/b34285f9-074c-4800-aab6-cda644d32a5e-kube-api-access-wk7hl\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:37:59.222118 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:59.222085    2581 generic.go:356] "Generic (PLEG): container finished" podID="b34285f9-074c-4800-aab6-cda644d32a5e" containerID="c7107dbe02ed2c69a4b4ac5fe5a16468b816f50d8e6c0e1d676bcd32e5cc1e9e" exitCode=0
Sep 19 01:37:59.222314 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:59.222141    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a7b600-79995f4d78-clb6d"
Sep 19 01:37:59.222314 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:59.222175    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a7b600-79995f4d78-clb6d" event={"ID":"b34285f9-074c-4800-aab6-cda644d32a5e","Type":"ContainerDied","Data":"c7107dbe02ed2c69a4b4ac5fe5a16468b816f50d8e6c0e1d676bcd32e5cc1e9e"}
Sep 19 01:37:59.222314 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:59.222264    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a7b600-79995f4d78-clb6d" event={"ID":"b34285f9-074c-4800-aab6-cda644d32a5e","Type":"ContainerDied","Data":"f0c4e4eea14d318cb94b55866aaa3bf074c4522d23fb1646ac92b5a4c9a9ec30"}
Sep 19 01:37:59.222314 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:59.222295    2581 scope.go:117] "RemoveContainer" containerID="c7107dbe02ed2c69a4b4ac5fe5a16468b816f50d8e6c0e1d676bcd32e5cc1e9e"
Sep 19 01:37:59.222536 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:59.222442    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-a7b600-d967c6dc-h4hw6" podUID="2c961b0a-4f34-4aee-9807-69d4e1f0e20b" containerName="payload-processing" containerID="cri-o://d70e7bf9bfc87967c4f576a586881ae901995e73ca6b54ecbebc4c43c25e97cb" gracePeriod=30
Sep 19 01:37:59.231781 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:59.231763    2581 scope.go:117] "RemoveContainer" containerID="c7107dbe02ed2c69a4b4ac5fe5a16468b816f50d8e6c0e1d676bcd32e5cc1e9e"
Sep 19 01:37:59.232088 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:37:59.232067    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7107dbe02ed2c69a4b4ac5fe5a16468b816f50d8e6c0e1d676bcd32e5cc1e9e\": container with ID starting with c7107dbe02ed2c69a4b4ac5fe5a16468b816f50d8e6c0e1d676bcd32e5cc1e9e not found: ID does not exist" containerID="c7107dbe02ed2c69a4b4ac5fe5a16468b816f50d8e6c0e1d676bcd32e5cc1e9e"
Sep 19 01:37:59.232153 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:59.232096    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7107dbe02ed2c69a4b4ac5fe5a16468b816f50d8e6c0e1d676bcd32e5cc1e9e"} err="failed to get container status \"c7107dbe02ed2c69a4b4ac5fe5a16468b816f50d8e6c0e1d676bcd32e5cc1e9e\": rpc error: code = NotFound desc = could not find container \"c7107dbe02ed2c69a4b4ac5fe5a16468b816f50d8e6c0e1d676bcd32e5cc1e9e\": container with ID starting with c7107dbe02ed2c69a4b4ac5fe5a16468b816f50d8e6c0e1d676bcd32e5cc1e9e not found: ID does not exist"
Sep 19 01:37:59.262685 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:59.262664    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a7b600-79995f4d78-clb6d"]
Sep 19 01:37:59.272009 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:59.271990    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9ee52ef-f51a-4efd-b715-daa77b8e20a0" path="/var/lib/kubelet/pods/a9ee52ef-f51a-4efd-b715-daa77b8e20a0/volumes"
Sep 19 01:37:59.272242 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:59.272230    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a7b600-79995f4d78-clb6d"]
Sep 19 01:37:59.472114 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:59.472086    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a7b600-d967c6dc-h4hw6"
Sep 19 01:37:59.572398 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:59.572321    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qwzth\" (UniqueName: \"kubernetes.io/projected/2c961b0a-4f34-4aee-9807-69d4e1f0e20b-kube-api-access-qwzth\") pod \"2c961b0a-4f34-4aee-9807-69d4e1f0e20b\" (UID: \"2c961b0a-4f34-4aee-9807-69d4e1f0e20b\") "
Sep 19 01:37:59.572541 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:59.572460    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c961b0a-4f34-4aee-9807-69d4e1f0e20b-plugins-config-volume\") pod \"2c961b0a-4f34-4aee-9807-69d4e1f0e20b\" (UID: \"2c961b0a-4f34-4aee-9807-69d4e1f0e20b\") "
Sep 19 01:37:59.572810 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:59.572789    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c961b0a-4f34-4aee-9807-69d4e1f0e20b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2c961b0a-4f34-4aee-9807-69d4e1f0e20b" (UID: "2c961b0a-4f34-4aee-9807-69d4e1f0e20b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 01:37:59.574466 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:59.574447    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c961b0a-4f34-4aee-9807-69d4e1f0e20b-kube-api-access-qwzth" (OuterVolumeSpecName: "kube-api-access-qwzth") pod "2c961b0a-4f34-4aee-9807-69d4e1f0e20b" (UID: "2c961b0a-4f34-4aee-9807-69d4e1f0e20b"). InnerVolumeSpecName "kube-api-access-qwzth". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 01:37:59.673689 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:59.673662    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c961b0a-4f34-4aee-9807-69d4e1f0e20b-plugins-config-volume\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:37:59.673689 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:37:59.673686    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwzth\" (UniqueName: \"kubernetes.io/projected/2c961b0a-4f34-4aee-9807-69d4e1f0e20b-kube-api-access-qwzth\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:38:00.228841 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:00.228808    2581 generic.go:356] "Generic (PLEG): container finished" podID="2c961b0a-4f34-4aee-9807-69d4e1f0e20b" containerID="d70e7bf9bfc87967c4f576a586881ae901995e73ca6b54ecbebc4c43c25e97cb" exitCode=0
Sep 19 01:38:00.229040 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:00.228848    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a7b600-d967c6dc-h4hw6"
Sep 19 01:38:00.229040 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:00.228881    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a7b600-d967c6dc-h4hw6" event={"ID":"2c961b0a-4f34-4aee-9807-69d4e1f0e20b","Type":"ContainerDied","Data":"d70e7bf9bfc87967c4f576a586881ae901995e73ca6b54ecbebc4c43c25e97cb"}
Sep 19 01:38:00.229040 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:00.228921    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a7b600-d967c6dc-h4hw6" event={"ID":"2c961b0a-4f34-4aee-9807-69d4e1f0e20b","Type":"ContainerDied","Data":"9b1ab25183bacd6c32e160a197ceaebde6b526a80e9ff77ae75ea0887db252bb"}
Sep 19 01:38:00.229040 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:00.228942    2581 scope.go:117] "RemoveContainer" containerID="d70e7bf9bfc87967c4f576a586881ae901995e73ca6b54ecbebc4c43c25e97cb"
Sep 19 01:38:00.237923 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:00.237905    2581 scope.go:117] "RemoveContainer" containerID="d70e7bf9bfc87967c4f576a586881ae901995e73ca6b54ecbebc4c43c25e97cb"
Sep 19 01:38:00.238209 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:38:00.238191    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d70e7bf9bfc87967c4f576a586881ae901995e73ca6b54ecbebc4c43c25e97cb\": container with ID starting with d70e7bf9bfc87967c4f576a586881ae901995e73ca6b54ecbebc4c43c25e97cb not found: ID does not exist" containerID="d70e7bf9bfc87967c4f576a586881ae901995e73ca6b54ecbebc4c43c25e97cb"
Sep 19 01:38:00.238273 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:00.238216    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d70e7bf9bfc87967c4f576a586881ae901995e73ca6b54ecbebc4c43c25e97cb"} err="failed to get container status \"d70e7bf9bfc87967c4f576a586881ae901995e73ca6b54ecbebc4c43c25e97cb\": rpc error: code = NotFound desc = could not find container \"d70e7bf9bfc87967c4f576a586881ae901995e73ca6b54ecbebc4c43c25e97cb\": container with ID starting with d70e7bf9bfc87967c4f576a586881ae901995e73ca6b54ecbebc4c43c25e97cb not found: ID does not exist"
Sep 19 01:38:00.249637 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:00.249615    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a7b600-d967c6dc-h4hw6"]
Sep 19 01:38:00.254491 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:00.254472    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a7b600-d967c6dc-h4hw6"]
Sep 19 01:38:01.271678 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:01.271646    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c961b0a-4f34-4aee-9807-69d4e1f0e20b" path="/var/lib/kubelet/pods/2c961b0a-4f34-4aee-9807-69d4e1f0e20b/volumes"
Sep 19 01:38:01.272133 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:01.272072    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b34285f9-074c-4800-aab6-cda644d32a5e" path="/var/lib/kubelet/pods/b34285f9-074c-4800-aab6-cda644d32a5e/volumes"
Sep 19 01:38:01.295634 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:01.295599    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-751f6a/sub-test-model-751f6a-kserve-655995547-8qrtb"]
Sep 19 01:38:01.295932 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:01.295919    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9ee52ef-f51a-4efd-b715-daa77b8e20a0" containerName="revoke-keys"
Sep 19 01:38:01.295985 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:01.295933    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9ee52ef-f51a-4efd-b715-daa77b8e20a0" containerName="revoke-keys"
Sep 19 01:38:01.295985 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:01.295951    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c961b0a-4f34-4aee-9807-69d4e1f0e20b" containerName="payload-processing"
Sep 19 01:38:01.295985 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:01.295957    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c961b0a-4f34-4aee-9807-69d4e1f0e20b" containerName="payload-processing"
Sep 19 01:38:01.295985 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:01.295968    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b34285f9-074c-4800-aab6-cda644d32a5e" containerName="payload-pre-processing"
Sep 19 01:38:01.295985 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:01.295973    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b34285f9-074c-4800-aab6-cda644d32a5e" containerName="payload-pre-processing"
Sep 19 01:38:01.296168 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:01.296023    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c961b0a-4f34-4aee-9807-69d4e1f0e20b" containerName="payload-processing"
Sep 19 01:38:01.296168 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:01.296031    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b34285f9-074c-4800-aab6-cda644d32a5e" containerName="payload-pre-processing"
Sep 19 01:38:01.296168 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:01.296038    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9ee52ef-f51a-4efd-b715-daa77b8e20a0" containerName="revoke-keys"
Sep 19 01:38:01.300050 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:01.300034    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-751f6a/sub-test-model-751f6a-kserve-655995547-8qrtb"
Sep 19 01:38:01.304974 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:01.304954    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-751f6a\"/\"openshift-service-ca.crt\""
Sep 19 01:38:01.305528 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:01.305508    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-751f6a\"/\"kube-root-ca.crt\""
Sep 19 01:38:01.305528 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:01.305538    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-751f6a\"/\"default-dockercfg-nxcm6\""
Sep 19 01:38:01.305752 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:01.305569    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-751f6a\"/\"sub-test-model-751f6a-kserve-self-signed-certs\""
Sep 19 01:38:01.315291 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:01.315271    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-751f6a/sub-test-model-751f6a-kserve-655995547-8qrtb"]
Sep 19 01:38:01.489745 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:01.489708    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1e40600-b622-4f22-84ba-a8892e02a87a-home\") pod \"sub-test-model-751f6a-kserve-655995547-8qrtb\" (UID: \"b1e40600-b622-4f22-84ba-a8892e02a87a\") " pod="ai-tenant-e2e-shared-a-w3-751f6a/sub-test-model-751f6a-kserve-655995547-8qrtb"
Sep 19 01:38:01.489961 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:01.489765    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chdt7\" (UniqueName: \"kubernetes.io/projected/b1e40600-b622-4f22-84ba-a8892e02a87a-kube-api-access-chdt7\") pod \"sub-test-model-751f6a-kserve-655995547-8qrtb\" (UID: \"b1e40600-b622-4f22-84ba-a8892e02a87a\") " pod="ai-tenant-e2e-shared-a-w3-751f6a/sub-test-model-751f6a-kserve-655995547-8qrtb"
Sep 19 01:38:01.489961 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:01.489830    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1e40600-b622-4f22-84ba-a8892e02a87a-model-cache\") pod \"sub-test-model-751f6a-kserve-655995547-8qrtb\" (UID: \"b1e40600-b622-4f22-84ba-a8892e02a87a\") " pod="ai-tenant-e2e-shared-a-w3-751f6a/sub-test-model-751f6a-kserve-655995547-8qrtb"
Sep 19 01:38:01.489961 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:01.489900    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1e40600-b622-4f22-84ba-a8892e02a87a-dshm\") pod \"sub-test-model-751f6a-kserve-655995547-8qrtb\" (UID: \"b1e40600-b622-4f22-84ba-a8892e02a87a\") " pod="ai-tenant-e2e-shared-a-w3-751f6a/sub-test-model-751f6a-kserve-655995547-8qrtb"
Sep 19 01:38:01.489961 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:01.489946    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1e40600-b622-4f22-84ba-a8892e02a87a-tmp-dir\") pod \"sub-test-model-751f6a-kserve-655995547-8qrtb\" (UID: \"b1e40600-b622-4f22-84ba-a8892e02a87a\") " pod="ai-tenant-e2e-shared-a-w3-751f6a/sub-test-model-751f6a-kserve-655995547-8qrtb"
Sep 19 01:38:01.490105 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:01.490053    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1e40600-b622-4f22-84ba-a8892e02a87a-tls-certs\") pod \"sub-test-model-751f6a-kserve-655995547-8qrtb\" (UID: \"b1e40600-b622-4f22-84ba-a8892e02a87a\") " pod="ai-tenant-e2e-shared-a-w3-751f6a/sub-test-model-751f6a-kserve-655995547-8qrtb"
Sep 19 01:38:01.590546 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:01.590444    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1e40600-b622-4f22-84ba-a8892e02a87a-dshm\") pod \"sub-test-model-751f6a-kserve-655995547-8qrtb\" (UID: \"b1e40600-b622-4f22-84ba-a8892e02a87a\") " pod="ai-tenant-e2e-shared-a-w3-751f6a/sub-test-model-751f6a-kserve-655995547-8qrtb"
Sep 19 01:38:01.590546 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:01.590479    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1e40600-b622-4f22-84ba-a8892e02a87a-tmp-dir\") pod \"sub-test-model-751f6a-kserve-655995547-8qrtb\" (UID: \"b1e40600-b622-4f22-84ba-a8892e02a87a\") " pod="ai-tenant-e2e-shared-a-w3-751f6a/sub-test-model-751f6a-kserve-655995547-8qrtb"
Sep 19 01:38:01.590780 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:01.590553    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1e40600-b622-4f22-84ba-a8892e02a87a-tls-certs\") pod \"sub-test-model-751f6a-kserve-655995547-8qrtb\" (UID: \"b1e40600-b622-4f22-84ba-a8892e02a87a\") " pod="ai-tenant-e2e-shared-a-w3-751f6a/sub-test-model-751f6a-kserve-655995547-8qrtb"
Sep 19 01:38:01.590780 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:01.590577    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1e40600-b622-4f22-84ba-a8892e02a87a-home\") pod \"sub-test-model-751f6a-kserve-655995547-8qrtb\" (UID: \"b1e40600-b622-4f22-84ba-a8892e02a87a\") " pod="ai-tenant-e2e-shared-a-w3-751f6a/sub-test-model-751f6a-kserve-655995547-8qrtb"
Sep 19 01:38:01.590780 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:01.590610    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chdt7\" (UniqueName: \"kubernetes.io/projected/b1e40600-b622-4f22-84ba-a8892e02a87a-kube-api-access-chdt7\") pod \"sub-test-model-751f6a-kserve-655995547-8qrtb\" (UID: \"b1e40600-b622-4f22-84ba-a8892e02a87a\") " pod="ai-tenant-e2e-shared-a-w3-751f6a/sub-test-model-751f6a-kserve-655995547-8qrtb"
Sep 19 01:38:01.590780 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:01.590630    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1e40600-b622-4f22-84ba-a8892e02a87a-model-cache\") pod \"sub-test-model-751f6a-kserve-655995547-8qrtb\" (UID: \"b1e40600-b622-4f22-84ba-a8892e02a87a\") " pod="ai-tenant-e2e-shared-a-w3-751f6a/sub-test-model-751f6a-kserve-655995547-8qrtb"
Sep 19 01:38:01.591109 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:01.590980    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1e40600-b622-4f22-84ba-a8892e02a87a-tmp-dir\") pod \"sub-test-model-751f6a-kserve-655995547-8qrtb\" (UID: \"b1e40600-b622-4f22-84ba-a8892e02a87a\") " pod="ai-tenant-e2e-shared-a-w3-751f6a/sub-test-model-751f6a-kserve-655995547-8qrtb"
Sep 19 01:38:01.591109 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:01.591033    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1e40600-b622-4f22-84ba-a8892e02a87a-home\") pod \"sub-test-model-751f6a-kserve-655995547-8qrtb\" (UID: \"b1e40600-b622-4f22-84ba-a8892e02a87a\") " pod="ai-tenant-e2e-shared-a-w3-751f6a/sub-test-model-751f6a-kserve-655995547-8qrtb"
Sep 19 01:38:01.591109 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:01.591042    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1e40600-b622-4f22-84ba-a8892e02a87a-model-cache\") pod \"sub-test-model-751f6a-kserve-655995547-8qrtb\" (UID: \"b1e40600-b622-4f22-84ba-a8892e02a87a\") " pod="ai-tenant-e2e-shared-a-w3-751f6a/sub-test-model-751f6a-kserve-655995547-8qrtb"
Sep 19 01:38:01.593156 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:01.593133    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1e40600-b622-4f22-84ba-a8892e02a87a-dshm\") pod \"sub-test-model-751f6a-kserve-655995547-8qrtb\" (UID: \"b1e40600-b622-4f22-84ba-a8892e02a87a\") " pod="ai-tenant-e2e-shared-a-w3-751f6a/sub-test-model-751f6a-kserve-655995547-8qrtb"
Sep 19 01:38:01.593487 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:01.593471    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1e40600-b622-4f22-84ba-a8892e02a87a-tls-certs\") pod \"sub-test-model-751f6a-kserve-655995547-8qrtb\" (UID: \"b1e40600-b622-4f22-84ba-a8892e02a87a\") " pod="ai-tenant-e2e-shared-a-w3-751f6a/sub-test-model-751f6a-kserve-655995547-8qrtb"
Sep 19 01:38:01.599430 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:01.599400    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chdt7\" (UniqueName: \"kubernetes.io/projected/b1e40600-b622-4f22-84ba-a8892e02a87a-kube-api-access-chdt7\") pod \"sub-test-model-751f6a-kserve-655995547-8qrtb\" (UID: \"b1e40600-b622-4f22-84ba-a8892e02a87a\") " pod="ai-tenant-e2e-shared-a-w3-751f6a/sub-test-model-751f6a-kserve-655995547-8qrtb"
Sep 19 01:38:01.609293 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:01.609275    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-751f6a/sub-test-model-751f6a-kserve-655995547-8qrtb"
Sep 19 01:38:01.894916 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:01.894887    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-751f6a/sub-test-model-751f6a-kserve-655995547-8qrtb"]
Sep 19 01:38:01.898240 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:38:01.898200    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb1e40600_b622_4f22_84ba_a8892e02a87a.slice/crio-daeff7088fed8fe5d5ea08986e317377cdeabe08909f6cded4928691fa7f5c24 WatchSource:0}: Error finding container daeff7088fed8fe5d5ea08986e317377cdeabe08909f6cded4928691fa7f5c24: Status 404 returned error can't find the container with id daeff7088fed8fe5d5ea08986e317377cdeabe08909f6cded4928691fa7f5c24
Sep 19 01:38:02.241354 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:02.241256    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-751f6a/sub-test-model-751f6a-kserve-655995547-8qrtb" event={"ID":"b1e40600-b622-4f22-84ba-a8892e02a87a","Type":"ContainerStarted","Data":"ed79110f841f1452f7e5a6c650bffd0221cab207ef33717bc2581af222044a5a"}
Sep 19 01:38:02.241354 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:02.241298    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-751f6a/sub-test-model-751f6a-kserve-655995547-8qrtb" event={"ID":"b1e40600-b622-4f22-84ba-a8892e02a87a","Type":"ContainerStarted","Data":"daeff7088fed8fe5d5ea08986e317377cdeabe08909f6cded4928691fa7f5c24"}
Sep 19 01:38:02.274631 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:02.274583    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-751f6a/sub-test-model-751f6a-kserve-655995547-8qrtb" podStartSLOduration=1.2745692659999999 podStartE2EDuration="1.274569266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 01:38:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 01:38:02.274214053 +0000 UTC m=+1533.161883130" watchObservedRunningTime="2026-09-19 01:38:02.274569266 +0000 UTC m=+1533.162238343"
Sep 19 01:38:05.470531 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:05.470502    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a7b600-openshift-default-c88cb89b8-jxm2q"]
Sep 19 01:38:05.470984 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:05.470686    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-a7b600-openshift-default-c88cb89b8-jxm2q" podUID="461f7ac7-05b6-412d-8d17-6d0a46f17fc2" containerName="istio-proxy" containerID="cri-o://c27a8eef84c402a6b26c9fb1906c11b8c31c155acd96bbd8baa8fb07635a6da7" gracePeriod=30
Sep 19 01:38:05.900282 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:05.900249    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-29d76c-5q2gv"]
Sep 19 01:38:05.903766 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:05.903748    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-29d76c-5q2gv"
Sep 19 01:38:05.906169 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:05.906149    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 19 01:38:05.906274 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:05.906151    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s7pvt\""
Sep 19 01:38:05.912883 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:05.912839    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-29d76c-5q2gv"]
Sep 19 01:38:05.925957 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:05.925927    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ff80952-d515-4c6e-ba11-06eaa27ede33-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-29d76c-5q2gv\" (UID: \"0ff80952-d515-4c6e-ba11-06eaa27ede33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-29d76c-5q2gv"
Sep 19 01:38:06.027233 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:06.027205    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ff80952-d515-4c6e-ba11-06eaa27ede33-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-29d76c-5q2gv\" (UID: \"0ff80952-d515-4c6e-ba11-06eaa27ede33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-29d76c-5q2gv"
Sep 19 01:38:06.027795 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:06.027779    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ff80952-d515-4c6e-ba11-06eaa27ede33-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-29d76c-5q2gv\" (UID: \"0ff80952-d515-4c6e-ba11-06eaa27ede33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-29d76c-5q2gv"
Sep 19 01:38:06.215277 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:06.215213    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-29d76c-5q2gv"
Sep 19 01:38:06.504594 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:06.504524    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-29d76c-5q2gv"]
Sep 19 01:38:06.507992 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:38:06.507967    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ff80952_d515_4c6e_ba11_06eaa27ede33.slice/crio-8240edccb2b507b0bc641098e06310325aca09b96df4ec791a2d5b2d9ac1238d WatchSource:0}: Error finding container 8240edccb2b507b0bc641098e06310325aca09b96df4ec791a2d5b2d9ac1238d: Status 404 returned error can't find the container with id 8240edccb2b507b0bc641098e06310325aca09b96df4ec791a2d5b2d9ac1238d
Sep 19 01:38:07.272639 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:07.272607    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-29d76c-5q2gv" event={"ID":"0ff80952-d515-4c6e-ba11-06eaa27ede33","Type":"ContainerStarted","Data":"56faa8f8efb49498dea7e0b9bedf63ca7afc7677a6dace6fd8eb9ec2a067f838"}
Sep 19 01:38:07.272786 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:07.272655    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-29d76c-5q2gv" event={"ID":"0ff80952-d515-4c6e-ba11-06eaa27ede33","Type":"ContainerStarted","Data":"8240edccb2b507b0bc641098e06310325aca09b96df4ec791a2d5b2d9ac1238d"}
Sep 19 01:38:07.294942 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:07.291713    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-29d76c-5q2gv" podStartSLOduration=2.291692943 podStartE2EDuration="2.291692943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 01:38:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 01:38:07.289020949 +0000 UTC m=+1538.176690050" watchObservedRunningTime="2026-09-19 01:38:07.291692943 +0000 UTC m=+1538.179362021"
Sep 19 01:38:07.586375 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:07.586311    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c9nsz"]
Sep 19 01:38:08.277379 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:08.277344    2581 generic.go:356] "Generic (PLEG): container finished" podID="0ff80952-d515-4c6e-ba11-06eaa27ede33" containerID="56faa8f8efb49498dea7e0b9bedf63ca7afc7677a6dace6fd8eb9ec2a067f838" exitCode=0
Sep 19 01:38:08.277569 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:08.277414    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-29d76c-5q2gv" event={"ID":"0ff80952-d515-4c6e-ba11-06eaa27ede33","Type":"ContainerDied","Data":"56faa8f8efb49498dea7e0b9bedf63ca7afc7677a6dace6fd8eb9ec2a067f838"}
Sep 19 01:38:08.945196 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:08.945154    2581 patch_prober.go:28] interesting pod/e2e-api-a-a7b600-openshift-default-c88cb89b8-jxm2q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 19 01:38:08.945566 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:08.945240    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-a7b600-openshift-default-c88cb89b8-jxm2q" podUID="461f7ac7-05b6-412d-8d17-6d0a46f17fc2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 19 01:38:09.411322 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:09.411301    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-29d76c-5q2gv"
Sep 19 01:38:09.452435 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:09.452412    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ff80952-d515-4c6e-ba11-06eaa27ede33-maas-api-service-ca\") pod \"0ff80952-d515-4c6e-ba11-06eaa27ede33\" (UID: \"0ff80952-d515-4c6e-ba11-06eaa27ede33\") "
Sep 19 01:38:09.452724 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:09.452701    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ff80952-d515-4c6e-ba11-06eaa27ede33-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0ff80952-d515-4c6e-ba11-06eaa27ede33" (UID: "0ff80952-d515-4c6e-ba11-06eaa27ede33"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 01:38:09.553700 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:09.553644    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ff80952-d515-4c6e-ba11-06eaa27ede33-maas-api-service-ca\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:38:10.287807 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:10.287781    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-29d76c-5q2gv"
Sep 19 01:38:10.288304 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:10.287826    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-29d76c-5q2gv" event={"ID":"0ff80952-d515-4c6e-ba11-06eaa27ede33","Type":"ContainerDied","Data":"8240edccb2b507b0bc641098e06310325aca09b96df4ec791a2d5b2d9ac1238d"}
Sep 19 01:38:10.288304 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:10.287885    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8240edccb2b507b0bc641098e06310325aca09b96df4ec791a2d5b2d9ac1238d"
Sep 19 01:38:10.727273 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:10.727249    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a7b600-openshift-default-c88cb89b8-jxm2q"
Sep 19 01:38:10.762949 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:10.762921    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/461f7ac7-05b6-412d-8d17-6d0a46f17fc2-workload-socket\") pod \"461f7ac7-05b6-412d-8d17-6d0a46f17fc2\" (UID: \"461f7ac7-05b6-412d-8d17-6d0a46f17fc2\") "
Sep 19 01:38:10.763079 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:10.762968    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/461f7ac7-05b6-412d-8d17-6d0a46f17fc2-istio-token\") pod \"461f7ac7-05b6-412d-8d17-6d0a46f17fc2\" (UID: \"461f7ac7-05b6-412d-8d17-6d0a46f17fc2\") "
Sep 19 01:38:10.763079 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:10.763010    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/461f7ac7-05b6-412d-8d17-6d0a46f17fc2-credential-socket\") pod \"461f7ac7-05b6-412d-8d17-6d0a46f17fc2\" (UID: \"461f7ac7-05b6-412d-8d17-6d0a46f17fc2\") "
Sep 19 01:38:10.763079 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:10.763055    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/461f7ac7-05b6-412d-8d17-6d0a46f17fc2-istio-data\") pod \"461f7ac7-05b6-412d-8d17-6d0a46f17fc2\" (UID: \"461f7ac7-05b6-412d-8d17-6d0a46f17fc2\") "
Sep 19 01:38:10.763200 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:10.763099    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/461f7ac7-05b6-412d-8d17-6d0a46f17fc2-istio-podinfo\") pod \"461f7ac7-05b6-412d-8d17-6d0a46f17fc2\" (UID: \"461f7ac7-05b6-412d-8d17-6d0a46f17fc2\") "
Sep 19 01:38:10.763200 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:10.763129    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nhxf6\" (UniqueName: \"kubernetes.io/projected/461f7ac7-05b6-412d-8d17-6d0a46f17fc2-kube-api-access-nhxf6\") pod \"461f7ac7-05b6-412d-8d17-6d0a46f17fc2\" (UID: \"461f7ac7-05b6-412d-8d17-6d0a46f17fc2\") "
Sep 19 01:38:10.763200 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:10.763161    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/461f7ac7-05b6-412d-8d17-6d0a46f17fc2-workload-certs\") pod \"461f7ac7-05b6-412d-8d17-6d0a46f17fc2\" (UID: \"461f7ac7-05b6-412d-8d17-6d0a46f17fc2\") "
Sep 19 01:38:10.763200 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:10.763189    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/461f7ac7-05b6-412d-8d17-6d0a46f17fc2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "461f7ac7-05b6-412d-8d17-6d0a46f17fc2" (UID: "461f7ac7-05b6-412d-8d17-6d0a46f17fc2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 01:38:10.763410 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:10.763195    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/461f7ac7-05b6-412d-8d17-6d0a46f17fc2-istiod-ca-cert\") pod \"461f7ac7-05b6-412d-8d17-6d0a46f17fc2\" (UID: \"461f7ac7-05b6-412d-8d17-6d0a46f17fc2\") "
Sep 19 01:38:10.763410 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:10.763257    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/461f7ac7-05b6-412d-8d17-6d0a46f17fc2-istio-envoy\") pod \"461f7ac7-05b6-412d-8d17-6d0a46f17fc2\" (UID: \"461f7ac7-05b6-412d-8d17-6d0a46f17fc2\") "
Sep 19 01:38:10.763410 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:10.763316    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/461f7ac7-05b6-412d-8d17-6d0a46f17fc2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "461f7ac7-05b6-412d-8d17-6d0a46f17fc2" (UID: "461f7ac7-05b6-412d-8d17-6d0a46f17fc2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 01:38:10.763410 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:10.763362    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/461f7ac7-05b6-412d-8d17-6d0a46f17fc2-istio-data" (OuterVolumeSpecName: "istio-data") pod "461f7ac7-05b6-412d-8d17-6d0a46f17fc2" (UID: "461f7ac7-05b6-412d-8d17-6d0a46f17fc2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 01:38:10.763636 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:10.763580    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/461f7ac7-05b6-412d-8d17-6d0a46f17fc2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "461f7ac7-05b6-412d-8d17-6d0a46f17fc2" (UID: "461f7ac7-05b6-412d-8d17-6d0a46f17fc2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 01:38:10.763636 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:10.763594    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/461f7ac7-05b6-412d-8d17-6d0a46f17fc2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "461f7ac7-05b6-412d-8d17-6d0a46f17fc2" (UID: "461f7ac7-05b6-412d-8d17-6d0a46f17fc2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 01:38:10.763737 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:10.763673    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/461f7ac7-05b6-412d-8d17-6d0a46f17fc2-istio-data\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:38:10.763737 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:10.763693    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/461f7ac7-05b6-412d-8d17-6d0a46f17fc2-workload-certs\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:38:10.763737 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:10.763704    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/461f7ac7-05b6-412d-8d17-6d0a46f17fc2-istiod-ca-cert\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:38:10.763737 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:10.763715    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/461f7ac7-05b6-412d-8d17-6d0a46f17fc2-workload-socket\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:38:10.763737 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:10.763728    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/461f7ac7-05b6-412d-8d17-6d0a46f17fc2-credential-socket\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:38:10.765591 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:10.765561    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/461f7ac7-05b6-412d-8d17-6d0a46f17fc2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "461f7ac7-05b6-412d-8d17-6d0a46f17fc2" (UID: "461f7ac7-05b6-412d-8d17-6d0a46f17fc2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 19 01:38:10.765700 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:10.765608    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/461f7ac7-05b6-412d-8d17-6d0a46f17fc2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "461f7ac7-05b6-412d-8d17-6d0a46f17fc2" (UID: "461f7ac7-05b6-412d-8d17-6d0a46f17fc2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 01:38:10.765700 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:10.765658    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/461f7ac7-05b6-412d-8d17-6d0a46f17fc2-kube-api-access-nhxf6" (OuterVolumeSpecName: "kube-api-access-nhxf6") pod "461f7ac7-05b6-412d-8d17-6d0a46f17fc2" (UID: "461f7ac7-05b6-412d-8d17-6d0a46f17fc2"). InnerVolumeSpecName "kube-api-access-nhxf6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 01:38:10.765995 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:10.765980    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/461f7ac7-05b6-412d-8d17-6d0a46f17fc2-istio-token" (OuterVolumeSpecName: "istio-token") pod "461f7ac7-05b6-412d-8d17-6d0a46f17fc2" (UID: "461f7ac7-05b6-412d-8d17-6d0a46f17fc2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 01:38:10.864220 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:10.864184    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhxf6\" (UniqueName: \"kubernetes.io/projected/461f7ac7-05b6-412d-8d17-6d0a46f17fc2-kube-api-access-nhxf6\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:38:10.864220 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:10.864219    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/461f7ac7-05b6-412d-8d17-6d0a46f17fc2-istio-envoy\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:38:10.864220 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:10.864229    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/461f7ac7-05b6-412d-8d17-6d0a46f17fc2-istio-token\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:38:10.864411 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:10.864236    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/461f7ac7-05b6-412d-8d17-6d0a46f17fc2-istio-podinfo\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:38:11.293807 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:11.293776    2581 generic.go:356] "Generic (PLEG): container finished" podID="461f7ac7-05b6-412d-8d17-6d0a46f17fc2" containerID="c27a8eef84c402a6b26c9fb1906c11b8c31c155acd96bbd8baa8fb07635a6da7" exitCode=0
Sep 19 01:38:11.294165 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:11.293825    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a7b600-openshift-default-c88cb89b8-jxm2q"
Sep 19 01:38:11.294165 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:11.293856    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a7b600-openshift-default-c88cb89b8-jxm2q" event={"ID":"461f7ac7-05b6-412d-8d17-6d0a46f17fc2","Type":"ContainerDied","Data":"c27a8eef84c402a6b26c9fb1906c11b8c31c155acd96bbd8baa8fb07635a6da7"}
Sep 19 01:38:11.294165 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:11.293923    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a7b600-openshift-default-c88cb89b8-jxm2q" event={"ID":"461f7ac7-05b6-412d-8d17-6d0a46f17fc2","Type":"ContainerDied","Data":"76a246d6f9dce647b92a6cfe88dc72a5259a0c18e2d9dad164d1f0f07c1ce322"}
Sep 19 01:38:11.294165 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:11.293944    2581 scope.go:117] "RemoveContainer" containerID="c27a8eef84c402a6b26c9fb1906c11b8c31c155acd96bbd8baa8fb07635a6da7"
Sep 19 01:38:11.302636 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:11.302619    2581 scope.go:117] "RemoveContainer" containerID="8f96814090cda0bb2759796fee43ab4933b836f613ab9364bfb62328725e9e74"
Sep 19 01:38:11.310671 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:11.310652    2581 scope.go:117] "RemoveContainer" containerID="c27a8eef84c402a6b26c9fb1906c11b8c31c155acd96bbd8baa8fb07635a6da7"
Sep 19 01:38:11.311035 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:38:11.311002    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c27a8eef84c402a6b26c9fb1906c11b8c31c155acd96bbd8baa8fb07635a6da7\": container with ID starting with c27a8eef84c402a6b26c9fb1906c11b8c31c155acd96bbd8baa8fb07635a6da7 not found: ID does not exist" containerID="c27a8eef84c402a6b26c9fb1906c11b8c31c155acd96bbd8baa8fb07635a6da7"
Sep 19 01:38:11.311152 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:11.311044    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c27a8eef84c402a6b26c9fb1906c11b8c31c155acd96bbd8baa8fb07635a6da7"} err="failed to get container status \"c27a8eef84c402a6b26c9fb1906c11b8c31c155acd96bbd8baa8fb07635a6da7\": rpc error: code = NotFound desc = could not find container \"c27a8eef84c402a6b26c9fb1906c11b8c31c155acd96bbd8baa8fb07635a6da7\": container with ID starting with c27a8eef84c402a6b26c9fb1906c11b8c31c155acd96bbd8baa8fb07635a6da7 not found: ID does not exist"
Sep 19 01:38:11.311152 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:11.311067    2581 scope.go:117] "RemoveContainer" containerID="8f96814090cda0bb2759796fee43ab4933b836f613ab9364bfb62328725e9e74"
Sep 19 01:38:11.311368 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:38:11.311343    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f96814090cda0bb2759796fee43ab4933b836f613ab9364bfb62328725e9e74\": container with ID starting with 8f96814090cda0bb2759796fee43ab4933b836f613ab9364bfb62328725e9e74 not found: ID does not exist" containerID="8f96814090cda0bb2759796fee43ab4933b836f613ab9364bfb62328725e9e74"
Sep 19 01:38:11.311483 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:11.311374    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f96814090cda0bb2759796fee43ab4933b836f613ab9364bfb62328725e9e74"} err="failed to get container status \"8f96814090cda0bb2759796fee43ab4933b836f613ab9364bfb62328725e9e74\": rpc error: code = NotFound desc = could not find container \"8f96814090cda0bb2759796fee43ab4933b836f613ab9364bfb62328725e9e74\": container with ID starting with 8f96814090cda0bb2759796fee43ab4933b836f613ab9364bfb62328725e9e74 not found: ID does not exist"
Sep 19 01:38:11.313051 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:11.313033    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a7b600-openshift-default-c88cb89b8-jxm2q"]
Sep 19 01:38:11.317379 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:11.317359    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-a7b600-openshift-default-c88cb89b8-jxm2q"]
Sep 19 01:38:11.610344 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:11.610313    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-751f6a/sub-test-model-751f6a-kserve-655995547-8qrtb"
Sep 19 01:38:11.610501 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:11.610355    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-751f6a/sub-test-model-751f6a-kserve-655995547-8qrtb"
Sep 19 01:38:11.622815 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:11.622795    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-751f6a/sub-test-model-751f6a-kserve-655995547-8qrtb"
Sep 19 01:38:12.310972 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:12.310945    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-751f6a/sub-test-model-751f6a-kserve-655995547-8qrtb"
Sep 19 01:38:13.277208 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:13.277169    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="461f7ac7-05b6-412d-8d17-6d0a46f17fc2" path="/var/lib/kubelet/pods/461f7ac7-05b6-412d-8d17-6d0a46f17fc2/volumes"
Sep 19 01:38:21.824381 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:21.824350    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-29d76c-5q2gv"]
Sep 19 01:38:21.826777 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:21.826756    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-29d76c-5q2gv"]
Sep 19 01:38:23.271935 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:23.271905    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ff80952-d515-4c6e-ba11-06eaa27ede33" path="/var/lib/kubelet/pods/0ff80952-d515-4c6e-ba11-06eaa27ede33/volumes"
Sep 19 01:38:26.388814 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:26.388779    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c9nsz"]
Sep 19 01:38:28.308182 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:28.308147    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-29d76c-openshift-default-5c894f7888-9hd9h"]
Sep 19 01:38:28.308929 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:28.308642    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-29d76c-openshift-default-5c894f7888-9hd9h" podUID="3b6722d3-0ee2-4f66-b026-7a343ac7d2a5" containerName="istio-proxy" containerID="cri-o://bf57a99a5fadeae25b08dea3df686e7e9638a3f5bd4ce3c89b62841c5c3e8e61" gracePeriod=30
Sep 19 01:38:28.722074 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:28.722040    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-10325b-wql2p"]
Sep 19 01:38:28.722367 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:28.722355    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="461f7ac7-05b6-412d-8d17-6d0a46f17fc2" containerName="istio-proxy"
Sep 19 01:38:28.722413 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:28.722369    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="461f7ac7-05b6-412d-8d17-6d0a46f17fc2" containerName="istio-proxy"
Sep 19 01:38:28.722413 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:28.722377    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ff80952-d515-4c6e-ba11-06eaa27ede33" containerName="revoke-keys"
Sep 19 01:38:28.722413 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:28.722383    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ff80952-d515-4c6e-ba11-06eaa27ede33" containerName="revoke-keys"
Sep 19 01:38:28.722514 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:28.722444    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="461f7ac7-05b6-412d-8d17-6d0a46f17fc2" containerName="istio-proxy"
Sep 19 01:38:28.722514 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:28.722451    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ff80952-d515-4c6e-ba11-06eaa27ede33" containerName="revoke-keys"
Sep 19 01:38:28.722514 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:28.722461    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="461f7ac7-05b6-412d-8d17-6d0a46f17fc2" containerName="istio-proxy"
Sep 19 01:38:28.725363 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:28.725347    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-10325b-wql2p"
Sep 19 01:38:28.728314 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:28.728294    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 19 01:38:28.729676 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:28.729661    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s7pvt\""
Sep 19 01:38:28.740134 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:28.740113    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-10325b-wql2p"]
Sep 19 01:38:28.812954 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:28.812928    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/854faa26-10cd-422c-9ecd-8c637ddf076c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-10325b-wql2p\" (UID: \"854faa26-10cd-422c-9ecd-8c637ddf076c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-10325b-wql2p"
Sep 19 01:38:28.913672 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:28.913640    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/854faa26-10cd-422c-9ecd-8c637ddf076c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-10325b-wql2p\" (UID: \"854faa26-10cd-422c-9ecd-8c637ddf076c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-10325b-wql2p"
Sep 19 01:38:28.914225 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:28.914208    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/854faa26-10cd-422c-9ecd-8c637ddf076c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-10325b-wql2p\" (UID: \"854faa26-10cd-422c-9ecd-8c637ddf076c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-10325b-wql2p"
Sep 19 01:38:29.034978 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:29.034918    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-10325b-wql2p"
Sep 19 01:38:29.308878 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:29.308840    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-10325b-wql2p"]
Sep 19 01:38:29.311200 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:38:29.311180    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod854faa26_10cd_422c_9ecd_8c637ddf076c.slice/crio-acac21aed26630e65c81ae26dc702ea033c6cfe8d71d6a4f699a4ab63a58301f WatchSource:0}: Error finding container acac21aed26630e65c81ae26dc702ea033c6cfe8d71d6a4f699a4ab63a58301f: Status 404 returned error can't find the container with id acac21aed26630e65c81ae26dc702ea033c6cfe8d71d6a4f699a4ab63a58301f
Sep 19 01:38:29.384913 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:29.384883    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-10325b-wql2p" event={"ID":"854faa26-10cd-422c-9ecd-8c637ddf076c","Type":"ContainerStarted","Data":"acac21aed26630e65c81ae26dc702ea033c6cfe8d71d6a4f699a4ab63a58301f"}
Sep 19 01:38:30.397667 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:30.397628    2581 generic.go:356] "Generic (PLEG): container finished" podID="854faa26-10cd-422c-9ecd-8c637ddf076c" containerID="20d2454e33a62d6d39069d49ddde919bc782b433078cd78e14e099335b7c389f" exitCode=0
Sep 19 01:38:30.398049 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:30.397684    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-10325b-wql2p" event={"ID":"854faa26-10cd-422c-9ecd-8c637ddf076c","Type":"ContainerDied","Data":"20d2454e33a62d6d39069d49ddde919bc782b433078cd78e14e099335b7c389f"}
Sep 19 01:38:30.997229 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:30.997191    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-bebef4/sub-test-model-bebef4-kserve-6dfb8869f5-qfz2p"]
Sep 19 01:38:30.997737 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:30.997715    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="461f7ac7-05b6-412d-8d17-6d0a46f17fc2" containerName="istio-proxy"
Sep 19 01:38:30.997817 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:30.997741    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="461f7ac7-05b6-412d-8d17-6d0a46f17fc2" containerName="istio-proxy"
Sep 19 01:38:31.000770 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:31.000754    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-bebef4/sub-test-model-bebef4-kserve-6dfb8869f5-qfz2p"
Sep 19 01:38:31.005160 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:31.005141    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-bebef4\"/\"openshift-service-ca.crt\""
Sep 19 01:38:31.005884 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:31.005840    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-bebef4\"/\"kube-root-ca.crt\""
Sep 19 01:38:31.006007 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:31.005939    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-bebef4\"/\"sub-test-model-bebef4-kserve-self-signed-certs\""
Sep 19 01:38:31.006212 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:31.006193    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-bebef4\"/\"default-dockercfg-lvlgd\""
Sep 19 01:38:31.012441 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:31.012422    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-bebef4/sub-test-model-bebef4-kserve-6dfb8869f5-qfz2p"]
Sep 19 01:38:31.131585 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:31.131547    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce81a90f-6d9c-4e1f-b899-4f34afb5567e-tmp-dir\") pod \"sub-test-model-bebef4-kserve-6dfb8869f5-qfz2p\" (UID: \"ce81a90f-6d9c-4e1f-b899-4f34afb5567e\") " pod="ai-tenant-e2e-shared-b-w3-bebef4/sub-test-model-bebef4-kserve-6dfb8869f5-qfz2p"
Sep 19 01:38:31.131585 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:31.131584    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sljjv\" (UniqueName: \"kubernetes.io/projected/ce81a90f-6d9c-4e1f-b899-4f34afb5567e-kube-api-access-sljjv\") pod \"sub-test-model-bebef4-kserve-6dfb8869f5-qfz2p\" (UID: \"ce81a90f-6d9c-4e1f-b899-4f34afb5567e\") " pod="ai-tenant-e2e-shared-b-w3-bebef4/sub-test-model-bebef4-kserve-6dfb8869f5-qfz2p"
Sep 19 01:38:31.131829 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:31.131617    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce81a90f-6d9c-4e1f-b899-4f34afb5567e-home\") pod \"sub-test-model-bebef4-kserve-6dfb8869f5-qfz2p\" (UID: \"ce81a90f-6d9c-4e1f-b899-4f34afb5567e\") " pod="ai-tenant-e2e-shared-b-w3-bebef4/sub-test-model-bebef4-kserve-6dfb8869f5-qfz2p"
Sep 19 01:38:31.131829 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:31.131662    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce81a90f-6d9c-4e1f-b899-4f34afb5567e-model-cache\") pod \"sub-test-model-bebef4-kserve-6dfb8869f5-qfz2p\" (UID: \"ce81a90f-6d9c-4e1f-b899-4f34afb5567e\") " pod="ai-tenant-e2e-shared-b-w3-bebef4/sub-test-model-bebef4-kserve-6dfb8869f5-qfz2p"
Sep 19 01:38:31.131829 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:31.131694    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce81a90f-6d9c-4e1f-b899-4f34afb5567e-dshm\") pod \"sub-test-model-bebef4-kserve-6dfb8869f5-qfz2p\" (UID: \"ce81a90f-6d9c-4e1f-b899-4f34afb5567e\") " pod="ai-tenant-e2e-shared-b-w3-bebef4/sub-test-model-bebef4-kserve-6dfb8869f5-qfz2p"
Sep 19 01:38:31.131829 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:31.131756    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce81a90f-6d9c-4e1f-b899-4f34afb5567e-tls-certs\") pod \"sub-test-model-bebef4-kserve-6dfb8869f5-qfz2p\" (UID: \"ce81a90f-6d9c-4e1f-b899-4f34afb5567e\") " pod="ai-tenant-e2e-shared-b-w3-bebef4/sub-test-model-bebef4-kserve-6dfb8869f5-qfz2p"
Sep 19 01:38:31.233025 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:31.232993    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce81a90f-6d9c-4e1f-b899-4f34afb5567e-tls-certs\") pod \"sub-test-model-bebef4-kserve-6dfb8869f5-qfz2p\" (UID: \"ce81a90f-6d9c-4e1f-b899-4f34afb5567e\") " pod="ai-tenant-e2e-shared-b-w3-bebef4/sub-test-model-bebef4-kserve-6dfb8869f5-qfz2p"
Sep 19 01:38:31.233186 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:31.233057    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce81a90f-6d9c-4e1f-b899-4f34afb5567e-tmp-dir\") pod \"sub-test-model-bebef4-kserve-6dfb8869f5-qfz2p\" (UID: \"ce81a90f-6d9c-4e1f-b899-4f34afb5567e\") " pod="ai-tenant-e2e-shared-b-w3-bebef4/sub-test-model-bebef4-kserve-6dfb8869f5-qfz2p"
Sep 19 01:38:31.233186 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:31.233077    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sljjv\" (UniqueName: \"kubernetes.io/projected/ce81a90f-6d9c-4e1f-b899-4f34afb5567e-kube-api-access-sljjv\") pod \"sub-test-model-bebef4-kserve-6dfb8869f5-qfz2p\" (UID: \"ce81a90f-6d9c-4e1f-b899-4f34afb5567e\") " pod="ai-tenant-e2e-shared-b-w3-bebef4/sub-test-model-bebef4-kserve-6dfb8869f5-qfz2p"
Sep 19 01:38:31.233186 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:31.233101    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce81a90f-6d9c-4e1f-b899-4f34afb5567e-home\") pod \"sub-test-model-bebef4-kserve-6dfb8869f5-qfz2p\" (UID: \"ce81a90f-6d9c-4e1f-b899-4f34afb5567e\") " pod="ai-tenant-e2e-shared-b-w3-bebef4/sub-test-model-bebef4-kserve-6dfb8869f5-qfz2p"
Sep 19 01:38:31.233186 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:31.233118    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce81a90f-6d9c-4e1f-b899-4f34afb5567e-model-cache\") pod \"sub-test-model-bebef4-kserve-6dfb8869f5-qfz2p\" (UID: \"ce81a90f-6d9c-4e1f-b899-4f34afb5567e\") " pod="ai-tenant-e2e-shared-b-w3-bebef4/sub-test-model-bebef4-kserve-6dfb8869f5-qfz2p"
Sep 19 01:38:31.233186 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:31.233138    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce81a90f-6d9c-4e1f-b899-4f34afb5567e-dshm\") pod \"sub-test-model-bebef4-kserve-6dfb8869f5-qfz2p\" (UID: \"ce81a90f-6d9c-4e1f-b899-4f34afb5567e\") " pod="ai-tenant-e2e-shared-b-w3-bebef4/sub-test-model-bebef4-kserve-6dfb8869f5-qfz2p"
Sep 19 01:38:31.233477 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:31.233453    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce81a90f-6d9c-4e1f-b899-4f34afb5567e-tmp-dir\") pod \"sub-test-model-bebef4-kserve-6dfb8869f5-qfz2p\" (UID: \"ce81a90f-6d9c-4e1f-b899-4f34afb5567e\") " pod="ai-tenant-e2e-shared-b-w3-bebef4/sub-test-model-bebef4-kserve-6dfb8869f5-qfz2p"
Sep 19 01:38:31.233477 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:31.233470    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce81a90f-6d9c-4e1f-b899-4f34afb5567e-home\") pod \"sub-test-model-bebef4-kserve-6dfb8869f5-qfz2p\" (UID: \"ce81a90f-6d9c-4e1f-b899-4f34afb5567e\") " pod="ai-tenant-e2e-shared-b-w3-bebef4/sub-test-model-bebef4-kserve-6dfb8869f5-qfz2p"
Sep 19 01:38:31.233558 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:31.233515    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce81a90f-6d9c-4e1f-b899-4f34afb5567e-model-cache\") pod \"sub-test-model-bebef4-kserve-6dfb8869f5-qfz2p\" (UID: \"ce81a90f-6d9c-4e1f-b899-4f34afb5567e\") " pod="ai-tenant-e2e-shared-b-w3-bebef4/sub-test-model-bebef4-kserve-6dfb8869f5-qfz2p"
Sep 19 01:38:31.235613 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:31.235590    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce81a90f-6d9c-4e1f-b899-4f34afb5567e-dshm\") pod \"sub-test-model-bebef4-kserve-6dfb8869f5-qfz2p\" (UID: \"ce81a90f-6d9c-4e1f-b899-4f34afb5567e\") " pod="ai-tenant-e2e-shared-b-w3-bebef4/sub-test-model-bebef4-kserve-6dfb8869f5-qfz2p"
Sep 19 01:38:31.235711 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:31.235623    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-bebef4\"/\"sub-test-model-bebef4-kserve-self-signed-certs\""
Sep 19 01:38:31.241458 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:31.241437    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-bebef4\"/\"kube-root-ca.crt\""
Sep 19 01:38:31.246367 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:31.246348    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce81a90f-6d9c-4e1f-b899-4f34afb5567e-tls-certs\") pod \"sub-test-model-bebef4-kserve-6dfb8869f5-qfz2p\" (UID: \"ce81a90f-6d9c-4e1f-b899-4f34afb5567e\") " pod="ai-tenant-e2e-shared-b-w3-bebef4/sub-test-model-bebef4-kserve-6dfb8869f5-qfz2p"
Sep 19 01:38:31.252378 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:31.252326    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-bebef4\"/\"openshift-service-ca.crt\""
Sep 19 01:38:31.262824 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:31.262803    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sljjv\" (UniqueName: \"kubernetes.io/projected/ce81a90f-6d9c-4e1f-b899-4f34afb5567e-kube-api-access-sljjv\") pod \"sub-test-model-bebef4-kserve-6dfb8869f5-qfz2p\" (UID: \"ce81a90f-6d9c-4e1f-b899-4f34afb5567e\") " pod="ai-tenant-e2e-shared-b-w3-bebef4/sub-test-model-bebef4-kserve-6dfb8869f5-qfz2p"
Sep 19 01:38:31.311991 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:31.311964    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-bebef4\"/\"default-dockercfg-lvlgd\""
Sep 19 01:38:31.319735 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:31.319714    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-bebef4/sub-test-model-bebef4-kserve-6dfb8869f5-qfz2p"
Sep 19 01:38:31.516085 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:31.516061    2581 scope.go:117] "RemoveContainer" containerID="7ad6ea9ccda53c113f865a2cd9ae68844013c2cefc9b6f008de981002746b265"
Sep 19 01:38:31.517748 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:31.517727    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-10325b-wql2p"
Sep 19 01:38:31.592135 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:31.592109    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-bebef4/sub-test-model-bebef4-kserve-6dfb8869f5-qfz2p"]
Sep 19 01:38:31.594003 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:38:31.593981    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podce81a90f_6d9c_4e1f_b899_4f34afb5567e.slice/crio-db08feb966ab9e8c2cf2996a4e866c3d25a36d40affc54ac445b6dea1ed20c3b WatchSource:0}: Error finding container db08feb966ab9e8c2cf2996a4e866c3d25a36d40affc54ac445b6dea1ed20c3b: Status 404 returned error can't find the container with id db08feb966ab9e8c2cf2996a4e866c3d25a36d40affc54ac445b6dea1ed20c3b
Sep 19 01:38:31.636322 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:31.636303    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/854faa26-10cd-422c-9ecd-8c637ddf076c-maas-api-service-ca\") pod \"854faa26-10cd-422c-9ecd-8c637ddf076c\" (UID: \"854faa26-10cd-422c-9ecd-8c637ddf076c\") "
Sep 19 01:38:31.636793 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:31.636768    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/854faa26-10cd-422c-9ecd-8c637ddf076c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "854faa26-10cd-422c-9ecd-8c637ddf076c" (UID: "854faa26-10cd-422c-9ecd-8c637ddf076c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 01:38:31.737927 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:31.737896    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/854faa26-10cd-422c-9ecd-8c637ddf076c-maas-api-service-ca\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:38:32.409160 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:32.409124    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-10325b-wql2p" event={"ID":"854faa26-10cd-422c-9ecd-8c637ddf076c","Type":"ContainerDied","Data":"acac21aed26630e65c81ae26dc702ea033c6cfe8d71d6a4f699a4ab63a58301f"}
Sep 19 01:38:32.409340 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:32.409170    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="acac21aed26630e65c81ae26dc702ea033c6cfe8d71d6a4f699a4ab63a58301f"
Sep 19 01:38:32.409340 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:32.409133    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-10325b-wql2p"
Sep 19 01:38:32.411041 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:32.411013    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-bebef4/sub-test-model-bebef4-kserve-6dfb8869f5-qfz2p" event={"ID":"ce81a90f-6d9c-4e1f-b899-4f34afb5567e","Type":"ContainerStarted","Data":"a561183cfcd8b1ce20412465cf021a9cb41e978d117a03b574476f7d9a3405e7"}
Sep 19 01:38:32.411165 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:32.411050    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-bebef4/sub-test-model-bebef4-kserve-6dfb8869f5-qfz2p" event={"ID":"ce81a90f-6d9c-4e1f-b899-4f34afb5567e","Type":"ContainerStarted","Data":"db08feb966ab9e8c2cf2996a4e866c3d25a36d40affc54ac445b6dea1ed20c3b"}
Sep 19 01:38:32.430627 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:32.430583    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-bebef4/sub-test-model-bebef4-kserve-6dfb8869f5-qfz2p" podStartSLOduration=2.430568238 podStartE2EDuration="2.430568238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 01:38:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 01:38:32.428950957 +0000 UTC m=+1563.316620049" watchObservedRunningTime="2026-09-19 01:38:32.430568238 +0000 UTC m=+1563.318237316"
Sep 19 01:38:33.551444 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:33.551420    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-29d76c-openshift-default-5c894f7888-9hd9h"
Sep 19 01:38:33.654713 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:33.654687    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b6722d3-0ee2-4f66-b026-7a343ac7d2a5-credential-socket\") pod \"3b6722d3-0ee2-4f66-b026-7a343ac7d2a5\" (UID: \"3b6722d3-0ee2-4f66-b026-7a343ac7d2a5\") "
Sep 19 01:38:33.654898 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:33.654733    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b6722d3-0ee2-4f66-b026-7a343ac7d2a5-istiod-ca-cert\") pod \"3b6722d3-0ee2-4f66-b026-7a343ac7d2a5\" (UID: \"3b6722d3-0ee2-4f66-b026-7a343ac7d2a5\") "
Sep 19 01:38:33.654898 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:33.654772    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b6722d3-0ee2-4f66-b026-7a343ac7d2a5-workload-certs\") pod \"3b6722d3-0ee2-4f66-b026-7a343ac7d2a5\" (UID: \"3b6722d3-0ee2-4f66-b026-7a343ac7d2a5\") "
Sep 19 01:38:33.654898 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:33.654829    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b6722d3-0ee2-4f66-b026-7a343ac7d2a5-istio-data\") pod \"3b6722d3-0ee2-4f66-b026-7a343ac7d2a5\" (UID: \"3b6722d3-0ee2-4f66-b026-7a343ac7d2a5\") "
Sep 19 01:38:33.655077 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:33.654909    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b6722d3-0ee2-4f66-b026-7a343ac7d2a5-istio-envoy\") pod \"3b6722d3-0ee2-4f66-b026-7a343ac7d2a5\" (UID: \"3b6722d3-0ee2-4f66-b026-7a343ac7d2a5\") "
Sep 19 01:38:33.655077 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:33.654941    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b6722d3-0ee2-4f66-b026-7a343ac7d2a5-istio-token\") pod \"3b6722d3-0ee2-4f66-b026-7a343ac7d2a5\" (UID: \"3b6722d3-0ee2-4f66-b026-7a343ac7d2a5\") "
Sep 19 01:38:33.655077 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:33.654968    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b6722d3-0ee2-4f66-b026-7a343ac7d2a5-workload-socket\") pod \"3b6722d3-0ee2-4f66-b026-7a343ac7d2a5\" (UID: \"3b6722d3-0ee2-4f66-b026-7a343ac7d2a5\") "
Sep 19 01:38:33.655077 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:33.654936    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b6722d3-0ee2-4f66-b026-7a343ac7d2a5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3b6722d3-0ee2-4f66-b026-7a343ac7d2a5" (UID: "3b6722d3-0ee2-4f66-b026-7a343ac7d2a5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 01:38:33.655077 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:33.655002    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l88br\" (UniqueName: \"kubernetes.io/projected/3b6722d3-0ee2-4f66-b026-7a343ac7d2a5-kube-api-access-l88br\") pod \"3b6722d3-0ee2-4f66-b026-7a343ac7d2a5\" (UID: \"3b6722d3-0ee2-4f66-b026-7a343ac7d2a5\") "
Sep 19 01:38:33.655077 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:33.655058    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b6722d3-0ee2-4f66-b026-7a343ac7d2a5-istio-podinfo\") pod \"3b6722d3-0ee2-4f66-b026-7a343ac7d2a5\" (UID: \"3b6722d3-0ee2-4f66-b026-7a343ac7d2a5\") "
Sep 19 01:38:33.655480 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:33.655091    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b6722d3-0ee2-4f66-b026-7a343ac7d2a5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3b6722d3-0ee2-4f66-b026-7a343ac7d2a5" (UID: "3b6722d3-0ee2-4f66-b026-7a343ac7d2a5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 01:38:33.655480 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:33.655199    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b6722d3-0ee2-4f66-b026-7a343ac7d2a5-istio-data" (OuterVolumeSpecName: "istio-data") pod "3b6722d3-0ee2-4f66-b026-7a343ac7d2a5" (UID: "3b6722d3-0ee2-4f66-b026-7a343ac7d2a5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 01:38:33.655480 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:33.655211    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b6722d3-0ee2-4f66-b026-7a343ac7d2a5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3b6722d3-0ee2-4f66-b026-7a343ac7d2a5" (UID: "3b6722d3-0ee2-4f66-b026-7a343ac7d2a5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 01:38:33.655480 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:33.655215    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b6722d3-0ee2-4f66-b026-7a343ac7d2a5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3b6722d3-0ee2-4f66-b026-7a343ac7d2a5" (UID: "3b6722d3-0ee2-4f66-b026-7a343ac7d2a5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 01:38:33.655480 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:33.655365    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b6722d3-0ee2-4f66-b026-7a343ac7d2a5-workload-socket\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:38:33.655480 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:33.655383    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b6722d3-0ee2-4f66-b026-7a343ac7d2a5-credential-socket\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:38:33.655480 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:33.655397    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b6722d3-0ee2-4f66-b026-7a343ac7d2a5-istiod-ca-cert\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:38:33.655480 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:33.655423    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b6722d3-0ee2-4f66-b026-7a343ac7d2a5-workload-certs\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:38:33.655480 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:33.655435    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b6722d3-0ee2-4f66-b026-7a343ac7d2a5-istio-data\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:38:33.657187 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:33.657159    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b6722d3-0ee2-4f66-b026-7a343ac7d2a5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3b6722d3-0ee2-4f66-b026-7a343ac7d2a5" (UID: "3b6722d3-0ee2-4f66-b026-7a343ac7d2a5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 01:38:33.657361 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:33.657343    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3b6722d3-0ee2-4f66-b026-7a343ac7d2a5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3b6722d3-0ee2-4f66-b026-7a343ac7d2a5" (UID: "3b6722d3-0ee2-4f66-b026-7a343ac7d2a5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 19 01:38:33.657661 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:33.657631    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b6722d3-0ee2-4f66-b026-7a343ac7d2a5-kube-api-access-l88br" (OuterVolumeSpecName: "kube-api-access-l88br") pod "3b6722d3-0ee2-4f66-b026-7a343ac7d2a5" (UID: "3b6722d3-0ee2-4f66-b026-7a343ac7d2a5"). InnerVolumeSpecName "kube-api-access-l88br". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 01:38:33.657661 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:33.657642    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b6722d3-0ee2-4f66-b026-7a343ac7d2a5-istio-token" (OuterVolumeSpecName: "istio-token") pod "3b6722d3-0ee2-4f66-b026-7a343ac7d2a5" (UID: "3b6722d3-0ee2-4f66-b026-7a343ac7d2a5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 01:38:33.756777 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:33.756743    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b6722d3-0ee2-4f66-b026-7a343ac7d2a5-istio-envoy\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:38:33.756777 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:33.756775    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b6722d3-0ee2-4f66-b026-7a343ac7d2a5-istio-token\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:38:33.756777 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:33.756783    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l88br\" (UniqueName: \"kubernetes.io/projected/3b6722d3-0ee2-4f66-b026-7a343ac7d2a5-kube-api-access-l88br\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:38:33.756972 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:33.756794    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b6722d3-0ee2-4f66-b026-7a343ac7d2a5-istio-podinfo\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:38:34.422028 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:34.421994    2581 generic.go:356] "Generic (PLEG): container finished" podID="3b6722d3-0ee2-4f66-b026-7a343ac7d2a5" containerID="bf57a99a5fadeae25b08dea3df686e7e9638a3f5bd4ce3c89b62841c5c3e8e61" exitCode=0
Sep 19 01:38:34.422204 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:34.422052    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-29d76c-openshift-default-5c894f7888-9hd9h"
Sep 19 01:38:34.422204 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:34.422069    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-29d76c-openshift-default-5c894f7888-9hd9h" event={"ID":"3b6722d3-0ee2-4f66-b026-7a343ac7d2a5","Type":"ContainerDied","Data":"bf57a99a5fadeae25b08dea3df686e7e9638a3f5bd4ce3c89b62841c5c3e8e61"}
Sep 19 01:38:34.422204 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:34.422137    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-29d76c-openshift-default-5c894f7888-9hd9h" event={"ID":"3b6722d3-0ee2-4f66-b026-7a343ac7d2a5","Type":"ContainerDied","Data":"e347f256a3b0e47a7f2989739bb9b9f03d6fa6e6005f3a47dafd49d3f6918dc7"}
Sep 19 01:38:34.422204 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:34.422159    2581 scope.go:117] "RemoveContainer" containerID="bf57a99a5fadeae25b08dea3df686e7e9638a3f5bd4ce3c89b62841c5c3e8e61"
Sep 19 01:38:34.432102 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:34.432045    2581 scope.go:117] "RemoveContainer" containerID="df07e1960f4778ff59a3ddeceb435131064a5a86457952be9c3cec8ee2f99e9e"
Sep 19 01:38:34.439885 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:34.439851    2581 scope.go:117] "RemoveContainer" containerID="bf57a99a5fadeae25b08dea3df686e7e9638a3f5bd4ce3c89b62841c5c3e8e61"
Sep 19 01:38:34.440154 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:38:34.440132    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf57a99a5fadeae25b08dea3df686e7e9638a3f5bd4ce3c89b62841c5c3e8e61\": container with ID starting with bf57a99a5fadeae25b08dea3df686e7e9638a3f5bd4ce3c89b62841c5c3e8e61 not found: ID does not exist" containerID="bf57a99a5fadeae25b08dea3df686e7e9638a3f5bd4ce3c89b62841c5c3e8e61"
Sep 19 01:38:34.440227 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:34.440165    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf57a99a5fadeae25b08dea3df686e7e9638a3f5bd4ce3c89b62841c5c3e8e61"} err="failed to get container status \"bf57a99a5fadeae25b08dea3df686e7e9638a3f5bd4ce3c89b62841c5c3e8e61\": rpc error: code = NotFound desc = could not find container \"bf57a99a5fadeae25b08dea3df686e7e9638a3f5bd4ce3c89b62841c5c3e8e61\": container with ID starting with bf57a99a5fadeae25b08dea3df686e7e9638a3f5bd4ce3c89b62841c5c3e8e61 not found: ID does not exist"
Sep 19 01:38:34.440227 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:34.440187    2581 scope.go:117] "RemoveContainer" containerID="df07e1960f4778ff59a3ddeceb435131064a5a86457952be9c3cec8ee2f99e9e"
Sep 19 01:38:34.440439 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:38:34.440418    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df07e1960f4778ff59a3ddeceb435131064a5a86457952be9c3cec8ee2f99e9e\": container with ID starting with df07e1960f4778ff59a3ddeceb435131064a5a86457952be9c3cec8ee2f99e9e not found: ID does not exist" containerID="df07e1960f4778ff59a3ddeceb435131064a5a86457952be9c3cec8ee2f99e9e"
Sep 19 01:38:34.440481 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:34.440446    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df07e1960f4778ff59a3ddeceb435131064a5a86457952be9c3cec8ee2f99e9e"} err="failed to get container status \"df07e1960f4778ff59a3ddeceb435131064a5a86457952be9c3cec8ee2f99e9e\": rpc error: code = NotFound desc = could not find container \"df07e1960f4778ff59a3ddeceb435131064a5a86457952be9c3cec8ee2f99e9e\": container with ID starting with df07e1960f4778ff59a3ddeceb435131064a5a86457952be9c3cec8ee2f99e9e not found: ID does not exist"
Sep 19 01:38:34.444199 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:34.444178    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-29d76c-openshift-default-5c894f7888-9hd9h"]
Sep 19 01:38:34.449025 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:34.449006    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-29d76c-openshift-default-5c894f7888-9hd9h"]
Sep 19 01:38:35.272388 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:35.272354    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b6722d3-0ee2-4f66-b026-7a343ac7d2a5" path="/var/lib/kubelet/pods/3b6722d3-0ee2-4f66-b026-7a343ac7d2a5/volumes"
Sep 19 01:38:38.087609 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:38.087577    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c9nsz"]
Sep 19 01:38:39.619716 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:39.619677    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-10325b-wql2p"]
Sep 19 01:38:39.622504 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:39.622479    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-10325b-wql2p"]
Sep 19 01:38:41.272210 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:41.272168    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="854faa26-10cd-422c-9ecd-8c637ddf076c" path="/var/lib/kubelet/pods/854faa26-10cd-422c-9ecd-8c637ddf076c/volumes"
Sep 19 01:38:41.320670 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:41.320649    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-bebef4/sub-test-model-bebef4-kserve-6dfb8869f5-qfz2p"
Sep 19 01:38:41.320778 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:41.320691    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-bebef4/sub-test-model-bebef4-kserve-6dfb8869f5-qfz2p"
Sep 19 01:38:41.333269 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:41.333250    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-bebef4/sub-test-model-bebef4-kserve-6dfb8869f5-qfz2p"
Sep 19 01:38:41.469306 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:41.469280    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-bebef4/sub-test-model-bebef4-kserve-6dfb8869f5-qfz2p"
Sep 19 01:38:47.338454 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:47.338425    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-0376cd-bzczg"]
Sep 19 01:38:47.338853 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:47.338708    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b6722d3-0ee2-4f66-b026-7a343ac7d2a5" containerName="istio-proxy"
Sep 19 01:38:47.338853 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:47.338719    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b6722d3-0ee2-4f66-b026-7a343ac7d2a5" containerName="istio-proxy"
Sep 19 01:38:47.338853 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:47.338736    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b6722d3-0ee2-4f66-b026-7a343ac7d2a5" containerName="istio-proxy"
Sep 19 01:38:47.338853 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:47.338742    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b6722d3-0ee2-4f66-b026-7a343ac7d2a5" containerName="istio-proxy"
Sep 19 01:38:47.338853 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:47.338754    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="854faa26-10cd-422c-9ecd-8c637ddf076c" containerName="revoke-keys"
Sep 19 01:38:47.338853 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:47.338759    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="854faa26-10cd-422c-9ecd-8c637ddf076c" containerName="revoke-keys"
Sep 19 01:38:47.338853 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:47.338812    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="854faa26-10cd-422c-9ecd-8c637ddf076c" containerName="revoke-keys"
Sep 19 01:38:47.338853 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:47.338821    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b6722d3-0ee2-4f66-b026-7a343ac7d2a5" containerName="istio-proxy"
Sep 19 01:38:47.343304 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:47.343285    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-0376cd-bzczg"
Sep 19 01:38:47.347755 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:47.347738    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s7pvt\""
Sep 19 01:38:47.348172 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:47.348158    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 19 01:38:47.357907 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:47.357884    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-0376cd-bzczg"]
Sep 19 01:38:47.471626 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:47.471600    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/890ae4ee-2848-428e-a267-e8ed129012b7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-0376cd-bzczg\" (UID: \"890ae4ee-2848-428e-a267-e8ed129012b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-0376cd-bzczg"
Sep 19 01:38:47.572514 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:47.572490    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/890ae4ee-2848-428e-a267-e8ed129012b7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-0376cd-bzczg\" (UID: \"890ae4ee-2848-428e-a267-e8ed129012b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-0376cd-bzczg"
Sep 19 01:38:47.573094 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:47.573077    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/890ae4ee-2848-428e-a267-e8ed129012b7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-0376cd-bzczg\" (UID: \"890ae4ee-2848-428e-a267-e8ed129012b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-0376cd-bzczg"
Sep 19 01:38:47.653255 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:47.653232    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-0376cd-bzczg"
Sep 19 01:38:47.936562 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:47.936535    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-0376cd-bzczg"]
Sep 19 01:38:47.938914 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:38:47.938886    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod890ae4ee_2848_428e_a267_e8ed129012b7.slice/crio-78fbc05a97583a12df3a4a371ea71b2ca7555d7b64cf1c939c7561e459576486 WatchSource:0}: Error finding container 78fbc05a97583a12df3a4a371ea71b2ca7555d7b64cf1c939c7561e459576486: Status 404 returned error can't find the container with id 78fbc05a97583a12df3a4a371ea71b2ca7555d7b64cf1c939c7561e459576486
Sep 19 01:38:48.490945 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:48.490855    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-0376cd-bzczg" event={"ID":"890ae4ee-2848-428e-a267-e8ed129012b7","Type":"ContainerStarted","Data":"106b266a10345a9c09016fa95bcf6b03fa61750a4c052db3bf71d689270a9bd6"}
Sep 19 01:38:48.490945 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:48.490926    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-0376cd-bzczg" event={"ID":"890ae4ee-2848-428e-a267-e8ed129012b7","Type":"ContainerStarted","Data":"78fbc05a97583a12df3a4a371ea71b2ca7555d7b64cf1c939c7561e459576486"}
Sep 19 01:38:48.508135 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:48.508090    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-0376cd-bzczg" podStartSLOduration=1.5080716920000001 podStartE2EDuration="1.508071692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 01:38:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 01:38:48.50718592 +0000 UTC m=+1579.394855021" watchObservedRunningTime="2026-09-19 01:38:48.508071692 +0000 UTC m=+1579.395740773"
Sep 19 01:38:49.497289 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:49.497255    2581 generic.go:356] "Generic (PLEG): container finished" podID="890ae4ee-2848-428e-a267-e8ed129012b7" containerID="106b266a10345a9c09016fa95bcf6b03fa61750a4c052db3bf71d689270a9bd6" exitCode=0
Sep 19 01:38:49.497289 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:49.497291    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-0376cd-bzczg" event={"ID":"890ae4ee-2848-428e-a267-e8ed129012b7","Type":"ContainerDied","Data":"106b266a10345a9c09016fa95bcf6b03fa61750a4c052db3bf71d689270a9bd6"}
Sep 19 01:38:50.631514 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:50.631490    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-0376cd-bzczg"
Sep 19 01:38:50.801351 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:50.801284    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/890ae4ee-2848-428e-a267-e8ed129012b7-maas-api-service-ca\") pod \"890ae4ee-2848-428e-a267-e8ed129012b7\" (UID: \"890ae4ee-2848-428e-a267-e8ed129012b7\") "
Sep 19 01:38:50.801659 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:50.801634    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/890ae4ee-2848-428e-a267-e8ed129012b7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "890ae4ee-2848-428e-a267-e8ed129012b7" (UID: "890ae4ee-2848-428e-a267-e8ed129012b7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 01:38:50.902072 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:50.902046    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/890ae4ee-2848-428e-a267-e8ed129012b7-maas-api-service-ca\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:38:51.508142 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:51.508110    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-0376cd-bzczg" event={"ID":"890ae4ee-2848-428e-a267-e8ed129012b7","Type":"ContainerDied","Data":"78fbc05a97583a12df3a4a371ea71b2ca7555d7b64cf1c939c7561e459576486"}
Sep 19 01:38:51.508283 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:51.508134    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-0376cd-bzczg"
Sep 19 01:38:51.508283 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:51.508156    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="78fbc05a97583a12df3a4a371ea71b2ca7555d7b64cf1c939c7561e459576486"
Sep 19 01:38:53.400304 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:53.400272    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c9nsz"]
Sep 19 01:38:58.828329 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:58.828291    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-0376cd-bzczg"]
Sep 19 01:38:58.831650 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:58.831621    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-0376cd-bzczg"]
Sep 19 01:38:59.272354 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:38:59.272323    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="890ae4ee-2848-428e-a267-e8ed129012b7" path="/var/lib/kubelet/pods/890ae4ee-2848-428e-a267-e8ed129012b7/volumes"
Sep 19 01:39:10.488615 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:10.488580    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c9nsz"]
Sep 19 01:39:13.067701 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:13.067670    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-751f6a/sub-test-model-751f6a-kserve-655995547-8qrtb"]
Sep 19 01:39:14.004982 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:14.004950    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-bebef4/sub-test-model-bebef4-kserve-6dfb8869f5-qfz2p"]
Sep 19 01:39:22.217633 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:22.217592    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c9nsz"]
Sep 19 01:39:28.087074 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:28.087027    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-751f6a/sub-test-model-751f6a-kserve-655995547-8qrtb" podUID="b1e40600-b622-4f22-84ba-a8892e02a87a" containerName="main" containerID="cri-o://ed79110f841f1452f7e5a6c650bffd0221cab207ef33717bc2581af222044a5a" gracePeriod=45
Sep 19 01:39:28.326121 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:28.326094    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-751f6a/sub-test-model-751f6a-kserve-655995547-8qrtb"
Sep 19 01:39:28.395643 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:28.395617    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1e40600-b622-4f22-84ba-a8892e02a87a-dshm\") pod \"b1e40600-b622-4f22-84ba-a8892e02a87a\" (UID: \"b1e40600-b622-4f22-84ba-a8892e02a87a\") "
Sep 19 01:39:28.395779 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:28.395659    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1e40600-b622-4f22-84ba-a8892e02a87a-tls-certs\") pod \"b1e40600-b622-4f22-84ba-a8892e02a87a\" (UID: \"b1e40600-b622-4f22-84ba-a8892e02a87a\") "
Sep 19 01:39:28.395779 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:28.395692    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1e40600-b622-4f22-84ba-a8892e02a87a-tmp-dir\") pod \"b1e40600-b622-4f22-84ba-a8892e02a87a\" (UID: \"b1e40600-b622-4f22-84ba-a8892e02a87a\") "
Sep 19 01:39:28.395779 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:28.395743    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1e40600-b622-4f22-84ba-a8892e02a87a-model-cache\") pod \"b1e40600-b622-4f22-84ba-a8892e02a87a\" (UID: \"b1e40600-b622-4f22-84ba-a8892e02a87a\") "
Sep 19 01:39:28.395936 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:28.395787    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1e40600-b622-4f22-84ba-a8892e02a87a-home\") pod \"b1e40600-b622-4f22-84ba-a8892e02a87a\" (UID: \"b1e40600-b622-4f22-84ba-a8892e02a87a\") "
Sep 19 01:39:28.395936 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:28.395828    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-chdt7\" (UniqueName: \"kubernetes.io/projected/b1e40600-b622-4f22-84ba-a8892e02a87a-kube-api-access-chdt7\") pod \"b1e40600-b622-4f22-84ba-a8892e02a87a\" (UID: \"b1e40600-b622-4f22-84ba-a8892e02a87a\") "
Sep 19 01:39:28.396043 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:28.396016    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1e40600-b622-4f22-84ba-a8892e02a87a-model-cache" (OuterVolumeSpecName: "model-cache") pod "b1e40600-b622-4f22-84ba-a8892e02a87a" (UID: "b1e40600-b622-4f22-84ba-a8892e02a87a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 01:39:28.396090 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:28.396031    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1e40600-b622-4f22-84ba-a8892e02a87a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b1e40600-b622-4f22-84ba-a8892e02a87a" (UID: "b1e40600-b622-4f22-84ba-a8892e02a87a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 01:39:28.396090 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:28.396067    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1e40600-b622-4f22-84ba-a8892e02a87a-home" (OuterVolumeSpecName: "home") pod "b1e40600-b622-4f22-84ba-a8892e02a87a" (UID: "b1e40600-b622-4f22-84ba-a8892e02a87a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 01:39:28.397916 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:28.396907    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1e40600-b622-4f22-84ba-a8892e02a87a-tmp-dir\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:39:28.397916 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:28.396939    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1e40600-b622-4f22-84ba-a8892e02a87a-model-cache\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:39:28.397916 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:28.396953    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1e40600-b622-4f22-84ba-a8892e02a87a-home\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:39:28.403567 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:28.403535    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1e40600-b622-4f22-84ba-a8892e02a87a-dshm" (OuterVolumeSpecName: "dshm") pod "b1e40600-b622-4f22-84ba-a8892e02a87a" (UID: "b1e40600-b622-4f22-84ba-a8892e02a87a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 01:39:28.403667 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:28.403580    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1e40600-b622-4f22-84ba-a8892e02a87a-kube-api-access-chdt7" (OuterVolumeSpecName: "kube-api-access-chdt7") pod "b1e40600-b622-4f22-84ba-a8892e02a87a" (UID: "b1e40600-b622-4f22-84ba-a8892e02a87a"). InnerVolumeSpecName "kube-api-access-chdt7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 01:39:28.403667 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:28.403593    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1e40600-b622-4f22-84ba-a8892e02a87a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b1e40600-b622-4f22-84ba-a8892e02a87a" (UID: "b1e40600-b622-4f22-84ba-a8892e02a87a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 01:39:28.498193 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:28.498161    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chdt7\" (UniqueName: \"kubernetes.io/projected/b1e40600-b622-4f22-84ba-a8892e02a87a-kube-api-access-chdt7\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:39:28.498193 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:28.498187    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1e40600-b622-4f22-84ba-a8892e02a87a-dshm\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:39:28.498193 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:28.498197    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1e40600-b622-4f22-84ba-a8892e02a87a-tls-certs\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:39:28.688079 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:28.688011    2581 generic.go:356] "Generic (PLEG): container finished" podID="b1e40600-b622-4f22-84ba-a8892e02a87a" containerID="ed79110f841f1452f7e5a6c650bffd0221cab207ef33717bc2581af222044a5a" exitCode=0
Sep 19 01:39:28.688079 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:28.688071    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-751f6a/sub-test-model-751f6a-kserve-655995547-8qrtb"
Sep 19 01:39:28.688215 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:28.688097    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-751f6a/sub-test-model-751f6a-kserve-655995547-8qrtb" event={"ID":"b1e40600-b622-4f22-84ba-a8892e02a87a","Type":"ContainerDied","Data":"ed79110f841f1452f7e5a6c650bffd0221cab207ef33717bc2581af222044a5a"}
Sep 19 01:39:28.688215 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:28.688145    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-751f6a/sub-test-model-751f6a-kserve-655995547-8qrtb" event={"ID":"b1e40600-b622-4f22-84ba-a8892e02a87a","Type":"ContainerDied","Data":"daeff7088fed8fe5d5ea08986e317377cdeabe08909f6cded4928691fa7f5c24"}
Sep 19 01:39:28.688215 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:28.688164    2581 scope.go:117] "RemoveContainer" containerID="ed79110f841f1452f7e5a6c650bffd0221cab207ef33717bc2581af222044a5a"
Sep 19 01:39:28.697496 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:28.697481    2581 scope.go:117] "RemoveContainer" containerID="ed79110f841f1452f7e5a6c650bffd0221cab207ef33717bc2581af222044a5a"
Sep 19 01:39:28.697753 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:39:28.697737    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed79110f841f1452f7e5a6c650bffd0221cab207ef33717bc2581af222044a5a\": container with ID starting with ed79110f841f1452f7e5a6c650bffd0221cab207ef33717bc2581af222044a5a not found: ID does not exist" containerID="ed79110f841f1452f7e5a6c650bffd0221cab207ef33717bc2581af222044a5a"
Sep 19 01:39:28.697804 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:28.697777    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed79110f841f1452f7e5a6c650bffd0221cab207ef33717bc2581af222044a5a"} err="failed to get container status \"ed79110f841f1452f7e5a6c650bffd0221cab207ef33717bc2581af222044a5a\": rpc error: code = NotFound desc = could not find container \"ed79110f841f1452f7e5a6c650bffd0221cab207ef33717bc2581af222044a5a\": container with ID starting with ed79110f841f1452f7e5a6c650bffd0221cab207ef33717bc2581af222044a5a not found: ID does not exist"
Sep 19 01:39:28.723421 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:28.723399    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-751f6a/sub-test-model-751f6a-kserve-655995547-8qrtb"]
Sep 19 01:39:28.733754 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:28.733734    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-751f6a/sub-test-model-751f6a-kserve-655995547-8qrtb"]
Sep 19 01:39:29.023760 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:29.023683    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-bebef4/sub-test-model-bebef4-kserve-6dfb8869f5-qfz2p" podUID="ce81a90f-6d9c-4e1f-b899-4f34afb5567e" containerName="main" containerID="cri-o://a561183cfcd8b1ce20412465cf021a9cb41e978d117a03b574476f7d9a3405e7" gracePeriod=45
Sep 19 01:39:29.263590 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:29.263565    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-bebef4/sub-test-model-bebef4-kserve-6dfb8869f5-qfz2p"
Sep 19 01:39:29.271803 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:29.271781    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1e40600-b622-4f22-84ba-a8892e02a87a" path="/var/lib/kubelet/pods/b1e40600-b622-4f22-84ba-a8892e02a87a/volumes"
Sep 19 01:39:29.305439 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:29.305384    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce81a90f-6d9c-4e1f-b899-4f34afb5567e-dshm\") pod \"ce81a90f-6d9c-4e1f-b899-4f34afb5567e\" (UID: \"ce81a90f-6d9c-4e1f-b899-4f34afb5567e\") "
Sep 19 01:39:29.305439 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:29.305430    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce81a90f-6d9c-4e1f-b899-4f34afb5567e-home\") pod \"ce81a90f-6d9c-4e1f-b899-4f34afb5567e\" (UID: \"ce81a90f-6d9c-4e1f-b899-4f34afb5567e\") "
Sep 19 01:39:29.305591 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:29.305492    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce81a90f-6d9c-4e1f-b899-4f34afb5567e-tmp-dir\") pod \"ce81a90f-6d9c-4e1f-b899-4f34afb5567e\" (UID: \"ce81a90f-6d9c-4e1f-b899-4f34afb5567e\") "
Sep 19 01:39:29.305591 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:29.305517    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce81a90f-6d9c-4e1f-b899-4f34afb5567e-model-cache\") pod \"ce81a90f-6d9c-4e1f-b899-4f34afb5567e\" (UID: \"ce81a90f-6d9c-4e1f-b899-4f34afb5567e\") "
Sep 19 01:39:29.305591 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:29.305566    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce81a90f-6d9c-4e1f-b899-4f34afb5567e-tls-certs\") pod \"ce81a90f-6d9c-4e1f-b899-4f34afb5567e\" (UID: \"ce81a90f-6d9c-4e1f-b899-4f34afb5567e\") "
Sep 19 01:39:29.305809 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:29.305602    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sljjv\" (UniqueName: \"kubernetes.io/projected/ce81a90f-6d9c-4e1f-b899-4f34afb5567e-kube-api-access-sljjv\") pod \"ce81a90f-6d9c-4e1f-b899-4f34afb5567e\" (UID: \"ce81a90f-6d9c-4e1f-b899-4f34afb5567e\") "
Sep 19 01:39:29.305809 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:29.305727    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce81a90f-6d9c-4e1f-b899-4f34afb5567e-home" (OuterVolumeSpecName: "home") pod "ce81a90f-6d9c-4e1f-b899-4f34afb5567e" (UID: "ce81a90f-6d9c-4e1f-b899-4f34afb5567e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 01:39:29.305809 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:29.305781    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce81a90f-6d9c-4e1f-b899-4f34afb5567e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ce81a90f-6d9c-4e1f-b899-4f34afb5567e" (UID: "ce81a90f-6d9c-4e1f-b899-4f34afb5567e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 01:39:29.305809 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:29.305798    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce81a90f-6d9c-4e1f-b899-4f34afb5567e-model-cache" (OuterVolumeSpecName: "model-cache") pod "ce81a90f-6d9c-4e1f-b899-4f34afb5567e" (UID: "ce81a90f-6d9c-4e1f-b899-4f34afb5567e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 01:39:29.306026 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:29.305946    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce81a90f-6d9c-4e1f-b899-4f34afb5567e-home\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:39:29.306026 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:29.305966    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce81a90f-6d9c-4e1f-b899-4f34afb5567e-tmp-dir\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:39:29.306026 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:29.305981    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce81a90f-6d9c-4e1f-b899-4f34afb5567e-model-cache\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:39:29.307794 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:29.307768    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce81a90f-6d9c-4e1f-b899-4f34afb5567e-dshm" (OuterVolumeSpecName: "dshm") pod "ce81a90f-6d9c-4e1f-b899-4f34afb5567e" (UID: "ce81a90f-6d9c-4e1f-b899-4f34afb5567e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 01:39:29.307928 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:29.307839    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce81a90f-6d9c-4e1f-b899-4f34afb5567e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ce81a90f-6d9c-4e1f-b899-4f34afb5567e" (UID: "ce81a90f-6d9c-4e1f-b899-4f34afb5567e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 01:39:29.307928 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:29.307913    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce81a90f-6d9c-4e1f-b899-4f34afb5567e-kube-api-access-sljjv" (OuterVolumeSpecName: "kube-api-access-sljjv") pod "ce81a90f-6d9c-4e1f-b899-4f34afb5567e" (UID: "ce81a90f-6d9c-4e1f-b899-4f34afb5567e"). InnerVolumeSpecName "kube-api-access-sljjv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 01:39:29.406485 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:29.406439    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce81a90f-6d9c-4e1f-b899-4f34afb5567e-tls-certs\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:39:29.406485 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:29.406480    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sljjv\" (UniqueName: \"kubernetes.io/projected/ce81a90f-6d9c-4e1f-b899-4f34afb5567e-kube-api-access-sljjv\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:39:29.406485 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:29.406490    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce81a90f-6d9c-4e1f-b899-4f34afb5567e-dshm\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:39:29.693385 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:29.693352    2581 generic.go:356] "Generic (PLEG): container finished" podID="ce81a90f-6d9c-4e1f-b899-4f34afb5567e" containerID="a561183cfcd8b1ce20412465cf021a9cb41e978d117a03b574476f7d9a3405e7" exitCode=0
Sep 19 01:39:29.693573 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:29.693394    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-bebef4/sub-test-model-bebef4-kserve-6dfb8869f5-qfz2p"
Sep 19 01:39:29.693573 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:29.693435    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-bebef4/sub-test-model-bebef4-kserve-6dfb8869f5-qfz2p" event={"ID":"ce81a90f-6d9c-4e1f-b899-4f34afb5567e","Type":"ContainerDied","Data":"a561183cfcd8b1ce20412465cf021a9cb41e978d117a03b574476f7d9a3405e7"}
Sep 19 01:39:29.693573 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:29.693475    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-bebef4/sub-test-model-bebef4-kserve-6dfb8869f5-qfz2p" event={"ID":"ce81a90f-6d9c-4e1f-b899-4f34afb5567e","Type":"ContainerDied","Data":"db08feb966ab9e8c2cf2996a4e866c3d25a36d40affc54ac445b6dea1ed20c3b"}
Sep 19 01:39:29.693573 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:29.693498    2581 scope.go:117] "RemoveContainer" containerID="a561183cfcd8b1ce20412465cf021a9cb41e978d117a03b574476f7d9a3405e7"
Sep 19 01:39:29.702911 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:29.702855    2581 scope.go:117] "RemoveContainer" containerID="a561183cfcd8b1ce20412465cf021a9cb41e978d117a03b574476f7d9a3405e7"
Sep 19 01:39:29.703217 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:39:29.703196    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a561183cfcd8b1ce20412465cf021a9cb41e978d117a03b574476f7d9a3405e7\": container with ID starting with a561183cfcd8b1ce20412465cf021a9cb41e978d117a03b574476f7d9a3405e7 not found: ID does not exist" containerID="a561183cfcd8b1ce20412465cf021a9cb41e978d117a03b574476f7d9a3405e7"
Sep 19 01:39:29.703297 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:29.703224    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a561183cfcd8b1ce20412465cf021a9cb41e978d117a03b574476f7d9a3405e7"} err="failed to get container status \"a561183cfcd8b1ce20412465cf021a9cb41e978d117a03b574476f7d9a3405e7\": rpc error: code = NotFound desc = could not find container \"a561183cfcd8b1ce20412465cf021a9cb41e978d117a03b574476f7d9a3405e7\": container with ID starting with a561183cfcd8b1ce20412465cf021a9cb41e978d117a03b574476f7d9a3405e7 not found: ID does not exist"
Sep 19 01:39:29.715729 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:29.715708    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-bebef4/sub-test-model-bebef4-kserve-6dfb8869f5-qfz2p"]
Sep 19 01:39:29.721369 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:29.721344    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-bebef4/sub-test-model-bebef4-kserve-6dfb8869f5-qfz2p"]
Sep 19 01:39:29.788003 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:29.787968    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c9nsz"]
Sep 19 01:39:31.271635 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:31.271607    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce81a90f-6d9c-4e1f-b899-4f34afb5567e" path="/var/lib/kubelet/pods/ce81a90f-6d9c-4e1f-b899-4f34afb5567e/volumes"
Sep 19 01:39:31.551644 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:31.551558    2581 scope.go:117] "RemoveContainer" containerID="d49496c10ac2e25a03114f39fe26a5e8fa1a3015d616c2baf1caaad732ea2797"
Sep 19 01:39:44.815881 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:44.815829    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-bebef4/rate-test-model-bebef4-kserve-b7445bc95-tvw6d"]
Sep 19 01:39:44.816364 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:44.816343    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce81a90f-6d9c-4e1f-b899-4f34afb5567e" containerName="main"
Sep 19 01:39:44.816445 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:44.816368    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce81a90f-6d9c-4e1f-b899-4f34afb5567e" containerName="main"
Sep 19 01:39:44.816445 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:44.816401    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1e40600-b622-4f22-84ba-a8892e02a87a" containerName="main"
Sep 19 01:39:44.816445 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:44.816410    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1e40600-b622-4f22-84ba-a8892e02a87a" containerName="main"
Sep 19 01:39:44.816445 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:44.816429    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="890ae4ee-2848-428e-a267-e8ed129012b7" containerName="revoke-keys"
Sep 19 01:39:44.816445 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:44.816438    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="890ae4ee-2848-428e-a267-e8ed129012b7" containerName="revoke-keys"
Sep 19 01:39:44.816696 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:44.816517    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b6722d3-0ee2-4f66-b026-7a343ac7d2a5" containerName="istio-proxy"
Sep 19 01:39:44.816696 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:44.816532    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1e40600-b622-4f22-84ba-a8892e02a87a" containerName="main"
Sep 19 01:39:44.816696 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:44.816542    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce81a90f-6d9c-4e1f-b899-4f34afb5567e" containerName="main"
Sep 19 01:39:44.816696 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:44.816551    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="890ae4ee-2848-428e-a267-e8ed129012b7" containerName="revoke-keys"
Sep 19 01:39:44.820783 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:44.820760    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-bebef4/rate-test-model-bebef4-kserve-b7445bc95-tvw6d"
Sep 19 01:39:44.823356 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:44.823328    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-bebef4\"/\"openshift-service-ca.crt\""
Sep 19 01:39:44.824110 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:44.824094    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-bebef4\"/\"kube-root-ca.crt\""
Sep 19 01:39:44.824340 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:44.824327    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-bebef4\"/\"default-dockercfg-lvlgd\""
Sep 19 01:39:44.824393 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:44.824384    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-bebef4\"/\"rate-test-model-bebef4-kserve-self-signed-certs\""
Sep 19 01:39:44.830546 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:44.830522    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-bebef4/rate-test-model-bebef4-kserve-b7445bc95-tvw6d"]
Sep 19 01:39:44.930968 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:44.930940    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/460f2660-4d2e-4290-8ba2-2cd70ca1551f-tls-certs\") pod \"rate-test-model-bebef4-kserve-b7445bc95-tvw6d\" (UID: \"460f2660-4d2e-4290-8ba2-2cd70ca1551f\") " pod="ai-tenant-e2e-shared-b-w3-bebef4/rate-test-model-bebef4-kserve-b7445bc95-tvw6d"
Sep 19 01:39:44.931079 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:44.930982    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/460f2660-4d2e-4290-8ba2-2cd70ca1551f-tmp-dir\") pod \"rate-test-model-bebef4-kserve-b7445bc95-tvw6d\" (UID: \"460f2660-4d2e-4290-8ba2-2cd70ca1551f\") " pod="ai-tenant-e2e-shared-b-w3-bebef4/rate-test-model-bebef4-kserve-b7445bc95-tvw6d"
Sep 19 01:39:44.931079 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:44.931019    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/460f2660-4d2e-4290-8ba2-2cd70ca1551f-model-cache\") pod \"rate-test-model-bebef4-kserve-b7445bc95-tvw6d\" (UID: \"460f2660-4d2e-4290-8ba2-2cd70ca1551f\") " pod="ai-tenant-e2e-shared-b-w3-bebef4/rate-test-model-bebef4-kserve-b7445bc95-tvw6d"
Sep 19 01:39:44.931079 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:44.931060    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/460f2660-4d2e-4290-8ba2-2cd70ca1551f-home\") pod \"rate-test-model-bebef4-kserve-b7445bc95-tvw6d\" (UID: \"460f2660-4d2e-4290-8ba2-2cd70ca1551f\") " pod="ai-tenant-e2e-shared-b-w3-bebef4/rate-test-model-bebef4-kserve-b7445bc95-tvw6d"
Sep 19 01:39:44.931184 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:44.931117    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4pjb\" (UniqueName: \"kubernetes.io/projected/460f2660-4d2e-4290-8ba2-2cd70ca1551f-kube-api-access-w4pjb\") pod \"rate-test-model-bebef4-kserve-b7445bc95-tvw6d\" (UID: \"460f2660-4d2e-4290-8ba2-2cd70ca1551f\") " pod="ai-tenant-e2e-shared-b-w3-bebef4/rate-test-model-bebef4-kserve-b7445bc95-tvw6d"
Sep 19 01:39:44.931224 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:44.931186    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/460f2660-4d2e-4290-8ba2-2cd70ca1551f-dshm\") pod \"rate-test-model-bebef4-kserve-b7445bc95-tvw6d\" (UID: \"460f2660-4d2e-4290-8ba2-2cd70ca1551f\") " pod="ai-tenant-e2e-shared-b-w3-bebef4/rate-test-model-bebef4-kserve-b7445bc95-tvw6d"
Sep 19 01:39:45.032235 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:45.032202    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4pjb\" (UniqueName: \"kubernetes.io/projected/460f2660-4d2e-4290-8ba2-2cd70ca1551f-kube-api-access-w4pjb\") pod \"rate-test-model-bebef4-kserve-b7445bc95-tvw6d\" (UID: \"460f2660-4d2e-4290-8ba2-2cd70ca1551f\") " pod="ai-tenant-e2e-shared-b-w3-bebef4/rate-test-model-bebef4-kserve-b7445bc95-tvw6d"
Sep 19 01:39:45.032369 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:45.032279    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/460f2660-4d2e-4290-8ba2-2cd70ca1551f-dshm\") pod \"rate-test-model-bebef4-kserve-b7445bc95-tvw6d\" (UID: \"460f2660-4d2e-4290-8ba2-2cd70ca1551f\") " pod="ai-tenant-e2e-shared-b-w3-bebef4/rate-test-model-bebef4-kserve-b7445bc95-tvw6d"
Sep 19 01:39:45.032369 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:45.032337    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/460f2660-4d2e-4290-8ba2-2cd70ca1551f-tls-certs\") pod \"rate-test-model-bebef4-kserve-b7445bc95-tvw6d\" (UID: \"460f2660-4d2e-4290-8ba2-2cd70ca1551f\") " pod="ai-tenant-e2e-shared-b-w3-bebef4/rate-test-model-bebef4-kserve-b7445bc95-tvw6d"
Sep 19 01:39:45.032369 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:45.032362    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/460f2660-4d2e-4290-8ba2-2cd70ca1551f-tmp-dir\") pod \"rate-test-model-bebef4-kserve-b7445bc95-tvw6d\" (UID: \"460f2660-4d2e-4290-8ba2-2cd70ca1551f\") " pod="ai-tenant-e2e-shared-b-w3-bebef4/rate-test-model-bebef4-kserve-b7445bc95-tvw6d"
Sep 19 01:39:45.032511 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:45.032394    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/460f2660-4d2e-4290-8ba2-2cd70ca1551f-model-cache\") pod \"rate-test-model-bebef4-kserve-b7445bc95-tvw6d\" (UID: \"460f2660-4d2e-4290-8ba2-2cd70ca1551f\") " pod="ai-tenant-e2e-shared-b-w3-bebef4/rate-test-model-bebef4-kserve-b7445bc95-tvw6d"
Sep 19 01:39:45.032511 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:45.032437    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/460f2660-4d2e-4290-8ba2-2cd70ca1551f-home\") pod \"rate-test-model-bebef4-kserve-b7445bc95-tvw6d\" (UID: \"460f2660-4d2e-4290-8ba2-2cd70ca1551f\") " pod="ai-tenant-e2e-shared-b-w3-bebef4/rate-test-model-bebef4-kserve-b7445bc95-tvw6d"
Sep 19 01:39:45.032762 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:45.032741    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/460f2660-4d2e-4290-8ba2-2cd70ca1551f-home\") pod \"rate-test-model-bebef4-kserve-b7445bc95-tvw6d\" (UID: \"460f2660-4d2e-4290-8ba2-2cd70ca1551f\") " pod="ai-tenant-e2e-shared-b-w3-bebef4/rate-test-model-bebef4-kserve-b7445bc95-tvw6d"
Sep 19 01:39:45.032855 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:45.032796    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/460f2660-4d2e-4290-8ba2-2cd70ca1551f-model-cache\") pod \"rate-test-model-bebef4-kserve-b7445bc95-tvw6d\" (UID: \"460f2660-4d2e-4290-8ba2-2cd70ca1551f\") " pod="ai-tenant-e2e-shared-b-w3-bebef4/rate-test-model-bebef4-kserve-b7445bc95-tvw6d"
Sep 19 01:39:45.032855 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:45.032811    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/460f2660-4d2e-4290-8ba2-2cd70ca1551f-tmp-dir\") pod \"rate-test-model-bebef4-kserve-b7445bc95-tvw6d\" (UID: \"460f2660-4d2e-4290-8ba2-2cd70ca1551f\") " pod="ai-tenant-e2e-shared-b-w3-bebef4/rate-test-model-bebef4-kserve-b7445bc95-tvw6d"
Sep 19 01:39:45.034836 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:45.034819    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/460f2660-4d2e-4290-8ba2-2cd70ca1551f-dshm\") pod \"rate-test-model-bebef4-kserve-b7445bc95-tvw6d\" (UID: \"460f2660-4d2e-4290-8ba2-2cd70ca1551f\") " pod="ai-tenant-e2e-shared-b-w3-bebef4/rate-test-model-bebef4-kserve-b7445bc95-tvw6d"
Sep 19 01:39:45.034912 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:45.034886    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/460f2660-4d2e-4290-8ba2-2cd70ca1551f-tls-certs\") pod \"rate-test-model-bebef4-kserve-b7445bc95-tvw6d\" (UID: \"460f2660-4d2e-4290-8ba2-2cd70ca1551f\") " pod="ai-tenant-e2e-shared-b-w3-bebef4/rate-test-model-bebef4-kserve-b7445bc95-tvw6d"
Sep 19 01:39:45.040495 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:45.040469    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4pjb\" (UniqueName: \"kubernetes.io/projected/460f2660-4d2e-4290-8ba2-2cd70ca1551f-kube-api-access-w4pjb\") pod \"rate-test-model-bebef4-kserve-b7445bc95-tvw6d\" (UID: \"460f2660-4d2e-4290-8ba2-2cd70ca1551f\") " pod="ai-tenant-e2e-shared-b-w3-bebef4/rate-test-model-bebef4-kserve-b7445bc95-tvw6d"
Sep 19 01:39:45.132125 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:45.132100    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-bebef4/rate-test-model-bebef4-kserve-b7445bc95-tvw6d"
Sep 19 01:39:45.401395 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:45.401373    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-bebef4/rate-test-model-bebef4-kserve-b7445bc95-tvw6d"]
Sep 19 01:39:45.403971 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:39:45.403939    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod460f2660_4d2e_4290_8ba2_2cd70ca1551f.slice/crio-5fd54aa21430aca980abe8aa54cfd8ffc043ed8becac565ebc44d7c1c0e77181 WatchSource:0}: Error finding container 5fd54aa21430aca980abe8aa54cfd8ffc043ed8becac565ebc44d7c1c0e77181: Status 404 returned error can't find the container with id 5fd54aa21430aca980abe8aa54cfd8ffc043ed8becac565ebc44d7c1c0e77181
Sep 19 01:39:45.771082 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:45.770997    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-bebef4/rate-test-model-bebef4-kserve-b7445bc95-tvw6d" event={"ID":"460f2660-4d2e-4290-8ba2-2cd70ca1551f","Type":"ContainerStarted","Data":"9d517d233030bb6291ab1975925695495f9b60d78715dd853e5d10bff0ef791e"}
Sep 19 01:39:45.771082 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:45.771038    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-bebef4/rate-test-model-bebef4-kserve-b7445bc95-tvw6d" event={"ID":"460f2660-4d2e-4290-8ba2-2cd70ca1551f","Type":"ContainerStarted","Data":"5fd54aa21430aca980abe8aa54cfd8ffc043ed8becac565ebc44d7c1c0e77181"}
Sep 19 01:39:45.790057 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:45.790015    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-bebef4/rate-test-model-bebef4-kserve-b7445bc95-tvw6d" podStartSLOduration=1.789999242 podStartE2EDuration="1.789999242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 01:39:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 01:39:45.788943941 +0000 UTC m=+1636.676613039" watchObservedRunningTime="2026-09-19 01:39:45.789999242 +0000 UTC m=+1636.677668319"
Sep 19 01:39:48.491672 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:48.491636    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c9nsz"]
Sep 19 01:39:55.132951 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:55.132916    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-bebef4/rate-test-model-bebef4-kserve-b7445bc95-tvw6d"
Sep 19 01:39:55.133369 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:55.132964    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-bebef4/rate-test-model-bebef4-kserve-b7445bc95-tvw6d"
Sep 19 01:39:55.145402 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:55.145378    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-bebef4/rate-test-model-bebef4-kserve-b7445bc95-tvw6d"
Sep 19 01:39:55.829161 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:39:55.829130    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-bebef4/rate-test-model-bebef4-kserve-b7445bc95-tvw6d"
Sep 19 01:40:00.684835 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:40:00.684797    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c9nsz"]
Sep 19 01:40:15.699403 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:40:15.699358    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c9nsz"]
Sep 19 01:40:23.147987 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:40:23.147952    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-bebef4/rate-test-model-bebef4-kserve-b7445bc95-tvw6d"]
Sep 19 01:40:26.842559 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:40:26.842523    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c9nsz"]
Sep 19 01:40:31.583428 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:40:31.583398    2581 scope.go:117] "RemoveContainer" containerID="562a61cccc510877b523ab935b17c05760393a428974c83e1693df8e4f831ac6"
Sep 19 01:40:38.166654 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:40:38.166611    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-bebef4/rate-test-model-bebef4-kserve-b7445bc95-tvw6d" podUID="460f2660-4d2e-4290-8ba2-2cd70ca1551f" containerName="main" containerID="cri-o://9d517d233030bb6291ab1975925695495f9b60d78715dd853e5d10bff0ef791e" gracePeriod=45
Sep 19 01:40:38.410100 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:40:38.410073    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-bebef4/rate-test-model-bebef4-kserve-b7445bc95-tvw6d"
Sep 19 01:40:38.457429 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:40:38.457362    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w4pjb\" (UniqueName: \"kubernetes.io/projected/460f2660-4d2e-4290-8ba2-2cd70ca1551f-kube-api-access-w4pjb\") pod \"460f2660-4d2e-4290-8ba2-2cd70ca1551f\" (UID: \"460f2660-4d2e-4290-8ba2-2cd70ca1551f\") "
Sep 19 01:40:38.457429 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:40:38.457396    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/460f2660-4d2e-4290-8ba2-2cd70ca1551f-model-cache\") pod \"460f2660-4d2e-4290-8ba2-2cd70ca1551f\" (UID: \"460f2660-4d2e-4290-8ba2-2cd70ca1551f\") "
Sep 19 01:40:38.457429 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:40:38.457430    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/460f2660-4d2e-4290-8ba2-2cd70ca1551f-dshm\") pod \"460f2660-4d2e-4290-8ba2-2cd70ca1551f\" (UID: \"460f2660-4d2e-4290-8ba2-2cd70ca1551f\") "
Sep 19 01:40:38.457615 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:40:38.457446    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/460f2660-4d2e-4290-8ba2-2cd70ca1551f-tmp-dir\") pod \"460f2660-4d2e-4290-8ba2-2cd70ca1551f\" (UID: \"460f2660-4d2e-4290-8ba2-2cd70ca1551f\") "
Sep 19 01:40:38.457615 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:40:38.457547    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/460f2660-4d2e-4290-8ba2-2cd70ca1551f-tls-certs\") pod \"460f2660-4d2e-4290-8ba2-2cd70ca1551f\" (UID: \"460f2660-4d2e-4290-8ba2-2cd70ca1551f\") "
Sep 19 01:40:38.457615 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:40:38.457577    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/460f2660-4d2e-4290-8ba2-2cd70ca1551f-home\") pod \"460f2660-4d2e-4290-8ba2-2cd70ca1551f\" (UID: \"460f2660-4d2e-4290-8ba2-2cd70ca1551f\") "
Sep 19 01:40:38.457776 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:40:38.457637    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/460f2660-4d2e-4290-8ba2-2cd70ca1551f-model-cache" (OuterVolumeSpecName: "model-cache") pod "460f2660-4d2e-4290-8ba2-2cd70ca1551f" (UID: "460f2660-4d2e-4290-8ba2-2cd70ca1551f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 01:40:38.457835 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:40:38.457783    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/460f2660-4d2e-4290-8ba2-2cd70ca1551f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "460f2660-4d2e-4290-8ba2-2cd70ca1551f" (UID: "460f2660-4d2e-4290-8ba2-2cd70ca1551f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 01:40:38.457919 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:40:38.457842    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/460f2660-4d2e-4290-8ba2-2cd70ca1551f-home" (OuterVolumeSpecName: "home") pod "460f2660-4d2e-4290-8ba2-2cd70ca1551f" (UID: "460f2660-4d2e-4290-8ba2-2cd70ca1551f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 01:40:38.457970 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:40:38.457941    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/460f2660-4d2e-4290-8ba2-2cd70ca1551f-home\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:40:38.457970 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:40:38.457960    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/460f2660-4d2e-4290-8ba2-2cd70ca1551f-model-cache\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:40:38.458050 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:40:38.457978    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/460f2660-4d2e-4290-8ba2-2cd70ca1551f-tmp-dir\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:40:38.459491 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:40:38.459466    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/460f2660-4d2e-4290-8ba2-2cd70ca1551f-dshm" (OuterVolumeSpecName: "dshm") pod "460f2660-4d2e-4290-8ba2-2cd70ca1551f" (UID: "460f2660-4d2e-4290-8ba2-2cd70ca1551f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 01:40:38.459614 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:40:38.459594    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/460f2660-4d2e-4290-8ba2-2cd70ca1551f-kube-api-access-w4pjb" (OuterVolumeSpecName: "kube-api-access-w4pjb") pod "460f2660-4d2e-4290-8ba2-2cd70ca1551f" (UID: "460f2660-4d2e-4290-8ba2-2cd70ca1551f"). InnerVolumeSpecName "kube-api-access-w4pjb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 01:40:38.459677 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:40:38.459637    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/460f2660-4d2e-4290-8ba2-2cd70ca1551f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "460f2660-4d2e-4290-8ba2-2cd70ca1551f" (UID: "460f2660-4d2e-4290-8ba2-2cd70ca1551f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 01:40:38.558933 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:40:38.558910    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4pjb\" (UniqueName: \"kubernetes.io/projected/460f2660-4d2e-4290-8ba2-2cd70ca1551f-kube-api-access-w4pjb\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:40:38.558933 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:40:38.558933    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/460f2660-4d2e-4290-8ba2-2cd70ca1551f-dshm\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:40:38.559073 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:40:38.558943    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/460f2660-4d2e-4290-8ba2-2cd70ca1551f-tls-certs\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:40:39.021561 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:40:39.021525    2581 generic.go:356] "Generic (PLEG): container finished" podID="460f2660-4d2e-4290-8ba2-2cd70ca1551f" containerID="9d517d233030bb6291ab1975925695495f9b60d78715dd853e5d10bff0ef791e" exitCode=0
Sep 19 01:40:39.021752 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:40:39.021574    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-bebef4/rate-test-model-bebef4-kserve-b7445bc95-tvw6d"
Sep 19 01:40:39.021752 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:40:39.021609    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-bebef4/rate-test-model-bebef4-kserve-b7445bc95-tvw6d" event={"ID":"460f2660-4d2e-4290-8ba2-2cd70ca1551f","Type":"ContainerDied","Data":"9d517d233030bb6291ab1975925695495f9b60d78715dd853e5d10bff0ef791e"}
Sep 19 01:40:39.021752 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:40:39.021657    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-bebef4/rate-test-model-bebef4-kserve-b7445bc95-tvw6d" event={"ID":"460f2660-4d2e-4290-8ba2-2cd70ca1551f","Type":"ContainerDied","Data":"5fd54aa21430aca980abe8aa54cfd8ffc043ed8becac565ebc44d7c1c0e77181"}
Sep 19 01:40:39.021752 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:40:39.021675    2581 scope.go:117] "RemoveContainer" containerID="9d517d233030bb6291ab1975925695495f9b60d78715dd853e5d10bff0ef791e"
Sep 19 01:40:39.030673 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:40:39.030656    2581 scope.go:117] "RemoveContainer" containerID="9d517d233030bb6291ab1975925695495f9b60d78715dd853e5d10bff0ef791e"
Sep 19 01:40:39.030994 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:40:39.030961    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d517d233030bb6291ab1975925695495f9b60d78715dd853e5d10bff0ef791e\": container with ID starting with 9d517d233030bb6291ab1975925695495f9b60d78715dd853e5d10bff0ef791e not found: ID does not exist" containerID="9d517d233030bb6291ab1975925695495f9b60d78715dd853e5d10bff0ef791e"
Sep 19 01:40:39.031062 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:40:39.031000    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d517d233030bb6291ab1975925695495f9b60d78715dd853e5d10bff0ef791e"} err="failed to get container status \"9d517d233030bb6291ab1975925695495f9b60d78715dd853e5d10bff0ef791e\": rpc error: code = NotFound desc = could not find container \"9d517d233030bb6291ab1975925695495f9b60d78715dd853e5d10bff0ef791e\": container with ID starting with 9d517d233030bb6291ab1975925695495f9b60d78715dd853e5d10bff0ef791e not found: ID does not exist"
Sep 19 01:40:39.043055 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:40:39.043032    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-bebef4/rate-test-model-bebef4-kserve-b7445bc95-tvw6d"]
Sep 19 01:40:39.049288 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:40:39.049263    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-bebef4/rate-test-model-bebef4-kserve-b7445bc95-tvw6d"]
Sep 19 01:40:39.192024 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:40:39.191992    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c9nsz"]
Sep 19 01:40:39.271460 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:40:39.271430    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="460f2660-4d2e-4290-8ba2-2cd70ca1551f" path="/var/lib/kubelet/pods/460f2660-4d2e-4290-8ba2-2cd70ca1551f/volumes"
Sep 19 01:41:04.426173 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:41:04.426139    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-14de36-77dfd7748b-4q4zn"]
Sep 19 01:41:04.426581 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:41:04.426450    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="460f2660-4d2e-4290-8ba2-2cd70ca1551f" containerName="main"
Sep 19 01:41:04.426581 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:41:04.426461    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="460f2660-4d2e-4290-8ba2-2cd70ca1551f" containerName="main"
Sep 19 01:41:04.426581 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:41:04.426518    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="460f2660-4d2e-4290-8ba2-2cd70ca1551f" containerName="main"
Sep 19 01:41:04.429376 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:41:04.429358    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-14de36-77dfd7748b-4q4zn"
Sep 19 01:41:04.432829 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:41:04.432809    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-14de36\""
Sep 19 01:41:04.433003 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:41:04.432982    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-14de36-dockercfg-f9vdn\""
Sep 19 01:41:04.449221 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:41:04.449192    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-14de36-77dfd7748b-4q4zn"]
Sep 19 01:41:04.572406 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:41:04.572368    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62sjs\" (UniqueName: \"kubernetes.io/projected/929b84a7-8e7f-4526-aa91-23d6c7de5de0-kube-api-access-62sjs\") pod \"payload-pre-processing-e2e-ipp-b-14de36-77dfd7748b-4q4zn\" (UID: \"929b84a7-8e7f-4526-aa91-23d6c7de5de0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-14de36-77dfd7748b-4q4zn"
Sep 19 01:41:04.572581 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:41:04.572497    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/929b84a7-8e7f-4526-aa91-23d6c7de5de0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-14de36-77dfd7748b-4q4zn\" (UID: \"929b84a7-8e7f-4526-aa91-23d6c7de5de0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-14de36-77dfd7748b-4q4zn"
Sep 19 01:41:04.673871 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:41:04.673842    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62sjs\" (UniqueName: \"kubernetes.io/projected/929b84a7-8e7f-4526-aa91-23d6c7de5de0-kube-api-access-62sjs\") pod \"payload-pre-processing-e2e-ipp-b-14de36-77dfd7748b-4q4zn\" (UID: \"929b84a7-8e7f-4526-aa91-23d6c7de5de0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-14de36-77dfd7748b-4q4zn"
Sep 19 01:41:04.673994 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:41:04.673925    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/929b84a7-8e7f-4526-aa91-23d6c7de5de0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-14de36-77dfd7748b-4q4zn\" (UID: \"929b84a7-8e7f-4526-aa91-23d6c7de5de0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-14de36-77dfd7748b-4q4zn"
Sep 19 01:41:04.674543 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:41:04.674523    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/929b84a7-8e7f-4526-aa91-23d6c7de5de0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-14de36-77dfd7748b-4q4zn\" (UID: \"929b84a7-8e7f-4526-aa91-23d6c7de5de0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-14de36-77dfd7748b-4q4zn"
Sep 19 01:41:04.684432 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:41:04.684385    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62sjs\" (UniqueName: \"kubernetes.io/projected/929b84a7-8e7f-4526-aa91-23d6c7de5de0-kube-api-access-62sjs\") pod \"payload-pre-processing-e2e-ipp-b-14de36-77dfd7748b-4q4zn\" (UID: \"929b84a7-8e7f-4526-aa91-23d6c7de5de0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-14de36-77dfd7748b-4q4zn"
Sep 19 01:41:04.739599 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:41:04.739570    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-14de36-77dfd7748b-4q4zn"
Sep 19 01:41:05.021323 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:41:05.021284    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-14de36-77dfd7748b-4q4zn"]
Sep 19 01:41:05.026964 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:41:05.026934    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod929b84a7_8e7f_4526_aa91_23d6c7de5de0.slice/crio-0f9a6d09a94f670559e5add2ab86cd3c5a6083680abde5106493f1e5d8967ed5 WatchSource:0}: Error finding container 0f9a6d09a94f670559e5add2ab86cd3c5a6083680abde5106493f1e5d8967ed5: Status 404 returned error can't find the container with id 0f9a6d09a94f670559e5add2ab86cd3c5a6083680abde5106493f1e5d8967ed5
Sep 19 01:41:05.139847 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:41:05.139812    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-14de36-77dfd7748b-4q4zn" event={"ID":"929b84a7-8e7f-4526-aa91-23d6c7de5de0","Type":"ContainerStarted","Data":"54405da5190cced0f65cb8132b369c5cd190599766965441f4d6cadc111b4ec3"}
Sep 19 01:41:05.139847 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:41:05.139872    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-14de36-77dfd7748b-4q4zn" event={"ID":"929b84a7-8e7f-4526-aa91-23d6c7de5de0","Type":"ContainerStarted","Data":"0f9a6d09a94f670559e5add2ab86cd3c5a6083680abde5106493f1e5d8967ed5"}
Sep 19 01:41:05.140063 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:41:05.139911    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-14de36-77dfd7748b-4q4zn"
Sep 19 01:41:05.156749 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:41:05.156700    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-14de36-77dfd7748b-4q4zn" podStartSLOduration=1.156684264 podStartE2EDuration="1.156684264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 01:41:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 01:41:05.156233056 +0000 UTC m=+1716.043902149" watchObservedRunningTime="2026-09-19 01:41:05.156684264 +0000 UTC m=+1716.044353383"
Sep 19 01:41:08.803167 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:41:08.803133    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c9nsz"]
Sep 19 01:41:16.146776 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:41:16.146746    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-14de36-77dfd7748b-4q4zn"
Sep 19 01:41:53.887950 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:41:53.887901    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-a4edd7/ipp-route-a4edd7-kserve-586cb699f9-f8prv"]
Sep 19 01:41:53.890982 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:41:53.890958    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-a4edd7/ipp-route-a4edd7-kserve-586cb699f9-f8prv"
Sep 19 01:41:53.893743 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:41:53.893718    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-a4edd7\"/\"default-dockercfg-2z7xb\""
Sep 19 01:41:53.894640 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:41:53.894622    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-a4edd7\"/\"ipp-route-a4edd7-kserve-self-signed-certs\""
Sep 19 01:41:53.894732 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:41:53.894622    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-a4edd7\"/\"kube-root-ca.crt\""
Sep 19 01:41:53.894732 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:41:53.894692    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-a4edd7\"/\"openshift-service-ca.crt\""
Sep 19 01:41:53.903776 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:41:53.903753    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-a4edd7/ipp-route-a4edd7-kserve-586cb699f9-f8prv"]
Sep 19 01:41:54.056388 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:41:54.056358    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7c7785f-649c-4d4f-b007-ea507b60b88e-home\") pod \"ipp-route-a4edd7-kserve-586cb699f9-f8prv\" (UID: \"b7c7785f-649c-4d4f-b007-ea507b60b88e\") " pod="ai-tenant-e2e-ipp-a-a4edd7/ipp-route-a4edd7-kserve-586cb699f9-f8prv"
Sep 19 01:41:54.056523 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:41:54.056413    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7c7785f-649c-4d4f-b007-ea507b60b88e-tls-certs\") pod \"ipp-route-a4edd7-kserve-586cb699f9-f8prv\" (UID: \"b7c7785f-649c-4d4f-b007-ea507b60b88e\") " pod="ai-tenant-e2e-ipp-a-a4edd7/ipp-route-a4edd7-kserve-586cb699f9-f8prv"
Sep 19 01:41:54.056523 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:41:54.056480    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7c7785f-649c-4d4f-b007-ea507b60b88e-dshm\") pod \"ipp-route-a4edd7-kserve-586cb699f9-f8prv\" (UID: \"b7c7785f-649c-4d4f-b007-ea507b60b88e\") " pod="ai-tenant-e2e-ipp-a-a4edd7/ipp-route-a4edd7-kserve-586cb699f9-f8prv"
Sep 19 01:41:54.056615 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:41:54.056539    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bc687\" (UniqueName: \"kubernetes.io/projected/b7c7785f-649c-4d4f-b007-ea507b60b88e-kube-api-access-bc687\") pod \"ipp-route-a4edd7-kserve-586cb699f9-f8prv\" (UID: \"b7c7785f-649c-4d4f-b007-ea507b60b88e\") " pod="ai-tenant-e2e-ipp-a-a4edd7/ipp-route-a4edd7-kserve-586cb699f9-f8prv"
Sep 19 01:41:54.056615 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:41:54.056565    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7c7785f-649c-4d4f-b007-ea507b60b88e-tmp-dir\") pod \"ipp-route-a4edd7-kserve-586cb699f9-f8prv\" (UID: \"b7c7785f-649c-4d4f-b007-ea507b60b88e\") " pod="ai-tenant-e2e-ipp-a-a4edd7/ipp-route-a4edd7-kserve-586cb699f9-f8prv"
Sep 19 01:41:54.056615 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:41:54.056584    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7c7785f-649c-4d4f-b007-ea507b60b88e-model-cache\") pod \"ipp-route-a4edd7-kserve-586cb699f9-f8prv\" (UID: \"b7c7785f-649c-4d4f-b007-ea507b60b88e\") " pod="ai-tenant-e2e-ipp-a-a4edd7/ipp-route-a4edd7-kserve-586cb699f9-f8prv"
Sep 19 01:41:54.156936 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:41:54.156878    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7c7785f-649c-4d4f-b007-ea507b60b88e-home\") pod \"ipp-route-a4edd7-kserve-586cb699f9-f8prv\" (UID: \"b7c7785f-649c-4d4f-b007-ea507b60b88e\") " pod="ai-tenant-e2e-ipp-a-a4edd7/ipp-route-a4edd7-kserve-586cb699f9-f8prv"
Sep 19 01:41:54.156936 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:41:54.156929    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7c7785f-649c-4d4f-b007-ea507b60b88e-tls-certs\") pod \"ipp-route-a4edd7-kserve-586cb699f9-f8prv\" (UID: \"b7c7785f-649c-4d4f-b007-ea507b60b88e\") " pod="ai-tenant-e2e-ipp-a-a4edd7/ipp-route-a4edd7-kserve-586cb699f9-f8prv"
Sep 19 01:41:54.157073 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:41:54.156954    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7c7785f-649c-4d4f-b007-ea507b60b88e-dshm\") pod \"ipp-route-a4edd7-kserve-586cb699f9-f8prv\" (UID: \"b7c7785f-649c-4d4f-b007-ea507b60b88e\") " pod="ai-tenant-e2e-ipp-a-a4edd7/ipp-route-a4edd7-kserve-586cb699f9-f8prv"
Sep 19 01:41:54.157073 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:41:54.156989    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bc687\" (UniqueName: \"kubernetes.io/projected/b7c7785f-649c-4d4f-b007-ea507b60b88e-kube-api-access-bc687\") pod \"ipp-route-a4edd7-kserve-586cb699f9-f8prv\" (UID: \"b7c7785f-649c-4d4f-b007-ea507b60b88e\") " pod="ai-tenant-e2e-ipp-a-a4edd7/ipp-route-a4edd7-kserve-586cb699f9-f8prv"
Sep 19 01:41:54.157073 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:41:54.157011    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7c7785f-649c-4d4f-b007-ea507b60b88e-tmp-dir\") pod \"ipp-route-a4edd7-kserve-586cb699f9-f8prv\" (UID: \"b7c7785f-649c-4d4f-b007-ea507b60b88e\") " pod="ai-tenant-e2e-ipp-a-a4edd7/ipp-route-a4edd7-kserve-586cb699f9-f8prv"
Sep 19 01:41:54.157073 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:41:54.157035    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7c7785f-649c-4d4f-b007-ea507b60b88e-model-cache\") pod \"ipp-route-a4edd7-kserve-586cb699f9-f8prv\" (UID: \"b7c7785f-649c-4d4f-b007-ea507b60b88e\") " pod="ai-tenant-e2e-ipp-a-a4edd7/ipp-route-a4edd7-kserve-586cb699f9-f8prv"
Sep 19 01:41:54.157294 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:41:54.157261    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7c7785f-649c-4d4f-b007-ea507b60b88e-home\") pod \"ipp-route-a4edd7-kserve-586cb699f9-f8prv\" (UID: \"b7c7785f-649c-4d4f-b007-ea507b60b88e\") " pod="ai-tenant-e2e-ipp-a-a4edd7/ipp-route-a4edd7-kserve-586cb699f9-f8prv"
Sep 19 01:41:54.157359 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:41:54.157302    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7c7785f-649c-4d4f-b007-ea507b60b88e-model-cache\") pod \"ipp-route-a4edd7-kserve-586cb699f9-f8prv\" (UID: \"b7c7785f-649c-4d4f-b007-ea507b60b88e\") " pod="ai-tenant-e2e-ipp-a-a4edd7/ipp-route-a4edd7-kserve-586cb699f9-f8prv"
Sep 19 01:41:54.157404 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:41:54.157378    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7c7785f-649c-4d4f-b007-ea507b60b88e-tmp-dir\") pod \"ipp-route-a4edd7-kserve-586cb699f9-f8prv\" (UID: \"b7c7785f-649c-4d4f-b007-ea507b60b88e\") " pod="ai-tenant-e2e-ipp-a-a4edd7/ipp-route-a4edd7-kserve-586cb699f9-f8prv"
Sep 19 01:41:54.159448 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:41:54.159428    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7c7785f-649c-4d4f-b007-ea507b60b88e-dshm\") pod \"ipp-route-a4edd7-kserve-586cb699f9-f8prv\" (UID: \"b7c7785f-649c-4d4f-b007-ea507b60b88e\") " pod="ai-tenant-e2e-ipp-a-a4edd7/ipp-route-a4edd7-kserve-586cb699f9-f8prv"
Sep 19 01:41:54.159589 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:41:54.159574    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7c7785f-649c-4d4f-b007-ea507b60b88e-tls-certs\") pod \"ipp-route-a4edd7-kserve-586cb699f9-f8prv\" (UID: \"b7c7785f-649c-4d4f-b007-ea507b60b88e\") " pod="ai-tenant-e2e-ipp-a-a4edd7/ipp-route-a4edd7-kserve-586cb699f9-f8prv"
Sep 19 01:41:54.165634 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:41:54.165612    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bc687\" (UniqueName: \"kubernetes.io/projected/b7c7785f-649c-4d4f-b007-ea507b60b88e-kube-api-access-bc687\") pod \"ipp-route-a4edd7-kserve-586cb699f9-f8prv\" (UID: \"b7c7785f-649c-4d4f-b007-ea507b60b88e\") " pod="ai-tenant-e2e-ipp-a-a4edd7/ipp-route-a4edd7-kserve-586cb699f9-f8prv"
Sep 19 01:41:54.200633 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:41:54.200614    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-a4edd7/ipp-route-a4edd7-kserve-586cb699f9-f8prv"
Sep 19 01:41:54.481306 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:41:54.481284    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-a4edd7/ipp-route-a4edd7-kserve-586cb699f9-f8prv"]
Sep 19 01:41:54.483744 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:41:54.483714    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb7c7785f_649c_4d4f_b007_ea507b60b88e.slice/crio-6827fbf2fef579aeb925e4c9867d556b77ec397eb3300b6642863434bcaf7629 WatchSource:0}: Error finding container 6827fbf2fef579aeb925e4c9867d556b77ec397eb3300b6642863434bcaf7629: Status 404 returned error can't find the container with id 6827fbf2fef579aeb925e4c9867d556b77ec397eb3300b6642863434bcaf7629
Sep 19 01:41:55.371326 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:41:55.371293    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-a4edd7/ipp-route-a4edd7-kserve-586cb699f9-f8prv" event={"ID":"b7c7785f-649c-4d4f-b007-ea507b60b88e","Type":"ContainerStarted","Data":"4d9892a1b5306c4ceb81cb4e273aa6b3e148addc87101abf1e8f7e19cc070424"}
Sep 19 01:41:55.371701 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:41:55.371338    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-a4edd7/ipp-route-a4edd7-kserve-586cb699f9-f8prv" event={"ID":"b7c7785f-649c-4d4f-b007-ea507b60b88e","Type":"ContainerStarted","Data":"6827fbf2fef579aeb925e4c9867d556b77ec397eb3300b6642863434bcaf7629"}
Sep 19 01:41:55.389878 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:41:55.389827    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-a4edd7/ipp-route-a4edd7-kserve-586cb699f9-f8prv" podStartSLOduration=2.389813778 podStartE2EDuration="2.389813778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 01:41:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 01:41:55.387813481 +0000 UTC m=+1766.275482577" watchObservedRunningTime="2026-09-19 01:41:55.389813778 +0000 UTC m=+1766.277482855"
Sep 19 01:41:57.890660 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:41:57.890627    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c9nsz"]
Sep 19 01:42:04.201305 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:42:04.201272    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-a4edd7/ipp-route-a4edd7-kserve-586cb699f9-f8prv"
Sep 19 01:42:04.201769 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:42:04.201329    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-a4edd7/ipp-route-a4edd7-kserve-586cb699f9-f8prv"
Sep 19 01:42:04.213565 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:42:04.213539    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-a4edd7/ipp-route-a4edd7-kserve-586cb699f9-f8prv"
Sep 19 01:42:04.421488 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:42:04.421455    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-a4edd7/ipp-route-a4edd7-kserve-586cb699f9-f8prv"
Sep 19 01:42:13.394302 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:42:13.394217    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c9nsz"]
Sep 19 01:42:28.345882 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:42:28.345832    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-df8e58-5d48d56fdb-j7gfp"]
Sep 19 01:42:28.349134 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:42:28.349118    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-df8e58-5d48d56fdb-j7gfp"
Sep 19 01:42:28.351337 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:42:28.351316    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-df8e58\""
Sep 19 01:42:28.351458 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:42:28.351435    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-df8e58-dockercfg-7mld8\""
Sep 19 01:42:28.360339 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:42:28.360318    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-df8e58-5d48d56fdb-j7gfp"]
Sep 19 01:42:28.416309 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:42:28.416287    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fe6aa15-9e9e-4704-b15f-36fa6afff608-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-df8e58-5d48d56fdb-j7gfp\" (UID: \"6fe6aa15-9e9e-4704-b15f-36fa6afff608\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-df8e58-5d48d56fdb-j7gfp"
Sep 19 01:42:28.416428 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:42:28.416384    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sw4c\" (UniqueName: \"kubernetes.io/projected/6fe6aa15-9e9e-4704-b15f-36fa6afff608-kube-api-access-4sw4c\") pod \"payload-pre-processing-e2e-ipp-cleanup-df8e58-5d48d56fdb-j7gfp\" (UID: \"6fe6aa15-9e9e-4704-b15f-36fa6afff608\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-df8e58-5d48d56fdb-j7gfp"
Sep 19 01:42:28.517345 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:42:28.517314    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sw4c\" (UniqueName: \"kubernetes.io/projected/6fe6aa15-9e9e-4704-b15f-36fa6afff608-kube-api-access-4sw4c\") pod \"payload-pre-processing-e2e-ipp-cleanup-df8e58-5d48d56fdb-j7gfp\" (UID: \"6fe6aa15-9e9e-4704-b15f-36fa6afff608\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-df8e58-5d48d56fdb-j7gfp"
Sep 19 01:42:28.517491 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:42:28.517369    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fe6aa15-9e9e-4704-b15f-36fa6afff608-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-df8e58-5d48d56fdb-j7gfp\" (UID: \"6fe6aa15-9e9e-4704-b15f-36fa6afff608\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-df8e58-5d48d56fdb-j7gfp"
Sep 19 01:42:28.518014 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:42:28.517996    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fe6aa15-9e9e-4704-b15f-36fa6afff608-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-df8e58-5d48d56fdb-j7gfp\" (UID: \"6fe6aa15-9e9e-4704-b15f-36fa6afff608\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-df8e58-5d48d56fdb-j7gfp"
Sep 19 01:42:28.526268 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:42:28.526252    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sw4c\" (UniqueName: \"kubernetes.io/projected/6fe6aa15-9e9e-4704-b15f-36fa6afff608-kube-api-access-4sw4c\") pod \"payload-pre-processing-e2e-ipp-cleanup-df8e58-5d48d56fdb-j7gfp\" (UID: \"6fe6aa15-9e9e-4704-b15f-36fa6afff608\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-df8e58-5d48d56fdb-j7gfp"
Sep 19 01:42:28.659543 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:42:28.659479    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-df8e58-5d48d56fdb-j7gfp"
Sep 19 01:42:28.928695 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:42:28.928670    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-df8e58-5d48d56fdb-j7gfp"]
Sep 19 01:42:28.930530 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:42:28.930503    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fe6aa15_9e9e_4704_b15f_36fa6afff608.slice/crio-1028388780ec2a560b5ce7c791fc70672fcda48245292fdf7233804465057417 WatchSource:0}: Error finding container 1028388780ec2a560b5ce7c791fc70672fcda48245292fdf7233804465057417: Status 404 returned error can't find the container with id 1028388780ec2a560b5ce7c791fc70672fcda48245292fdf7233804465057417
Sep 19 01:42:29.523457 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:42:29.523411    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-df8e58-5d48d56fdb-j7gfp" event={"ID":"6fe6aa15-9e9e-4704-b15f-36fa6afff608","Type":"ContainerStarted","Data":"a231401b4030327e09b8766f2445a1c80f01974ae440716da8ba6113bdcc75f1"}
Sep 19 01:42:29.523874 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:42:29.523600    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-df8e58-5d48d56fdb-j7gfp"
Sep 19 01:42:29.523874 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:42:29.523644    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-df8e58-5d48d56fdb-j7gfp" event={"ID":"6fe6aa15-9e9e-4704-b15f-36fa6afff608","Type":"ContainerStarted","Data":"1028388780ec2a560b5ce7c791fc70672fcda48245292fdf7233804465057417"}
Sep 19 01:42:29.543010 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:42:29.542961    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-df8e58-5d48d56fdb-j7gfp" podStartSLOduration=1.542947817 podStartE2EDuration="1.542947817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 01:42:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 01:42:29.5418508 +0000 UTC m=+1800.429519891" watchObservedRunningTime="2026-09-19 01:42:29.542947817 +0000 UTC m=+1800.430616893"
Sep 19 01:42:31.296942 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:42:31.296911    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x785d_f759a12f-5b6f-4924-96f2-dd5e00fc1348/ovn-acl-logging/0.log"
Sep 19 01:42:31.299772 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:42:31.299752    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x785d_f759a12f-5b6f-4924-96f2-dd5e00fc1348/ovn-acl-logging/0.log"
Sep 19 01:42:31.614474 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:42:31.614449    2581 scope.go:117] "RemoveContainer" containerID="1b9cd47f5bf044527ffcd790a9d01b644c8b8287d3245ea2d15985f95d9c4a8d"
Sep 19 01:42:33.415064 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:42:33.415030    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c9nsz"]
Sep 19 01:42:40.529885 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:42:40.529831    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-df8e58-5d48d56fdb-j7gfp"
Sep 19 01:42:55.188025 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:42:55.187986    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-df8e58-nxmvb"]
Sep 19 01:42:55.191124 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:42:55.191108    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-df8e58-nxmvb"
Sep 19 01:42:55.193282 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:42:55.193259    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 19 01:42:55.193422 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:42:55.193259    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s7pvt\""
Sep 19 01:42:55.199920 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:42:55.199895    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-df8e58-nxmvb"]
Sep 19 01:42:55.337169 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:42:55.337142    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/141bf411-5ec2-4cc9-96be-cf5a68501a8e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-df8e58-nxmvb\" (UID: \"141bf411-5ec2-4cc9-96be-cf5a68501a8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-df8e58-nxmvb"
Sep 19 01:42:55.437808 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:42:55.437783    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/141bf411-5ec2-4cc9-96be-cf5a68501a8e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-df8e58-nxmvb\" (UID: \"141bf411-5ec2-4cc9-96be-cf5a68501a8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-df8e58-nxmvb"
Sep 19 01:42:55.438401 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:42:55.438357    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/141bf411-5ec2-4cc9-96be-cf5a68501a8e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-df8e58-nxmvb\" (UID: \"141bf411-5ec2-4cc9-96be-cf5a68501a8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-df8e58-nxmvb"
Sep 19 01:42:55.502510 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:42:55.502488    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-df8e58-nxmvb"
Sep 19 01:42:55.778731 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:42:55.778701    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-df8e58-nxmvb"]
Sep 19 01:42:55.779178 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:42:55.779152    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod141bf411_5ec2_4cc9_96be_cf5a68501a8e.slice/crio-ba31b48354e14ca44ab83325d64c5fa6650a170fa7dc636d69ddd80417a5f1ea WatchSource:0}: Error finding container ba31b48354e14ca44ab83325d64c5fa6650a170fa7dc636d69ddd80417a5f1ea: Status 404 returned error can't find the container with id ba31b48354e14ca44ab83325d64c5fa6650a170fa7dc636d69ddd80417a5f1ea
Sep 19 01:42:56.645893 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:42:56.645805    2581 generic.go:356] "Generic (PLEG): container finished" podID="141bf411-5ec2-4cc9-96be-cf5a68501a8e" containerID="d27602c66d0097af2ddff59250e9e2b39853928226c537d2a86133abde2f3742" exitCode=0
Sep 19 01:42:56.646211 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:42:56.645905    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-df8e58-nxmvb" event={"ID":"141bf411-5ec2-4cc9-96be-cf5a68501a8e","Type":"ContainerDied","Data":"d27602c66d0097af2ddff59250e9e2b39853928226c537d2a86133abde2f3742"}
Sep 19 01:42:56.646211 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:42:56.645978    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-df8e58-nxmvb" event={"ID":"141bf411-5ec2-4cc9-96be-cf5a68501a8e","Type":"ContainerStarted","Data":"ba31b48354e14ca44ab83325d64c5fa6650a170fa7dc636d69ddd80417a5f1ea"}
Sep 19 01:42:57.777229 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:42:57.777203    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-df8e58-nxmvb"
Sep 19 01:42:57.857953 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:42:57.857919    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/141bf411-5ec2-4cc9-96be-cf5a68501a8e-maas-api-service-ca\") pod \"141bf411-5ec2-4cc9-96be-cf5a68501a8e\" (UID: \"141bf411-5ec2-4cc9-96be-cf5a68501a8e\") "
Sep 19 01:42:57.858327 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:42:57.858297    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/141bf411-5ec2-4cc9-96be-cf5a68501a8e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "141bf411-5ec2-4cc9-96be-cf5a68501a8e" (UID: "141bf411-5ec2-4cc9-96be-cf5a68501a8e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 01:42:57.959511 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:42:57.959450    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/141bf411-5ec2-4cc9-96be-cf5a68501a8e-maas-api-service-ca\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:42:58.655732 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:42:58.655703    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-df8e58-nxmvb" event={"ID":"141bf411-5ec2-4cc9-96be-cf5a68501a8e","Type":"ContainerDied","Data":"ba31b48354e14ca44ab83325d64c5fa6650a170fa7dc636d69ddd80417a5f1ea"}
Sep 19 01:42:58.655732 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:42:58.655729    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-df8e58-nxmvb"
Sep 19 01:42:58.656018 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:42:58.655750    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba31b48354e14ca44ab83325d64c5fa6650a170fa7dc636d69ddd80417a5f1ea"
Sep 19 01:43:06.103432 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:43:06.103396    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-df8e58-5d48d56fdb-j7gfp"]
Sep 19 01:43:06.103896 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:43:06.103608    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-df8e58-5d48d56fdb-j7gfp" podUID="6fe6aa15-9e9e-4704-b15f-36fa6afff608" containerName="payload-pre-processing" containerID="cri-o://a231401b4030327e09b8766f2445a1c80f01974ae440716da8ba6113bdcc75f1" gracePeriod=30
Sep 19 01:43:06.278014 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:43:06.277959    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-df8e58-nxmvb"]
Sep 19 01:43:06.279846 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:43:06.279818    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-df8e58-nxmvb"]
Sep 19 01:43:06.358499 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:43:06.358442    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-df8e58-5d48d56fdb-j7gfp"
Sep 19 01:43:06.426229 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:43:06.426193    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fe6aa15-9e9e-4704-b15f-36fa6afff608-plugins-config-volume\") pod \"6fe6aa15-9e9e-4704-b15f-36fa6afff608\" (UID: \"6fe6aa15-9e9e-4704-b15f-36fa6afff608\") "
Sep 19 01:43:06.426413 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:43:06.426266    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4sw4c\" (UniqueName: \"kubernetes.io/projected/6fe6aa15-9e9e-4704-b15f-36fa6afff608-kube-api-access-4sw4c\") pod \"6fe6aa15-9e9e-4704-b15f-36fa6afff608\" (UID: \"6fe6aa15-9e9e-4704-b15f-36fa6afff608\") "
Sep 19 01:43:06.426727 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:43:06.426684    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fe6aa15-9e9e-4704-b15f-36fa6afff608-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6fe6aa15-9e9e-4704-b15f-36fa6afff608" (UID: "6fe6aa15-9e9e-4704-b15f-36fa6afff608"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 01:43:06.428605 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:43:06.428574    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fe6aa15-9e9e-4704-b15f-36fa6afff608-kube-api-access-4sw4c" (OuterVolumeSpecName: "kube-api-access-4sw4c") pod "6fe6aa15-9e9e-4704-b15f-36fa6afff608" (UID: "6fe6aa15-9e9e-4704-b15f-36fa6afff608"). InnerVolumeSpecName "kube-api-access-4sw4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 01:43:06.527179 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:43:06.527140    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fe6aa15-9e9e-4704-b15f-36fa6afff608-plugins-config-volume\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:43:06.527179 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:43:06.527173    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4sw4c\" (UniqueName: \"kubernetes.io/projected/6fe6aa15-9e9e-4704-b15f-36fa6afff608-kube-api-access-4sw4c\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:43:06.694089 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:43:06.694011    2581 generic.go:356] "Generic (PLEG): container finished" podID="6fe6aa15-9e9e-4704-b15f-36fa6afff608" containerID="a231401b4030327e09b8766f2445a1c80f01974ae440716da8ba6113bdcc75f1" exitCode=0
Sep 19 01:43:06.694089 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:43:06.694047    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-df8e58-5d48d56fdb-j7gfp"
Sep 19 01:43:06.694260 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:43:06.694089    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-df8e58-5d48d56fdb-j7gfp" event={"ID":"6fe6aa15-9e9e-4704-b15f-36fa6afff608","Type":"ContainerDied","Data":"a231401b4030327e09b8766f2445a1c80f01974ae440716da8ba6113bdcc75f1"}
Sep 19 01:43:06.694260 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:43:06.694145    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-df8e58-5d48d56fdb-j7gfp" event={"ID":"6fe6aa15-9e9e-4704-b15f-36fa6afff608","Type":"ContainerDied","Data":"1028388780ec2a560b5ce7c791fc70672fcda48245292fdf7233804465057417"}
Sep 19 01:43:06.694260 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:43:06.694169    2581 scope.go:117] "RemoveContainer" containerID="a231401b4030327e09b8766f2445a1c80f01974ae440716da8ba6113bdcc75f1"
Sep 19 01:43:06.704849 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:43:06.704820    2581 scope.go:117] "RemoveContainer" containerID="a231401b4030327e09b8766f2445a1c80f01974ae440716da8ba6113bdcc75f1"
Sep 19 01:43:06.707153 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:43:06.707125    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a231401b4030327e09b8766f2445a1c80f01974ae440716da8ba6113bdcc75f1\": container with ID starting with a231401b4030327e09b8766f2445a1c80f01974ae440716da8ba6113bdcc75f1 not found: ID does not exist" containerID="a231401b4030327e09b8766f2445a1c80f01974ae440716da8ba6113bdcc75f1"
Sep 19 01:43:06.707259 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:43:06.707157    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a231401b4030327e09b8766f2445a1c80f01974ae440716da8ba6113bdcc75f1"} err="failed to get container status \"a231401b4030327e09b8766f2445a1c80f01974ae440716da8ba6113bdcc75f1\": rpc error: code = NotFound desc = could not find container \"a231401b4030327e09b8766f2445a1c80f01974ae440716da8ba6113bdcc75f1\": container with ID starting with a231401b4030327e09b8766f2445a1c80f01974ae440716da8ba6113bdcc75f1 not found: ID does not exist"
Sep 19 01:43:06.719662 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:43:06.719634    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-df8e58-5d48d56fdb-j7gfp"]
Sep 19 01:43:06.724750 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:43:06.724730    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-df8e58-5d48d56fdb-j7gfp"]
Sep 19 01:43:07.272530 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:43:07.272496    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="141bf411-5ec2-4cc9-96be-cf5a68501a8e" path="/var/lib/kubelet/pods/141bf411-5ec2-4cc9-96be-cf5a68501a8e/volumes"
Sep 19 01:43:07.272918 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:43:07.272828    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6fe6aa15-9e9e-4704-b15f-36fa6afff608" path="/var/lib/kubelet/pods/6fe6aa15-9e9e-4704-b15f-36fa6afff608/volumes"
Sep 19 01:43:10.684625 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:43:10.684588    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c9nsz"]
Sep 19 01:43:17.332612 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:43:17.332575    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a4edd7-fdk7v"]
Sep 19 01:43:17.333087 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:43:17.332940    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fe6aa15-9e9e-4704-b15f-36fa6afff608" containerName="payload-pre-processing"
Sep 19 01:43:17.333087 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:43:17.332954    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fe6aa15-9e9e-4704-b15f-36fa6afff608" containerName="payload-pre-processing"
Sep 19 01:43:17.333087 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:43:17.332975    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="141bf411-5ec2-4cc9-96be-cf5a68501a8e" containerName="revoke-keys"
Sep 19 01:43:17.333087 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:43:17.332981    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="141bf411-5ec2-4cc9-96be-cf5a68501a8e" containerName="revoke-keys"
Sep 19 01:43:17.333087 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:43:17.333035    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fe6aa15-9e9e-4704-b15f-36fa6afff608" containerName="payload-pre-processing"
Sep 19 01:43:17.333087 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:43:17.333047    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="141bf411-5ec2-4cc9-96be-cf5a68501a8e" containerName="revoke-keys"
Sep 19 01:43:17.337238 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:43:17.337221    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a4edd7-fdk7v"
Sep 19 01:43:17.339590 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:43:17.339567    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s7pvt\""
Sep 19 01:43:17.339590 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:43:17.339573    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 19 01:43:17.343679 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:43:17.343655    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a4edd7-fdk7v"]
Sep 19 01:43:17.519182 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:43:17.519148    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a060f55a-e968-4047-b5cf-db37d8c93504-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a4edd7-fdk7v\" (UID: \"a060f55a-e968-4047-b5cf-db37d8c93504\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a4edd7-fdk7v"
Sep 19 01:43:17.619614 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:43:17.619591    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a060f55a-e968-4047-b5cf-db37d8c93504-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a4edd7-fdk7v\" (UID: \"a060f55a-e968-4047-b5cf-db37d8c93504\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a4edd7-fdk7v"
Sep 19 01:43:17.620233 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:43:17.620213    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a060f55a-e968-4047-b5cf-db37d8c93504-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a4edd7-fdk7v\" (UID: \"a060f55a-e968-4047-b5cf-db37d8c93504\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a4edd7-fdk7v"
Sep 19 01:43:17.648706 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:43:17.648690    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a4edd7-fdk7v"
Sep 19 01:43:17.915768 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:43:17.915747    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a4edd7-fdk7v"]
Sep 19 01:43:17.918202 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:43:17.918166    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda060f55a_e968_4047_b5cf_db37d8c93504.slice/crio-4f716e5c310ca73a9c31dcb0f3a5eb5209fd678d134da35f8d1ee56aaa8a541f WatchSource:0}: Error finding container 4f716e5c310ca73a9c31dcb0f3a5eb5209fd678d134da35f8d1ee56aaa8a541f: Status 404 returned error can't find the container with id 4f716e5c310ca73a9c31dcb0f3a5eb5209fd678d134da35f8d1ee56aaa8a541f
Sep 19 01:43:17.990532 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:43:17.990456    2581 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda060f55a_e968_4047_b5cf_db37d8c93504.slice/crio-4f29adf87c4a346e1b3659adf789ee005b1306cc366d07c8ef2f58091c4752a9.scope\": RecentStats: unable to find data in memory cache]"
Sep 19 01:43:17.993472 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:43:17.993442    2581 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda060f55a_e968_4047_b5cf_db37d8c93504.slice/crio-4f29adf87c4a346e1b3659adf789ee005b1306cc366d07c8ef2f58091c4752a9.scope\": RecentStats: unable to find data in memory cache]"
Sep 19 01:43:18.751115 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:43:18.751080    2581 generic.go:356] "Generic (PLEG): container finished" podID="a060f55a-e968-4047-b5cf-db37d8c93504" containerID="4f29adf87c4a346e1b3659adf789ee005b1306cc366d07c8ef2f58091c4752a9" exitCode=0
Sep 19 01:43:18.751486 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:43:18.751175    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a4edd7-fdk7v" event={"ID":"a060f55a-e968-4047-b5cf-db37d8c93504","Type":"ContainerDied","Data":"4f29adf87c4a346e1b3659adf789ee005b1306cc366d07c8ef2f58091c4752a9"}
Sep 19 01:43:18.751486 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:43:18.751263    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a4edd7-fdk7v" event={"ID":"a060f55a-e968-4047-b5cf-db37d8c93504","Type":"ContainerStarted","Data":"4f716e5c310ca73a9c31dcb0f3a5eb5209fd678d134da35f8d1ee56aaa8a541f"}
Sep 19 01:43:19.883660 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:43:19.883630    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a4edd7-fdk7v"
Sep 19 01:43:19.939595 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:43:19.939566    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a060f55a-e968-4047-b5cf-db37d8c93504-maas-api-service-ca\") pod \"a060f55a-e968-4047-b5cf-db37d8c93504\" (UID: \"a060f55a-e968-4047-b5cf-db37d8c93504\") "
Sep 19 01:43:19.939947 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:43:19.939925    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a060f55a-e968-4047-b5cf-db37d8c93504-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a060f55a-e968-4047-b5cf-db37d8c93504" (UID: "a060f55a-e968-4047-b5cf-db37d8c93504"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 01:43:20.040916 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:43:20.040834    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a060f55a-e968-4047-b5cf-db37d8c93504-maas-api-service-ca\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:43:20.762088 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:43:20.762062    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a4edd7-fdk7v"
Sep 19 01:43:20.762088 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:43:20.762080    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a4edd7-fdk7v" event={"ID":"a060f55a-e968-4047-b5cf-db37d8c93504","Type":"ContainerDied","Data":"4f716e5c310ca73a9c31dcb0f3a5eb5209fd678d134da35f8d1ee56aaa8a541f"}
Sep 19 01:43:20.762260 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:43:20.762127    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f716e5c310ca73a9c31dcb0f3a5eb5209fd678d134da35f8d1ee56aaa8a541f"
Sep 19 01:43:30.889210 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:43:30.889177    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c9nsz"]
Sep 19 01:43:33.420226 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:43:33.420186    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a4edd7-fdk7v"]
Sep 19 01:43:33.423333 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:43:33.423302    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a4edd7-fdk7v"]
Sep 19 01:43:35.277107 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:43:35.277071    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a060f55a-e968-4047-b5cf-db37d8c93504" path="/var/lib/kubelet/pods/a060f55a-e968-4047-b5cf-db37d8c93504/volumes"
Sep 19 01:43:38.799591 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:43:38.799559    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-a4edd7/ipp-route-a4edd7-kserve-586cb699f9-f8prv"]
Sep 19 01:43:44.293014 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:43:44.292935    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c9nsz"]
Sep 19 01:43:53.818839 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:43:53.818787    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-a4edd7/ipp-route-a4edd7-kserve-586cb699f9-f8prv" podUID="b7c7785f-649c-4d4f-b007-ea507b60b88e" containerName="main" containerID="cri-o://4d9892a1b5306c4ceb81cb4e273aa6b3e148addc87101abf1e8f7e19cc070424" gracePeriod=45
Sep 19 01:43:54.066561 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:43:54.066534    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-a4edd7/ipp-route-a4edd7-kserve-586cb699f9-f8prv"
Sep 19 01:43:54.202491 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:43:54.202456    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7c7785f-649c-4d4f-b007-ea507b60b88e-dshm\") pod \"b7c7785f-649c-4d4f-b007-ea507b60b88e\" (UID: \"b7c7785f-649c-4d4f-b007-ea507b60b88e\") "
Sep 19 01:43:54.202669 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:43:54.202510    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7c7785f-649c-4d4f-b007-ea507b60b88e-home\") pod \"b7c7785f-649c-4d4f-b007-ea507b60b88e\" (UID: \"b7c7785f-649c-4d4f-b007-ea507b60b88e\") "
Sep 19 01:43:54.202669 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:43:54.202536    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7c7785f-649c-4d4f-b007-ea507b60b88e-tls-certs\") pod \"b7c7785f-649c-4d4f-b007-ea507b60b88e\" (UID: \"b7c7785f-649c-4d4f-b007-ea507b60b88e\") "
Sep 19 01:43:54.202669 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:43:54.202611    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7c7785f-649c-4d4f-b007-ea507b60b88e-model-cache\") pod \"b7c7785f-649c-4d4f-b007-ea507b60b88e\" (UID: \"b7c7785f-649c-4d4f-b007-ea507b60b88e\") "
Sep 19 01:43:54.202669 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:43:54.202642    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7c7785f-649c-4d4f-b007-ea507b60b88e-tmp-dir\") pod \"b7c7785f-649c-4d4f-b007-ea507b60b88e\" (UID: \"b7c7785f-649c-4d4f-b007-ea507b60b88e\") "
Sep 19 01:43:54.202911 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:43:54.202672    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bc687\" (UniqueName: \"kubernetes.io/projected/b7c7785f-649c-4d4f-b007-ea507b60b88e-kube-api-access-bc687\") pod \"b7c7785f-649c-4d4f-b007-ea507b60b88e\" (UID: \"b7c7785f-649c-4d4f-b007-ea507b60b88e\") "
Sep 19 01:43:54.202911 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:43:54.202821    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7c7785f-649c-4d4f-b007-ea507b60b88e-home" (OuterVolumeSpecName: "home") pod "b7c7785f-649c-4d4f-b007-ea507b60b88e" (UID: "b7c7785f-649c-4d4f-b007-ea507b60b88e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 01:43:54.202911 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:43:54.202841    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7c7785f-649c-4d4f-b007-ea507b60b88e-model-cache" (OuterVolumeSpecName: "model-cache") pod "b7c7785f-649c-4d4f-b007-ea507b60b88e" (UID: "b7c7785f-649c-4d4f-b007-ea507b60b88e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 01:43:54.203087 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:43:54.202974    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7c7785f-649c-4d4f-b007-ea507b60b88e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b7c7785f-649c-4d4f-b007-ea507b60b88e" (UID: "b7c7785f-649c-4d4f-b007-ea507b60b88e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 01:43:54.203087 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:43:54.202989    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7c7785f-649c-4d4f-b007-ea507b60b88e-home\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:43:54.203087 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:43:54.203033    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7c7785f-649c-4d4f-b007-ea507b60b88e-model-cache\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:43:54.204938 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:43:54.204909    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7c7785f-649c-4d4f-b007-ea507b60b88e-dshm" (OuterVolumeSpecName: "dshm") pod "b7c7785f-649c-4d4f-b007-ea507b60b88e" (UID: "b7c7785f-649c-4d4f-b007-ea507b60b88e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 01:43:54.205066 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:43:54.205029    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7c7785f-649c-4d4f-b007-ea507b60b88e-kube-api-access-bc687" (OuterVolumeSpecName: "kube-api-access-bc687") pod "b7c7785f-649c-4d4f-b007-ea507b60b88e" (UID: "b7c7785f-649c-4d4f-b007-ea507b60b88e"). InnerVolumeSpecName "kube-api-access-bc687". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 01:43:54.205218 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:43:54.205194    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7c7785f-649c-4d4f-b007-ea507b60b88e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b7c7785f-649c-4d4f-b007-ea507b60b88e" (UID: "b7c7785f-649c-4d4f-b007-ea507b60b88e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 01:43:54.303494 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:43:54.303471    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bc687\" (UniqueName: \"kubernetes.io/projected/b7c7785f-649c-4d4f-b007-ea507b60b88e-kube-api-access-bc687\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:43:54.303494 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:43:54.303491    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7c7785f-649c-4d4f-b007-ea507b60b88e-dshm\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:43:54.303629 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:43:54.303502    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7c7785f-649c-4d4f-b007-ea507b60b88e-tls-certs\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:43:54.303629 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:43:54.303511    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7c7785f-649c-4d4f-b007-ea507b60b88e-tmp-dir\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:43:54.920623 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:43:54.920590    2581 generic.go:356] "Generic (PLEG): container finished" podID="b7c7785f-649c-4d4f-b007-ea507b60b88e" containerID="4d9892a1b5306c4ceb81cb4e273aa6b3e148addc87101abf1e8f7e19cc070424" exitCode=0
Sep 19 01:43:54.921007 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:43:54.920634    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-a4edd7/ipp-route-a4edd7-kserve-586cb699f9-f8prv"
Sep 19 01:43:54.921007 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:43:54.920642    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-a4edd7/ipp-route-a4edd7-kserve-586cb699f9-f8prv" event={"ID":"b7c7785f-649c-4d4f-b007-ea507b60b88e","Type":"ContainerDied","Data":"4d9892a1b5306c4ceb81cb4e273aa6b3e148addc87101abf1e8f7e19cc070424"}
Sep 19 01:43:54.921007 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:43:54.920680    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-a4edd7/ipp-route-a4edd7-kserve-586cb699f9-f8prv" event={"ID":"b7c7785f-649c-4d4f-b007-ea507b60b88e","Type":"ContainerDied","Data":"6827fbf2fef579aeb925e4c9867d556b77ec397eb3300b6642863434bcaf7629"}
Sep 19 01:43:54.921007 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:43:54.920702    2581 scope.go:117] "RemoveContainer" containerID="4d9892a1b5306c4ceb81cb4e273aa6b3e148addc87101abf1e8f7e19cc070424"
Sep 19 01:43:54.929734 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:43:54.929717    2581 scope.go:117] "RemoveContainer" containerID="4d9892a1b5306c4ceb81cb4e273aa6b3e148addc87101abf1e8f7e19cc070424"
Sep 19 01:43:54.930030 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:43:54.930012    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d9892a1b5306c4ceb81cb4e273aa6b3e148addc87101abf1e8f7e19cc070424\": container with ID starting with 4d9892a1b5306c4ceb81cb4e273aa6b3e148addc87101abf1e8f7e19cc070424 not found: ID does not exist" containerID="4d9892a1b5306c4ceb81cb4e273aa6b3e148addc87101abf1e8f7e19cc070424"
Sep 19 01:43:54.930083 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:43:54.930039    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d9892a1b5306c4ceb81cb4e273aa6b3e148addc87101abf1e8f7e19cc070424"} err="failed to get container status \"4d9892a1b5306c4ceb81cb4e273aa6b3e148addc87101abf1e8f7e19cc070424\": rpc error: code = NotFound desc = could not find container \"4d9892a1b5306c4ceb81cb4e273aa6b3e148addc87101abf1e8f7e19cc070424\": container with ID starting with 4d9892a1b5306c4ceb81cb4e273aa6b3e148addc87101abf1e8f7e19cc070424 not found: ID does not exist"
Sep 19 01:43:54.940808 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:43:54.940783    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-a4edd7/ipp-route-a4edd7-kserve-586cb699f9-f8prv"]
Sep 19 01:43:54.944710 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:43:54.944689    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-a4edd7/ipp-route-a4edd7-kserve-586cb699f9-f8prv"]
Sep 19 01:43:55.273021 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:43:55.272940    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7c7785f-649c-4d4f-b007-ea507b60b88e" path="/var/lib/kubelet/pods/b7c7785f-649c-4d4f-b007-ea507b60b88e/volumes"
Sep 19 01:44:28.910128 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:28.910090    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-14de36-pbrsx"]
Sep 19 01:44:28.910672 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:28.910555    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a060f55a-e968-4047-b5cf-db37d8c93504" containerName="revoke-keys"
Sep 19 01:44:28.910672 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:28.910575    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a060f55a-e968-4047-b5cf-db37d8c93504" containerName="revoke-keys"
Sep 19 01:44:28.910672 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:28.910591    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7c7785f-649c-4d4f-b007-ea507b60b88e" containerName="main"
Sep 19 01:44:28.910672 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:28.910601    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7c7785f-649c-4d4f-b007-ea507b60b88e" containerName="main"
Sep 19 01:44:28.910887 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:28.910697    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7c7785f-649c-4d4f-b007-ea507b60b88e" containerName="main"
Sep 19 01:44:28.910887 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:28.910711    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a060f55a-e968-4047-b5cf-db37d8c93504" containerName="revoke-keys"
Sep 19 01:44:28.918119 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:28.918091    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-14de36-pbrsx"
Sep 19 01:44:28.922532 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:28.922504    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s7pvt\""
Sep 19 01:44:28.923455 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:28.923411    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 19 01:44:28.924008 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:28.923977    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-14de36-pbrsx"]
Sep 19 01:44:28.980998 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:28.980962    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/932b4c8b-cf88-43a7-9c27-5e469ba9af81-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-14de36-pbrsx\" (UID: \"932b4c8b-cf88-43a7-9c27-5e469ba9af81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-14de36-pbrsx"
Sep 19 01:44:29.082058 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:29.082026    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/932b4c8b-cf88-43a7-9c27-5e469ba9af81-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-14de36-pbrsx\" (UID: \"932b4c8b-cf88-43a7-9c27-5e469ba9af81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-14de36-pbrsx"
Sep 19 01:44:29.082624 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:29.082604    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/932b4c8b-cf88-43a7-9c27-5e469ba9af81-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-14de36-pbrsx\" (UID: \"932b4c8b-cf88-43a7-9c27-5e469ba9af81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-14de36-pbrsx"
Sep 19 01:44:29.230791 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:29.230716    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-14de36-pbrsx"
Sep 19 01:44:29.501613 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:29.501590    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-14de36-pbrsx"]
Sep 19 01:44:29.503991 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:44:29.503965    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod932b4c8b_cf88_43a7_9c27_5e469ba9af81.slice/crio-e26f262bd5f4d9b231cba257ea3a67bf99ae8061f4e458bf413eb09d78ef30bd WatchSource:0}: Error finding container e26f262bd5f4d9b231cba257ea3a67bf99ae8061f4e458bf413eb09d78ef30bd: Status 404 returned error can't find the container with id e26f262bd5f4d9b231cba257ea3a67bf99ae8061f4e458bf413eb09d78ef30bd
Sep 19 01:44:30.086559 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:30.086477    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-14de36-pbrsx" event={"ID":"932b4c8b-cf88-43a7-9c27-5e469ba9af81","Type":"ContainerStarted","Data":"3b45cfbf9ac33102cf5f92c576a335182a2d05ba9d1a27686c0d778cee080bc5"}
Sep 19 01:44:30.086559 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:30.086525    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-14de36-pbrsx" event={"ID":"932b4c8b-cf88-43a7-9c27-5e469ba9af81","Type":"ContainerStarted","Data":"e26f262bd5f4d9b231cba257ea3a67bf99ae8061f4e458bf413eb09d78ef30bd"}
Sep 19 01:44:30.103122 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:30.103081    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-14de36-pbrsx" podStartSLOduration=2.103065492 podStartE2EDuration="2.103065492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 01:44:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 01:44:30.101693873 +0000 UTC m=+1920.989362962" watchObservedRunningTime="2026-09-19 01:44:30.103065492 +0000 UTC m=+1920.990734568"
Sep 19 01:44:31.092360 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:31.092319    2581 generic.go:356] "Generic (PLEG): container finished" podID="932b4c8b-cf88-43a7-9c27-5e469ba9af81" containerID="3b45cfbf9ac33102cf5f92c576a335182a2d05ba9d1a27686c0d778cee080bc5" exitCode=0
Sep 19 01:44:31.092766 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:31.092394    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-14de36-pbrsx" event={"ID":"932b4c8b-cf88-43a7-9c27-5e469ba9af81","Type":"ContainerDied","Data":"3b45cfbf9ac33102cf5f92c576a335182a2d05ba9d1a27686c0d778cee080bc5"}
Sep 19 01:44:31.645345 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:31.645315    2581 scope.go:117] "RemoveContainer" containerID="56faa8f8efb49498dea7e0b9bedf63ca7afc7677a6dace6fd8eb9ec2a067f838"
Sep 19 01:44:31.653264 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:31.653244    2581 scope.go:117] "RemoveContainer" containerID="b3f3a7eb0280e1b198dc842cfbea7bdf2fa9914fba1ee4fc1b939e2615bee5ec"
Sep 19 01:44:31.660753 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:31.660731    2581 scope.go:117] "RemoveContainer" containerID="20d2454e33a62d6d39069d49ddde919bc782b433078cd78e14e099335b7c389f"
Sep 19 01:44:32.225848 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:32.225822    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-14de36-pbrsx"
Sep 19 01:44:32.308065 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:32.308038    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/932b4c8b-cf88-43a7-9c27-5e469ba9af81-maas-api-service-ca\") pod \"932b4c8b-cf88-43a7-9c27-5e469ba9af81\" (UID: \"932b4c8b-cf88-43a7-9c27-5e469ba9af81\") "
Sep 19 01:44:32.308335 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:32.308314    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/932b4c8b-cf88-43a7-9c27-5e469ba9af81-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "932b4c8b-cf88-43a7-9c27-5e469ba9af81" (UID: "932b4c8b-cf88-43a7-9c27-5e469ba9af81"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 01:44:32.409151 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:32.409072    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/932b4c8b-cf88-43a7-9c27-5e469ba9af81-maas-api-service-ca\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:44:33.103084 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:33.103056    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-14de36-pbrsx"
Sep 19 01:44:33.103084 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:33.103072    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-14de36-pbrsx" event={"ID":"932b4c8b-cf88-43a7-9c27-5e469ba9af81","Type":"ContainerDied","Data":"e26f262bd5f4d9b231cba257ea3a67bf99ae8061f4e458bf413eb09d78ef30bd"}
Sep 19 01:44:33.103317 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:33.103114    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e26f262bd5f4d9b231cba257ea3a67bf99ae8061f4e458bf413eb09d78ef30bd"
Sep 19 01:44:39.579284 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:39.579250    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-14de36-77dfd7748b-4q4zn"]
Sep 19 01:44:39.579818 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:39.579502    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-14de36-77dfd7748b-4q4zn" podUID="929b84a7-8e7f-4526-aa91-23d6c7de5de0" containerName="payload-pre-processing" containerID="cri-o://54405da5190cced0f65cb8132b369c5cd190599766965441f4d6cadc111b4ec3" gracePeriod=30
Sep 19 01:44:39.767173 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:39.767140    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-14de36-pbrsx"]
Sep 19 01:44:39.773369 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:39.773344    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-14de36-pbrsx"]
Sep 19 01:44:39.833908 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:39.833836    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-14de36-77dfd7748b-4q4zn"
Sep 19 01:44:39.967116 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:39.967085    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/929b84a7-8e7f-4526-aa91-23d6c7de5de0-plugins-config-volume\") pod \"929b84a7-8e7f-4526-aa91-23d6c7de5de0\" (UID: \"929b84a7-8e7f-4526-aa91-23d6c7de5de0\") "
Sep 19 01:44:39.967284 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:39.967181    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-62sjs\" (UniqueName: \"kubernetes.io/projected/929b84a7-8e7f-4526-aa91-23d6c7de5de0-kube-api-access-62sjs\") pod \"929b84a7-8e7f-4526-aa91-23d6c7de5de0\" (UID: \"929b84a7-8e7f-4526-aa91-23d6c7de5de0\") "
Sep 19 01:44:39.967463 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:39.967440    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/929b84a7-8e7f-4526-aa91-23d6c7de5de0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "929b84a7-8e7f-4526-aa91-23d6c7de5de0" (UID: "929b84a7-8e7f-4526-aa91-23d6c7de5de0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 01:44:39.969451 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:39.969430    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/929b84a7-8e7f-4526-aa91-23d6c7de5de0-kube-api-access-62sjs" (OuterVolumeSpecName: "kube-api-access-62sjs") pod "929b84a7-8e7f-4526-aa91-23d6c7de5de0" (UID: "929b84a7-8e7f-4526-aa91-23d6c7de5de0"). InnerVolumeSpecName "kube-api-access-62sjs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 01:44:40.068234 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:40.068200    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62sjs\" (UniqueName: \"kubernetes.io/projected/929b84a7-8e7f-4526-aa91-23d6c7de5de0-kube-api-access-62sjs\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:44:40.068234 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:40.068226    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/929b84a7-8e7f-4526-aa91-23d6c7de5de0-plugins-config-volume\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:44:40.135481 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:40.135452    2581 generic.go:356] "Generic (PLEG): container finished" podID="929b84a7-8e7f-4526-aa91-23d6c7de5de0" containerID="54405da5190cced0f65cb8132b369c5cd190599766965441f4d6cadc111b4ec3" exitCode=0
Sep 19 01:44:40.135600 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:40.135499    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-14de36-77dfd7748b-4q4zn"
Sep 19 01:44:40.135600 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:40.135519    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-14de36-77dfd7748b-4q4zn" event={"ID":"929b84a7-8e7f-4526-aa91-23d6c7de5de0","Type":"ContainerDied","Data":"54405da5190cced0f65cb8132b369c5cd190599766965441f4d6cadc111b4ec3"}
Sep 19 01:44:40.135600 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:40.135558    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-14de36-77dfd7748b-4q4zn" event={"ID":"929b84a7-8e7f-4526-aa91-23d6c7de5de0","Type":"ContainerDied","Data":"0f9a6d09a94f670559e5add2ab86cd3c5a6083680abde5106493f1e5d8967ed5"}
Sep 19 01:44:40.135600 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:40.135582    2581 scope.go:117] "RemoveContainer" containerID="54405da5190cced0f65cb8132b369c5cd190599766965441f4d6cadc111b4ec3"
Sep 19 01:44:40.144750 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:40.144733    2581 scope.go:117] "RemoveContainer" containerID="54405da5190cced0f65cb8132b369c5cd190599766965441f4d6cadc111b4ec3"
Sep 19 01:44:40.145071 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:44:40.145053    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54405da5190cced0f65cb8132b369c5cd190599766965441f4d6cadc111b4ec3\": container with ID starting with 54405da5190cced0f65cb8132b369c5cd190599766965441f4d6cadc111b4ec3 not found: ID does not exist" containerID="54405da5190cced0f65cb8132b369c5cd190599766965441f4d6cadc111b4ec3"
Sep 19 01:44:40.145127 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:40.145081    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54405da5190cced0f65cb8132b369c5cd190599766965441f4d6cadc111b4ec3"} err="failed to get container status \"54405da5190cced0f65cb8132b369c5cd190599766965441f4d6cadc111b4ec3\": rpc error: code = NotFound desc = could not find container \"54405da5190cced0f65cb8132b369c5cd190599766965441f4d6cadc111b4ec3\": container with ID starting with 54405da5190cced0f65cb8132b369c5cd190599766965441f4d6cadc111b4ec3 not found: ID does not exist"
Sep 19 01:44:40.156050 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:40.156030    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-14de36-77dfd7748b-4q4zn"]
Sep 19 01:44:40.166954 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:40.166925    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-14de36-77dfd7748b-4q4zn"]
Sep 19 01:44:41.271831 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:41.271799    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="929b84a7-8e7f-4526-aa91-23d6c7de5de0" path="/var/lib/kubelet/pods/929b84a7-8e7f-4526-aa91-23d6c7de5de0/volumes"
Sep 19 01:44:41.272244 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:41.272173    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="932b4c8b-cf88-43a7-9c27-5e469ba9af81" path="/var/lib/kubelet/pods/932b4c8b-cf88-43a7-9c27-5e469ba9af81/volumes"
Sep 19 01:44:43.215079 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:43.215042    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c9nsz"]
Sep 19 01:44:46.934048 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:46.934018    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-751f6a-kq4st"]
Sep 19 01:44:46.934430 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:46.934325    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="932b4c8b-cf88-43a7-9c27-5e469ba9af81" containerName="revoke-keys"
Sep 19 01:44:46.934430 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:46.934336    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="932b4c8b-cf88-43a7-9c27-5e469ba9af81" containerName="revoke-keys"
Sep 19 01:44:46.934430 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:46.934352    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="929b84a7-8e7f-4526-aa91-23d6c7de5de0" containerName="payload-pre-processing"
Sep 19 01:44:46.934430 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:46.934357    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="929b84a7-8e7f-4526-aa91-23d6c7de5de0" containerName="payload-pre-processing"
Sep 19 01:44:46.934430 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:46.934409    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="932b4c8b-cf88-43a7-9c27-5e469ba9af81" containerName="revoke-keys"
Sep 19 01:44:46.934430 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:46.934419    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="929b84a7-8e7f-4526-aa91-23d6c7de5de0" containerName="payload-pre-processing"
Sep 19 01:44:46.938769 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:46.938753    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-751f6a-kq4st"
Sep 19 01:44:46.941054 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:46.941033    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s7pvt\""
Sep 19 01:44:46.941147 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:46.941033    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 19 01:44:46.945580 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:46.945550    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-751f6a-kq4st"]
Sep 19 01:44:47.020386 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:47.020354    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b1e0ab1-61a5-404f-9fdb-8e259376ed10-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-751f6a-kq4st\" (UID: \"5b1e0ab1-61a5-404f-9fdb-8e259376ed10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-751f6a-kq4st"
Sep 19 01:44:47.121116 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:47.121088    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b1e0ab1-61a5-404f-9fdb-8e259376ed10-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-751f6a-kq4st\" (UID: \"5b1e0ab1-61a5-404f-9fdb-8e259376ed10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-751f6a-kq4st"
Sep 19 01:44:47.121707 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:47.121689    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b1e0ab1-61a5-404f-9fdb-8e259376ed10-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-751f6a-kq4st\" (UID: \"5b1e0ab1-61a5-404f-9fdb-8e259376ed10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-751f6a-kq4st"
Sep 19 01:44:47.250601 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:47.250535    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-751f6a-kq4st"
Sep 19 01:44:47.519993 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:47.519922    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-751f6a-kq4st"]
Sep 19 01:44:47.522571 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:44:47.522541    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b1e0ab1_61a5_404f_9fdb_8e259376ed10.slice/crio-2ab7a356327e3af4ef29e180783870edb4fc5a1d6411124a53403da4eec1156f WatchSource:0}: Error finding container 2ab7a356327e3af4ef29e180783870edb4fc5a1d6411124a53403da4eec1156f: Status 404 returned error can't find the container with id 2ab7a356327e3af4ef29e180783870edb4fc5a1d6411124a53403da4eec1156f
Sep 19 01:44:48.173996 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:48.173958    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-751f6a-kq4st" event={"ID":"5b1e0ab1-61a5-404f-9fdb-8e259376ed10","Type":"ContainerStarted","Data":"b3c374bdf2d6437d4feb9d67035f4130cb9d06d7fcf315f4161158337812ed31"}
Sep 19 01:44:48.174359 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:48.174007    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-751f6a-kq4st" event={"ID":"5b1e0ab1-61a5-404f-9fdb-8e259376ed10","Type":"ContainerStarted","Data":"2ab7a356327e3af4ef29e180783870edb4fc5a1d6411124a53403da4eec1156f"}
Sep 19 01:44:48.192950 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:48.192912    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-751f6a-kq4st" podStartSLOduration=2.192895971 podStartE2EDuration="2.192895971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 01:44:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 01:44:48.191950758 +0000 UTC m=+1939.079619855" watchObservedRunningTime="2026-09-19 01:44:48.192895971 +0000 UTC m=+1939.080565049"
Sep 19 01:44:49.180237 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:49.180198    2581 generic.go:356] "Generic (PLEG): container finished" podID="5b1e0ab1-61a5-404f-9fdb-8e259376ed10" containerID="b3c374bdf2d6437d4feb9d67035f4130cb9d06d7fcf315f4161158337812ed31" exitCode=0
Sep 19 01:44:49.180633 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:49.180283    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-751f6a-kq4st" event={"ID":"5b1e0ab1-61a5-404f-9fdb-8e259376ed10","Type":"ContainerDied","Data":"b3c374bdf2d6437d4feb9d67035f4130cb9d06d7fcf315f4161158337812ed31"}
Sep 19 01:44:50.309284 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:50.309262    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-751f6a-kq4st"
Sep 19 01:44:50.450033 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:50.449969    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b1e0ab1-61a5-404f-9fdb-8e259376ed10-maas-api-service-ca\") pod \"5b1e0ab1-61a5-404f-9fdb-8e259376ed10\" (UID: \"5b1e0ab1-61a5-404f-9fdb-8e259376ed10\") "
Sep 19 01:44:50.450288 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:50.450268    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b1e0ab1-61a5-404f-9fdb-8e259376ed10-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5b1e0ab1-61a5-404f-9fdb-8e259376ed10" (UID: "5b1e0ab1-61a5-404f-9fdb-8e259376ed10"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 01:44:50.550959 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:50.550928    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b1e0ab1-61a5-404f-9fdb-8e259376ed10-maas-api-service-ca\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:44:51.191637 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:51.191610    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-751f6a-kq4st"
Sep 19 01:44:51.191637 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:51.191628    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-751f6a-kq4st" event={"ID":"5b1e0ab1-61a5-404f-9fdb-8e259376ed10","Type":"ContainerDied","Data":"2ab7a356327e3af4ef29e180783870edb4fc5a1d6411124a53403da4eec1156f"}
Sep 19 01:44:51.191830 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:51.191670    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ab7a356327e3af4ef29e180783870edb4fc5a1d6411124a53403da4eec1156f"
Sep 19 01:44:57.600274 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:57.600240    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-751f6a-f5b56d759-n267t"]
Sep 19 01:44:57.600738 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:57.600432    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-751f6a-f5b56d759-n267t" podUID="950dbb17-8019-4b84-859b-dbc14565592b" containerName="payload-processing" containerID="cri-o://a4259819185e0b06ed90590e3541e6eb05cda95799be4614d06ff31d4d57c847" gracePeriod=30
Sep 19 01:44:57.615123 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:57.615064    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-751f6a-8546c5d9f8-zw744"]
Sep 19 01:44:57.615300 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:57.615275    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-751f6a-8546c5d9f8-zw744" podUID="e985fb28-5b5b-4515-b453-e3c0e1721e04" containerName="payload-pre-processing" containerID="cri-o://d0697cee6805fbc76735bb84618d8f41373e0f48096e78d30d8e9f6cea96349a" gracePeriod=30
Sep 19 01:44:57.863243 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:57.863143    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-751f6a-kq4st"]
Sep 19 01:44:57.866139 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:57.866108    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-751f6a-kq4st"]
Sep 19 01:44:57.890204 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:57.890187    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-751f6a-f5b56d759-n267t"
Sep 19 01:44:57.892764 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:57.892736    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-751f6a-8546c5d9f8-zw744"
Sep 19 01:44:58.007888 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:58.007836    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-drqx2\" (UniqueName: \"kubernetes.io/projected/950dbb17-8019-4b84-859b-dbc14565592b-kube-api-access-drqx2\") pod \"950dbb17-8019-4b84-859b-dbc14565592b\" (UID: \"950dbb17-8019-4b84-859b-dbc14565592b\") "
Sep 19 01:44:58.008073 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:58.007930    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r9nb4\" (UniqueName: \"kubernetes.io/projected/e985fb28-5b5b-4515-b453-e3c0e1721e04-kube-api-access-r9nb4\") pod \"e985fb28-5b5b-4515-b453-e3c0e1721e04\" (UID: \"e985fb28-5b5b-4515-b453-e3c0e1721e04\") "
Sep 19 01:44:58.008073 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:58.007966    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/950dbb17-8019-4b84-859b-dbc14565592b-plugins-config-volume\") pod \"950dbb17-8019-4b84-859b-dbc14565592b\" (UID: \"950dbb17-8019-4b84-859b-dbc14565592b\") "
Sep 19 01:44:58.008073 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:58.008022    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e985fb28-5b5b-4515-b453-e3c0e1721e04-plugins-config-volume\") pod \"e985fb28-5b5b-4515-b453-e3c0e1721e04\" (UID: \"e985fb28-5b5b-4515-b453-e3c0e1721e04\") "
Sep 19 01:44:58.008436 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:58.008404    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/950dbb17-8019-4b84-859b-dbc14565592b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "950dbb17-8019-4b84-859b-dbc14565592b" (UID: "950dbb17-8019-4b84-859b-dbc14565592b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 01:44:58.008436 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:58.008416    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e985fb28-5b5b-4515-b453-e3c0e1721e04-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e985fb28-5b5b-4515-b453-e3c0e1721e04" (UID: "e985fb28-5b5b-4515-b453-e3c0e1721e04"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 01:44:58.010218 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:58.010192    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e985fb28-5b5b-4515-b453-e3c0e1721e04-kube-api-access-r9nb4" (OuterVolumeSpecName: "kube-api-access-r9nb4") pod "e985fb28-5b5b-4515-b453-e3c0e1721e04" (UID: "e985fb28-5b5b-4515-b453-e3c0e1721e04"). InnerVolumeSpecName "kube-api-access-r9nb4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 01:44:58.010312 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:58.010242    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/950dbb17-8019-4b84-859b-dbc14565592b-kube-api-access-drqx2" (OuterVolumeSpecName: "kube-api-access-drqx2") pod "950dbb17-8019-4b84-859b-dbc14565592b" (UID: "950dbb17-8019-4b84-859b-dbc14565592b"). InnerVolumeSpecName "kube-api-access-drqx2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 01:44:58.109531 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:58.109509    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/950dbb17-8019-4b84-859b-dbc14565592b-plugins-config-volume\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:44:58.109531 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:58.109532    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e985fb28-5b5b-4515-b453-e3c0e1721e04-plugins-config-volume\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:44:58.109652 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:58.109542    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drqx2\" (UniqueName: \"kubernetes.io/projected/950dbb17-8019-4b84-859b-dbc14565592b-kube-api-access-drqx2\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:44:58.109652 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:58.109551    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9nb4\" (UniqueName: \"kubernetes.io/projected/e985fb28-5b5b-4515-b453-e3c0e1721e04-kube-api-access-r9nb4\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:44:58.224415 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:58.224329    2581 generic.go:356] "Generic (PLEG): container finished" podID="e985fb28-5b5b-4515-b453-e3c0e1721e04" containerID="d0697cee6805fbc76735bb84618d8f41373e0f48096e78d30d8e9f6cea96349a" exitCode=0
Sep 19 01:44:58.224415 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:58.224383    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-751f6a-8546c5d9f8-zw744"
Sep 19 01:44:58.224613 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:58.224426    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-751f6a-8546c5d9f8-zw744" event={"ID":"e985fb28-5b5b-4515-b453-e3c0e1721e04","Type":"ContainerDied","Data":"d0697cee6805fbc76735bb84618d8f41373e0f48096e78d30d8e9f6cea96349a"}
Sep 19 01:44:58.224613 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:58.224483    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-751f6a-8546c5d9f8-zw744" event={"ID":"e985fb28-5b5b-4515-b453-e3c0e1721e04","Type":"ContainerDied","Data":"29a97e33e5888c09f164582ffa4179ce7e2f414b68287fa5cc3b8b0e054e754c"}
Sep 19 01:44:58.224613 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:58.224512    2581 scope.go:117] "RemoveContainer" containerID="d0697cee6805fbc76735bb84618d8f41373e0f48096e78d30d8e9f6cea96349a"
Sep 19 01:44:58.225932 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:58.225903    2581 generic.go:356] "Generic (PLEG): container finished" podID="950dbb17-8019-4b84-859b-dbc14565592b" containerID="a4259819185e0b06ed90590e3541e6eb05cda95799be4614d06ff31d4d57c847" exitCode=0
Sep 19 01:44:58.226042 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:58.225950    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-751f6a-f5b56d759-n267t"
Sep 19 01:44:58.226042 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:58.225999    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-751f6a-f5b56d759-n267t" event={"ID":"950dbb17-8019-4b84-859b-dbc14565592b","Type":"ContainerDied","Data":"a4259819185e0b06ed90590e3541e6eb05cda95799be4614d06ff31d4d57c847"}
Sep 19 01:44:58.226170 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:58.226054    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-751f6a-f5b56d759-n267t" event={"ID":"950dbb17-8019-4b84-859b-dbc14565592b","Type":"ContainerDied","Data":"9b8b78bd7023435768f9c46e49094f1a7d2b4226ca0c8f762582425ae99b7ee9"}
Sep 19 01:44:58.235237 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:58.235219    2581 scope.go:117] "RemoveContainer" containerID="d0697cee6805fbc76735bb84618d8f41373e0f48096e78d30d8e9f6cea96349a"
Sep 19 01:44:58.235563 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:44:58.235543    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0697cee6805fbc76735bb84618d8f41373e0f48096e78d30d8e9f6cea96349a\": container with ID starting with d0697cee6805fbc76735bb84618d8f41373e0f48096e78d30d8e9f6cea96349a not found: ID does not exist" containerID="d0697cee6805fbc76735bb84618d8f41373e0f48096e78d30d8e9f6cea96349a"
Sep 19 01:44:58.235621 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:58.235570    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0697cee6805fbc76735bb84618d8f41373e0f48096e78d30d8e9f6cea96349a"} err="failed to get container status \"d0697cee6805fbc76735bb84618d8f41373e0f48096e78d30d8e9f6cea96349a\": rpc error: code = NotFound desc = could not find container \"d0697cee6805fbc76735bb84618d8f41373e0f48096e78d30d8e9f6cea96349a\": container with ID starting with d0697cee6805fbc76735bb84618d8f41373e0f48096e78d30d8e9f6cea96349a not found: ID does not exist"
Sep 19 01:44:58.235621 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:58.235585    2581 scope.go:117] "RemoveContainer" containerID="a4259819185e0b06ed90590e3541e6eb05cda95799be4614d06ff31d4d57c847"
Sep 19 01:44:58.243332 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:58.243312    2581 scope.go:117] "RemoveContainer" containerID="a4259819185e0b06ed90590e3541e6eb05cda95799be4614d06ff31d4d57c847"
Sep 19 01:44:58.243603 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:44:58.243586    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4259819185e0b06ed90590e3541e6eb05cda95799be4614d06ff31d4d57c847\": container with ID starting with a4259819185e0b06ed90590e3541e6eb05cda95799be4614d06ff31d4d57c847 not found: ID does not exist" containerID="a4259819185e0b06ed90590e3541e6eb05cda95799be4614d06ff31d4d57c847"
Sep 19 01:44:58.243655 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:58.243610    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4259819185e0b06ed90590e3541e6eb05cda95799be4614d06ff31d4d57c847"} err="failed to get container status \"a4259819185e0b06ed90590e3541e6eb05cda95799be4614d06ff31d4d57c847\": rpc error: code = NotFound desc = could not find container \"a4259819185e0b06ed90590e3541e6eb05cda95799be4614d06ff31d4d57c847\": container with ID starting with a4259819185e0b06ed90590e3541e6eb05cda95799be4614d06ff31d4d57c847 not found: ID does not exist"
Sep 19 01:44:58.247145 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:58.247126    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-751f6a-8546c5d9f8-zw744"]
Sep 19 01:44:58.252967 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:58.252946    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-751f6a-8546c5d9f8-zw744"]
Sep 19 01:44:58.268055 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:58.268033    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-751f6a-f5b56d759-n267t"]
Sep 19 01:44:58.270601 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:58.270574    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-751f6a-f5b56d759-n267t"]
Sep 19 01:44:59.271261 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:59.271229    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b1e0ab1-61a5-404f-9fdb-8e259376ed10" path="/var/lib/kubelet/pods/5b1e0ab1-61a5-404f-9fdb-8e259376ed10/volumes"
Sep 19 01:44:59.271636 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:59.271570    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="950dbb17-8019-4b84-859b-dbc14565592b" path="/var/lib/kubelet/pods/950dbb17-8019-4b84-859b-dbc14565592b/volumes"
Sep 19 01:44:59.271902 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:44:59.271875    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e985fb28-5b5b-4515-b453-e3c0e1721e04" path="/var/lib/kubelet/pods/e985fb28-5b5b-4515-b453-e3c0e1721e04/volumes"
Sep 19 01:45:00.145009 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:00.144978    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29829705-b2hc7"]
Sep 19 01:45:00.145285 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:00.145273    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b1e0ab1-61a5-404f-9fdb-8e259376ed10" containerName="revoke-keys"
Sep 19 01:45:00.145329 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:00.145287    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b1e0ab1-61a5-404f-9fdb-8e259376ed10" containerName="revoke-keys"
Sep 19 01:45:00.145329 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:00.145304    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="950dbb17-8019-4b84-859b-dbc14565592b" containerName="payload-processing"
Sep 19 01:45:00.145329 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:00.145309    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="950dbb17-8019-4b84-859b-dbc14565592b" containerName="payload-processing"
Sep 19 01:45:00.145329 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:00.145317    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e985fb28-5b5b-4515-b453-e3c0e1721e04" containerName="payload-pre-processing"
Sep 19 01:45:00.145329 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:00.145323    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e985fb28-5b5b-4515-b453-e3c0e1721e04" containerName="payload-pre-processing"
Sep 19 01:45:00.145484 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:00.145376    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="950dbb17-8019-4b84-859b-dbc14565592b" containerName="payload-processing"
Sep 19 01:45:00.145484 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:00.145388    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e985fb28-5b5b-4515-b453-e3c0e1721e04" containerName="payload-pre-processing"
Sep 19 01:45:00.145484 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:00.145399    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b1e0ab1-61a5-404f-9fdb-8e259376ed10" containerName="revoke-keys"
Sep 19 01:45:00.149777 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:00.149758    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829705-b2hc7"
Sep 19 01:45:00.152615 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:00.152594    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s7pvt\""
Sep 19 01:45:00.159178 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:00.159160    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829705-b2hc7"
Sep 19 01:45:00.162228 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:00.162096    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29829705-b2hc7"]
Sep 19 01:45:00.451261 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:00.451234    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29829705-b2hc7"]
Sep 19 01:45:00.452507 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:45:00.452483    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24a134dc_0f1b_4486_944a_371247762b9f.slice/crio-f99fa073a5a9bb8260ba8350cff45d699af115533b38cdd278751b2fc947f57e WatchSource:0}: Error finding container f99fa073a5a9bb8260ba8350cff45d699af115533b38cdd278751b2fc947f57e: Status 404 returned error can't find the container with id f99fa073a5a9bb8260ba8350cff45d699af115533b38cdd278751b2fc947f57e
Sep 19 01:45:00.603370 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:00.603336    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c9nsz"]
Sep 19 01:45:01.248524 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:01.248491    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829705-b2hc7" event={"ID":"24a134dc-0f1b-4486-944a-371247762b9f","Type":"ContainerStarted","Data":"50d8e4895a9ed2f443974405aaeecb5c68ae43d28ddc0898519bc795a2afac37"}
Sep 19 01:45:01.248691 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:01.248541    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829705-b2hc7" event={"ID":"24a134dc-0f1b-4486-944a-371247762b9f","Type":"ContainerStarted","Data":"f99fa073a5a9bb8260ba8350cff45d699af115533b38cdd278751b2fc947f57e"}
Sep 19 01:45:01.288633 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:01.288579    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829705-b2hc7" podStartSLOduration=1.2885647869999999 podStartE2EDuration="1.288564787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 01:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 01:45:01.288109085 +0000 UTC m=+1952.175778163" watchObservedRunningTime="2026-09-19 01:45:01.288564787 +0000 UTC m=+1952.176233864"
Sep 19 01:45:02.254231 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:02.254194    2581 generic.go:356] "Generic (PLEG): container finished" podID="24a134dc-0f1b-4486-944a-371247762b9f" containerID="50d8e4895a9ed2f443974405aaeecb5c68ae43d28ddc0898519bc795a2afac37" exitCode=0
Sep 19 01:45:02.254231 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:02.254233    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829705-b2hc7" event={"ID":"24a134dc-0f1b-4486-944a-371247762b9f","Type":"ContainerDied","Data":"50d8e4895a9ed2f443974405aaeecb5c68ae43d28ddc0898519bc795a2afac37"}
Sep 19 01:45:03.385775 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:03.385750    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829705-b2hc7"
Sep 19 01:45:04.264598 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:04.264562    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829705-b2hc7" event={"ID":"24a134dc-0f1b-4486-944a-371247762b9f","Type":"ContainerDied","Data":"f99fa073a5a9bb8260ba8350cff45d699af115533b38cdd278751b2fc947f57e"}
Sep 19 01:45:04.264598 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:04.264598    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829705-b2hc7"
Sep 19 01:45:04.264801 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:04.264622    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f99fa073a5a9bb8260ba8350cff45d699af115533b38cdd278751b2fc947f57e"
Sep 19 01:45:05.985472 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:05.985439    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-751f6a-openshift-default-549c5bdf4d-78c97"]
Sep 19 01:45:05.985934 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:05.985629    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-751f6a-openshift-default-549c5bdf4d-78c97" podUID="89c7c2d8-487c-4c80-a630-dc0ffc760475" containerName="istio-proxy" containerID="cri-o://5de733c4c3746e2d9e64c623588af7cee5806b96a4c7be8e8f14542c6e0fa3df" gracePeriod=30
Sep 19 01:45:06.355592 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:06.355559    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bebef4-4s7ct"]
Sep 19 01:45:06.355889 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:06.355854    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24a134dc-0f1b-4486-944a-371247762b9f" containerName="cleanup"
Sep 19 01:45:06.355889 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:06.355884    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="24a134dc-0f1b-4486-944a-371247762b9f" containerName="cleanup"
Sep 19 01:45:06.356045 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:06.355937    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="24a134dc-0f1b-4486-944a-371247762b9f" containerName="cleanup"
Sep 19 01:45:06.358738 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:06.358710    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bebef4-4s7ct"
Sep 19 01:45:06.361485 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:06.361467    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 19 01:45:06.361579 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:06.361565    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s7pvt\""
Sep 19 01:45:06.369568 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:06.369544    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bebef4-4s7ct"]
Sep 19 01:45:06.485299 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:06.485270    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3099cff8-6ff8-43e0-99bb-cc8697f89e28-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-bebef4-4s7ct\" (UID: \"3099cff8-6ff8-43e0-99bb-cc8697f89e28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bebef4-4s7ct"
Sep 19 01:45:06.586643 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:06.586599    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3099cff8-6ff8-43e0-99bb-cc8697f89e28-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-bebef4-4s7ct\" (UID: \"3099cff8-6ff8-43e0-99bb-cc8697f89e28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bebef4-4s7ct"
Sep 19 01:45:06.587241 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:06.587219    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3099cff8-6ff8-43e0-99bb-cc8697f89e28-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-bebef4-4s7ct\" (UID: \"3099cff8-6ff8-43e0-99bb-cc8697f89e28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bebef4-4s7ct"
Sep 19 01:45:06.669508 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:06.669444    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bebef4-4s7ct"
Sep 19 01:45:06.954820 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:06.954786    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bebef4-4s7ct"]
Sep 19 01:45:06.957368 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:45:06.957333    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3099cff8_6ff8_43e0_99bb_cc8697f89e28.slice/crio-e29b28317d48ba324839267366a71913b59d13fc306f9c4ffc8f1ef534ef3b14 WatchSource:0}: Error finding container e29b28317d48ba324839267366a71913b59d13fc306f9c4ffc8f1ef534ef3b14: Status 404 returned error can't find the container with id e29b28317d48ba324839267366a71913b59d13fc306f9c4ffc8f1ef534ef3b14
Sep 19 01:45:07.278330 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:07.278302    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bebef4-4s7ct" event={"ID":"3099cff8-6ff8-43e0-99bb-cc8697f89e28","Type":"ContainerStarted","Data":"28be2d6aa666916f7b95e4c5b3cf41d317da06beec5b83eb46a8e1e6f1c00a70"}
Sep 19 01:45:07.278624 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:07.278356    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bebef4-4s7ct" event={"ID":"3099cff8-6ff8-43e0-99bb-cc8697f89e28","Type":"ContainerStarted","Data":"e29b28317d48ba324839267366a71913b59d13fc306f9c4ffc8f1ef534ef3b14"}
Sep 19 01:45:07.304879 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:07.304817    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bebef4-4s7ct" podStartSLOduration=1.304797092 podStartE2EDuration="1.304797092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 01:45:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 01:45:07.302883811 +0000 UTC m=+1958.190552903" watchObservedRunningTime="2026-09-19 01:45:07.304797092 +0000 UTC m=+1958.192466173"
Sep 19 01:45:08.284802 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:08.284770    2581 generic.go:356] "Generic (PLEG): container finished" podID="3099cff8-6ff8-43e0-99bb-cc8697f89e28" containerID="28be2d6aa666916f7b95e4c5b3cf41d317da06beec5b83eb46a8e1e6f1c00a70" exitCode=0
Sep 19 01:45:08.285174 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:08.284832    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bebef4-4s7ct" event={"ID":"3099cff8-6ff8-43e0-99bb-cc8697f89e28","Type":"ContainerDied","Data":"28be2d6aa666916f7b95e4c5b3cf41d317da06beec5b83eb46a8e1e6f1c00a70"}
Sep 19 01:45:09.418941 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:09.418916    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bebef4-4s7ct"
Sep 19 01:45:09.611190 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:09.611166    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3099cff8-6ff8-43e0-99bb-cc8697f89e28-maas-api-service-ca\") pod \"3099cff8-6ff8-43e0-99bb-cc8697f89e28\" (UID: \"3099cff8-6ff8-43e0-99bb-cc8697f89e28\") "
Sep 19 01:45:09.611496 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:09.611473    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3099cff8-6ff8-43e0-99bb-cc8697f89e28-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3099cff8-6ff8-43e0-99bb-cc8697f89e28" (UID: "3099cff8-6ff8-43e0-99bb-cc8697f89e28"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 01:45:09.712237 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:09.712212    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3099cff8-6ff8-43e0-99bb-cc8697f89e28-maas-api-service-ca\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:45:10.295236 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:10.295211    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bebef4-4s7ct"
Sep 19 01:45:10.295412 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:10.295244    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bebef4-4s7ct" event={"ID":"3099cff8-6ff8-43e0-99bb-cc8697f89e28","Type":"ContainerDied","Data":"e29b28317d48ba324839267366a71913b59d13fc306f9c4ffc8f1ef534ef3b14"}
Sep 19 01:45:10.295412 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:10.295287    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e29b28317d48ba324839267366a71913b59d13fc306f9c4ffc8f1ef534ef3b14"
Sep 19 01:45:11.090921 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:11.090843    2581 patch_prober.go:28] interesting pod/e2e-shared-a-w3-751f6a-openshift-default-549c5bdf4d-78c97 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Sep 19 01:45:11.091311 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:11.090963    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-751f6a-openshift-default-549c5bdf4d-78c97" podUID="89c7c2d8-487c-4c80-a630-dc0ffc760475" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Sep 19 01:45:11.234016 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:11.233994    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-751f6a-openshift-default-549c5bdf4d-78c97"
Sep 19 01:45:11.301990 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:11.301913    2581 generic.go:356] "Generic (PLEG): container finished" podID="89c7c2d8-487c-4c80-a630-dc0ffc760475" containerID="5de733c4c3746e2d9e64c623588af7cee5806b96a4c7be8e8f14542c6e0fa3df" exitCode=0
Sep 19 01:45:11.301990 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:11.301956    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-751f6a-openshift-default-549c5bdf4d-78c97" event={"ID":"89c7c2d8-487c-4c80-a630-dc0ffc760475","Type":"ContainerDied","Data":"5de733c4c3746e2d9e64c623588af7cee5806b96a4c7be8e8f14542c6e0fa3df"}
Sep 19 01:45:11.301990 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:11.301959    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-751f6a-openshift-default-549c5bdf4d-78c97"
Sep 19 01:45:11.302238 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:11.301993    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-751f6a-openshift-default-549c5bdf4d-78c97" event={"ID":"89c7c2d8-487c-4c80-a630-dc0ffc760475","Type":"ContainerDied","Data":"304086b076e7820a915a7a3c81b57c49120326ffc88e7bf477ffd96343188a8d"}
Sep 19 01:45:11.302238 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:11.302017    2581 scope.go:117] "RemoveContainer" containerID="5de733c4c3746e2d9e64c623588af7cee5806b96a4c7be8e8f14542c6e0fa3df"
Sep 19 01:45:11.310608 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:11.310591    2581 scope.go:117] "RemoveContainer" containerID="30a726cf43deddaf51c8914f3a6e84104c511c46ccd290b406d5491592e2aa3e"
Sep 19 01:45:11.318229 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:11.318212    2581 scope.go:117] "RemoveContainer" containerID="5de733c4c3746e2d9e64c623588af7cee5806b96a4c7be8e8f14542c6e0fa3df"
Sep 19 01:45:11.318504 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:45:11.318490    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5de733c4c3746e2d9e64c623588af7cee5806b96a4c7be8e8f14542c6e0fa3df\": container with ID starting with 5de733c4c3746e2d9e64c623588af7cee5806b96a4c7be8e8f14542c6e0fa3df not found: ID does not exist" containerID="5de733c4c3746e2d9e64c623588af7cee5806b96a4c7be8e8f14542c6e0fa3df"
Sep 19 01:45:11.318554 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:11.318511    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5de733c4c3746e2d9e64c623588af7cee5806b96a4c7be8e8f14542c6e0fa3df"} err="failed to get container status \"5de733c4c3746e2d9e64c623588af7cee5806b96a4c7be8e8f14542c6e0fa3df\": rpc error: code = NotFound desc = could not find container \"5de733c4c3746e2d9e64c623588af7cee5806b96a4c7be8e8f14542c6e0fa3df\": container with ID starting with 5de733c4c3746e2d9e64c623588af7cee5806b96a4c7be8e8f14542c6e0fa3df not found: ID does not exist"
Sep 19 01:45:11.318554 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:11.318534    2581 scope.go:117] "RemoveContainer" containerID="30a726cf43deddaf51c8914f3a6e84104c511c46ccd290b406d5491592e2aa3e"
Sep 19 01:45:11.318788 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:45:11.318768    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30a726cf43deddaf51c8914f3a6e84104c511c46ccd290b406d5491592e2aa3e\": container with ID starting with 30a726cf43deddaf51c8914f3a6e84104c511c46ccd290b406d5491592e2aa3e not found: ID does not exist" containerID="30a726cf43deddaf51c8914f3a6e84104c511c46ccd290b406d5491592e2aa3e"
Sep 19 01:45:11.318832 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:11.318803    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30a726cf43deddaf51c8914f3a6e84104c511c46ccd290b406d5491592e2aa3e"} err="failed to get container status \"30a726cf43deddaf51c8914f3a6e84104c511c46ccd290b406d5491592e2aa3e\": rpc error: code = NotFound desc = could not find container \"30a726cf43deddaf51c8914f3a6e84104c511c46ccd290b406d5491592e2aa3e\": container with ID starting with 30a726cf43deddaf51c8914f3a6e84104c511c46ccd290b406d5491592e2aa3e not found: ID does not exist"
Sep 19 01:45:11.426532 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:11.426512    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89c7c2d8-487c-4c80-a630-dc0ffc760475-workload-socket\") pod \"89c7c2d8-487c-4c80-a630-dc0ffc760475\" (UID: \"89c7c2d8-487c-4c80-a630-dc0ffc760475\") "
Sep 19 01:45:11.426652 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:11.426552    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rdsbj\" (UniqueName: \"kubernetes.io/projected/89c7c2d8-487c-4c80-a630-dc0ffc760475-kube-api-access-rdsbj\") pod \"89c7c2d8-487c-4c80-a630-dc0ffc760475\" (UID: \"89c7c2d8-487c-4c80-a630-dc0ffc760475\") "
Sep 19 01:45:11.426652 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:11.426581    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89c7c2d8-487c-4c80-a630-dc0ffc760475-credential-socket\") pod \"89c7c2d8-487c-4c80-a630-dc0ffc760475\" (UID: \"89c7c2d8-487c-4c80-a630-dc0ffc760475\") "
Sep 19 01:45:11.426652 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:11.426614    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89c7c2d8-487c-4c80-a630-dc0ffc760475-istio-envoy\") pod \"89c7c2d8-487c-4c80-a630-dc0ffc760475\" (UID: \"89c7c2d8-487c-4c80-a630-dc0ffc760475\") "
Sep 19 01:45:11.426652 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:11.426644    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89c7c2d8-487c-4c80-a630-dc0ffc760475-istio-data\") pod \"89c7c2d8-487c-4c80-a630-dc0ffc760475\" (UID: \"89c7c2d8-487c-4c80-a630-dc0ffc760475\") "
Sep 19 01:45:11.426899 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:11.426686    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89c7c2d8-487c-4c80-a630-dc0ffc760475-istiod-ca-cert\") pod \"89c7c2d8-487c-4c80-a630-dc0ffc760475\" (UID: \"89c7c2d8-487c-4c80-a630-dc0ffc760475\") "
Sep 19 01:45:11.426899 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:11.426732    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89c7c2d8-487c-4c80-a630-dc0ffc760475-istio-podinfo\") pod \"89c7c2d8-487c-4c80-a630-dc0ffc760475\" (UID: \"89c7c2d8-487c-4c80-a630-dc0ffc760475\") "
Sep 19 01:45:11.426899 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:11.426781    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89c7c2d8-487c-4c80-a630-dc0ffc760475-istio-token\") pod \"89c7c2d8-487c-4c80-a630-dc0ffc760475\" (UID: \"89c7c2d8-487c-4c80-a630-dc0ffc760475\") "
Sep 19 01:45:11.426899 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:11.426787    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89c7c2d8-487c-4c80-a630-dc0ffc760475-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "89c7c2d8-487c-4c80-a630-dc0ffc760475" (UID: "89c7c2d8-487c-4c80-a630-dc0ffc760475"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 01:45:11.426899 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:11.426825    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89c7c2d8-487c-4c80-a630-dc0ffc760475-workload-certs\") pod \"89c7c2d8-487c-4c80-a630-dc0ffc760475\" (UID: \"89c7c2d8-487c-4c80-a630-dc0ffc760475\") "
Sep 19 01:45:11.427159 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:11.426959    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89c7c2d8-487c-4c80-a630-dc0ffc760475-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "89c7c2d8-487c-4c80-a630-dc0ffc760475" (UID: "89c7c2d8-487c-4c80-a630-dc0ffc760475"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 01:45:11.427159 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:11.426983    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89c7c2d8-487c-4c80-a630-dc0ffc760475-istio-data" (OuterVolumeSpecName: "istio-data") pod "89c7c2d8-487c-4c80-a630-dc0ffc760475" (UID: "89c7c2d8-487c-4c80-a630-dc0ffc760475"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 01:45:11.427282 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:11.427193    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89c7c2d8-487c-4c80-a630-dc0ffc760475-workload-socket\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:45:11.427282 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:11.427203    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89c7c2d8-487c-4c80-a630-dc0ffc760475-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "89c7c2d8-487c-4c80-a630-dc0ffc760475" (UID: "89c7c2d8-487c-4c80-a630-dc0ffc760475"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 01:45:11.427282 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:11.427216    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89c7c2d8-487c-4c80-a630-dc0ffc760475-credential-socket\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:45:11.427282 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:11.427241    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89c7c2d8-487c-4c80-a630-dc0ffc760475-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "89c7c2d8-487c-4c80-a630-dc0ffc760475" (UID: "89c7c2d8-487c-4c80-a630-dc0ffc760475"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 01:45:11.427458 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:11.427272    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89c7c2d8-487c-4c80-a630-dc0ffc760475-istio-data\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:45:11.428939 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:11.428907    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89c7c2d8-487c-4c80-a630-dc0ffc760475-kube-api-access-rdsbj" (OuterVolumeSpecName: "kube-api-access-rdsbj") pod "89c7c2d8-487c-4c80-a630-dc0ffc760475" (UID: "89c7c2d8-487c-4c80-a630-dc0ffc760475"). InnerVolumeSpecName "kube-api-access-rdsbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 01:45:11.429052 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:11.428976    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89c7c2d8-487c-4c80-a630-dc0ffc760475-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "89c7c2d8-487c-4c80-a630-dc0ffc760475" (UID: "89c7c2d8-487c-4c80-a630-dc0ffc760475"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 01:45:11.429123 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:11.429059    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89c7c2d8-487c-4c80-a630-dc0ffc760475-istio-token" (OuterVolumeSpecName: "istio-token") pod "89c7c2d8-487c-4c80-a630-dc0ffc760475" (UID: "89c7c2d8-487c-4c80-a630-dc0ffc760475"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 01:45:11.429390 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:11.429370    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/89c7c2d8-487c-4c80-a630-dc0ffc760475-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "89c7c2d8-487c-4c80-a630-dc0ffc760475" (UID: "89c7c2d8-487c-4c80-a630-dc0ffc760475"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 19 01:45:11.527913 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:11.527894    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89c7c2d8-487c-4c80-a630-dc0ffc760475-istio-envoy\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:45:11.527913 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:11.527914    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89c7c2d8-487c-4c80-a630-dc0ffc760475-istiod-ca-cert\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:45:11.528053 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:11.527923    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89c7c2d8-487c-4c80-a630-dc0ffc760475-istio-podinfo\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:45:11.528053 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:11.527931    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89c7c2d8-487c-4c80-a630-dc0ffc760475-istio-token\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:45:11.528053 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:11.527941    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89c7c2d8-487c-4c80-a630-dc0ffc760475-workload-certs\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:45:11.528053 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:11.527949    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdsbj\" (UniqueName: \"kubernetes.io/projected/89c7c2d8-487c-4c80-a630-dc0ffc760475-kube-api-access-rdsbj\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:45:11.627855 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:11.627828    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-751f6a-openshift-default-549c5bdf4d-78c97"]
Sep 19 01:45:11.632065 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:11.632044    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-751f6a-openshift-default-549c5bdf4d-78c97"]
Sep 19 01:45:13.275807 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:13.275775    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89c7c2d8-487c-4c80-a630-dc0ffc760475" path="/var/lib/kubelet/pods/89c7c2d8-487c-4c80-a630-dc0ffc760475/volumes"
Sep 19 01:45:16.873771 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:16.873740    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-bebef4-6f75dbf44f-s242z"]
Sep 19 01:45:16.874228 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:16.873995    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bebef4-6f75dbf44f-s242z" podUID="bc8ef285-2554-4f8d-a4b4-cb76af7ccec8" containerName="payload-processing" containerID="cri-o://ddf8559f9210b71fbe3c611e8fd742a59b8e13efc98b6092428da98c7c7d3bb9" gracePeriod=30
Sep 19 01:45:17.063569 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:17.063540    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bebef4-4s7ct"]
Sep 19 01:45:17.068480 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:17.068453    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bebef4-4s7ct"]
Sep 19 01:45:17.120248 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:17.120221    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bebef4-6f75dbf44f-s242z"
Sep 19 01:45:17.167639 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:17.167575    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc8ef285-2554-4f8d-a4b4-cb76af7ccec8-plugins-config-volume\") pod \"bc8ef285-2554-4f8d-a4b4-cb76af7ccec8\" (UID: \"bc8ef285-2554-4f8d-a4b4-cb76af7ccec8\") "
Sep 19 01:45:17.167877 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:17.167643    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kksdl\" (UniqueName: \"kubernetes.io/projected/bc8ef285-2554-4f8d-a4b4-cb76af7ccec8-kube-api-access-kksdl\") pod \"bc8ef285-2554-4f8d-a4b4-cb76af7ccec8\" (UID: \"bc8ef285-2554-4f8d-a4b4-cb76af7ccec8\") "
Sep 19 01:45:17.168028 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:17.168001    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc8ef285-2554-4f8d-a4b4-cb76af7ccec8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bc8ef285-2554-4f8d-a4b4-cb76af7ccec8" (UID: "bc8ef285-2554-4f8d-a4b4-cb76af7ccec8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 01:45:17.170244 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:17.170223    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc8ef285-2554-4f8d-a4b4-cb76af7ccec8-kube-api-access-kksdl" (OuterVolumeSpecName: "kube-api-access-kksdl") pod "bc8ef285-2554-4f8d-a4b4-cb76af7ccec8" (UID: "bc8ef285-2554-4f8d-a4b4-cb76af7ccec8"). InnerVolumeSpecName "kube-api-access-kksdl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 01:45:17.268168 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:17.268150    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc8ef285-2554-4f8d-a4b4-cb76af7ccec8-plugins-config-volume\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:45:17.268251 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:17.268170    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kksdl\" (UniqueName: \"kubernetes.io/projected/bc8ef285-2554-4f8d-a4b4-cb76af7ccec8-kube-api-access-kksdl\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:45:17.271243 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:17.271219    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3099cff8-6ff8-43e0-99bb-cc8697f89e28" path="/var/lib/kubelet/pods/3099cff8-6ff8-43e0-99bb-cc8697f89e28/volumes"
Sep 19 01:45:17.329903 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:17.329856    2581 generic.go:356] "Generic (PLEG): container finished" podID="bc8ef285-2554-4f8d-a4b4-cb76af7ccec8" containerID="ddf8559f9210b71fbe3c611e8fd742a59b8e13efc98b6092428da98c7c7d3bb9" exitCode=0
Sep 19 01:45:17.329983 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:17.329913    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bebef4-6f75dbf44f-s242z"
Sep 19 01:45:17.329983 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:17.329957    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bebef4-6f75dbf44f-s242z" event={"ID":"bc8ef285-2554-4f8d-a4b4-cb76af7ccec8","Type":"ContainerDied","Data":"ddf8559f9210b71fbe3c611e8fd742a59b8e13efc98b6092428da98c7c7d3bb9"}
Sep 19 01:45:17.330060 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:17.330001    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bebef4-6f75dbf44f-s242z" event={"ID":"bc8ef285-2554-4f8d-a4b4-cb76af7ccec8","Type":"ContainerDied","Data":"60a4d279fba2187d693c90ff41cd7e7b9cb4e1c15e6dcc8c0848aee4fc46e063"}
Sep 19 01:45:17.330060 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:17.330021    2581 scope.go:117] "RemoveContainer" containerID="ddf8559f9210b71fbe3c611e8fd742a59b8e13efc98b6092428da98c7c7d3bb9"
Sep 19 01:45:17.338282 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:17.338268    2581 scope.go:117] "RemoveContainer" containerID="ddf8559f9210b71fbe3c611e8fd742a59b8e13efc98b6092428da98c7c7d3bb9"
Sep 19 01:45:17.338573 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:45:17.338558    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddf8559f9210b71fbe3c611e8fd742a59b8e13efc98b6092428da98c7c7d3bb9\": container with ID starting with ddf8559f9210b71fbe3c611e8fd742a59b8e13efc98b6092428da98c7c7d3bb9 not found: ID does not exist" containerID="ddf8559f9210b71fbe3c611e8fd742a59b8e13efc98b6092428da98c7c7d3bb9"
Sep 19 01:45:17.338630 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:17.338579    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddf8559f9210b71fbe3c611e8fd742a59b8e13efc98b6092428da98c7c7d3bb9"} err="failed to get container status \"ddf8559f9210b71fbe3c611e8fd742a59b8e13efc98b6092428da98c7c7d3bb9\": rpc error: code = NotFound desc = could not find container \"ddf8559f9210b71fbe3c611e8fd742a59b8e13efc98b6092428da98c7c7d3bb9\": container with ID starting with ddf8559f9210b71fbe3c611e8fd742a59b8e13efc98b6092428da98c7c7d3bb9 not found: ID does not exist"
Sep 19 01:45:17.345131 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:17.345110    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-bebef4-6f75dbf44f-s242z"]
Sep 19 01:45:17.351246 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:17.351226    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-bebef4-6f75dbf44f-s242z"]
Sep 19 01:45:19.271248 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:19.271215    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc8ef285-2554-4f8d-a4b4-cb76af7ccec8" path="/var/lib/kubelet/pods/bc8ef285-2554-4f8d-a4b4-cb76af7ccec8/volumes"
Sep 19 01:45:19.499106 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:19.499075    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c9nsz"]
Sep 19 01:45:29.308761 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:29.308715    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c9nsz"]
Sep 19 01:45:31.689237 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:45:31.689201    2581 scope.go:117] "RemoveContainer" containerID="106b266a10345a9c09016fa95bcf6b03fa61750a4c052db3bf71d689270a9bd6"
Sep 19 01:46:53.699069 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:46:53.698976    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c9nsz"]
Sep 19 01:47:31.322406 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:47:31.322302    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x785d_f759a12f-5b6f-4924-96f2-dd5e00fc1348/ovn-acl-logging/0.log"
Sep 19 01:47:31.326989 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:47:31.326968    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x785d_f759a12f-5b6f-4924-96f2-dd5e00fc1348/ovn-acl-logging/0.log"
Sep 19 01:47:32.895672 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:47:32.895638    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c9nsz"]
Sep 19 01:47:40.610944 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:47:40.610909    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c9nsz"]
Sep 19 01:47:48.215781 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:47:48.215747    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c9nsz"]
Sep 19 01:47:55.603949 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:47:55.603914    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c9nsz"]
Sep 19 01:48:10.596553 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:48:10.596509    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c9nsz"]
Sep 19 01:48:18.889380 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:48:18.889338    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c9nsz"]
Sep 19 01:49:31.744255 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:49:31.744220    2581 scope.go:117] "RemoveContainer" containerID="4f29adf87c4a346e1b3659adf789ee005b1306cc366d07c8ef2f58091c4752a9"
Sep 19 01:49:31.752804 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:49:31.752784    2581 scope.go:117] "RemoveContainer" containerID="d27602c66d0097af2ddff59250e9e2b39853928226c537d2a86133abde2f3742"
Sep 19 01:49:47.340344 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:49:47.340257    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c9nsz"]
Sep 19 01:49:55.299288 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:49:55.299255    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c9nsz"]
Sep 19 01:50:01.209692 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:50:01.209651    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c9nsz"]
Sep 19 01:50:10.888329 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:50:10.888296    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-0d586241-kserve-677fd49dc-gb2qv"]
Sep 19 01:50:10.888689 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:50:10.888615    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3099cff8-6ff8-43e0-99bb-cc8697f89e28" containerName="revoke-keys"
Sep 19 01:50:10.888689 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:50:10.888628    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3099cff8-6ff8-43e0-99bb-cc8697f89e28" containerName="revoke-keys"
Sep 19 01:50:10.888689 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:50:10.888637    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc8ef285-2554-4f8d-a4b4-cb76af7ccec8" containerName="payload-processing"
Sep 19 01:50:10.888689 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:50:10.888644    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc8ef285-2554-4f8d-a4b4-cb76af7ccec8" containerName="payload-processing"
Sep 19 01:50:10.888689 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:50:10.888655    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89c7c2d8-487c-4c80-a630-dc0ffc760475" containerName="istio-proxy"
Sep 19 01:50:10.888689 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:50:10.888660    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="89c7c2d8-487c-4c80-a630-dc0ffc760475" containerName="istio-proxy"
Sep 19 01:50:10.888911 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:50:10.888703    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89c7c2d8-487c-4c80-a630-dc0ffc760475" containerName="istio-proxy"
Sep 19 01:50:10.888911 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:50:10.888709    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="89c7c2d8-487c-4c80-a630-dc0ffc760475" containerName="istio-proxy"
Sep 19 01:50:10.888911 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:50:10.888764    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="89c7c2d8-487c-4c80-a630-dc0ffc760475" containerName="istio-proxy"
Sep 19 01:50:10.888911 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:50:10.888772    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc8ef285-2554-4f8d-a4b4-cb76af7ccec8" containerName="payload-processing"
Sep 19 01:50:10.888911 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:50:10.888780    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="89c7c2d8-487c-4c80-a630-dc0ffc760475" containerName="istio-proxy"
Sep 19 01:50:10.888911 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:50:10.888787    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3099cff8-6ff8-43e0-99bb-cc8697f89e28" containerName="revoke-keys"
Sep 19 01:50:10.891684 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:50:10.891668    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-0d586241-kserve-677fd49dc-gb2qv"
Sep 19 01:50:10.894196 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:50:10.894169    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-0d586241-kserve-self-signed-certs\""
Sep 19 01:50:10.905006 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:50:10.904982    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-0d586241-kserve-677fd49dc-gb2qv"]
Sep 19 01:50:11.008053 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:50:11.008025    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9nld\" (UniqueName: \"kubernetes.io/projected/f3089cc6-9e87-4674-8a41-ce1b1fe21838-kube-api-access-j9nld\") pod \"e2e-same-id-a-0d586241-kserve-677fd49dc-gb2qv\" (UID: \"f3089cc6-9e87-4674-8a41-ce1b1fe21838\") " pod="llm/e2e-same-id-a-0d586241-kserve-677fd49dc-gb2qv"
Sep 19 01:50:11.008174 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:50:11.008068    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3089cc6-9e87-4674-8a41-ce1b1fe21838-tmp-dir\") pod \"e2e-same-id-a-0d586241-kserve-677fd49dc-gb2qv\" (UID: \"f3089cc6-9e87-4674-8a41-ce1b1fe21838\") " pod="llm/e2e-same-id-a-0d586241-kserve-677fd49dc-gb2qv"
Sep 19 01:50:11.008174 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:50:11.008085    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3089cc6-9e87-4674-8a41-ce1b1fe21838-model-cache\") pod \"e2e-same-id-a-0d586241-kserve-677fd49dc-gb2qv\" (UID: \"f3089cc6-9e87-4674-8a41-ce1b1fe21838\") " pod="llm/e2e-same-id-a-0d586241-kserve-677fd49dc-gb2qv"
Sep 19 01:50:11.008174 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:50:11.008105    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3089cc6-9e87-4674-8a41-ce1b1fe21838-tls-certs\") pod \"e2e-same-id-a-0d586241-kserve-677fd49dc-gb2qv\" (UID: \"f3089cc6-9e87-4674-8a41-ce1b1fe21838\") " pod="llm/e2e-same-id-a-0d586241-kserve-677fd49dc-gb2qv"
Sep 19 01:50:11.008283 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:50:11.008164    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3089cc6-9e87-4674-8a41-ce1b1fe21838-home\") pod \"e2e-same-id-a-0d586241-kserve-677fd49dc-gb2qv\" (UID: \"f3089cc6-9e87-4674-8a41-ce1b1fe21838\") " pod="llm/e2e-same-id-a-0d586241-kserve-677fd49dc-gb2qv"
Sep 19 01:50:11.008283 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:50:11.008208    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3089cc6-9e87-4674-8a41-ce1b1fe21838-dshm\") pod \"e2e-same-id-a-0d586241-kserve-677fd49dc-gb2qv\" (UID: \"f3089cc6-9e87-4674-8a41-ce1b1fe21838\") " pod="llm/e2e-same-id-a-0d586241-kserve-677fd49dc-gb2qv"
Sep 19 01:50:11.109068 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:50:11.109039    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3089cc6-9e87-4674-8a41-ce1b1fe21838-tmp-dir\") pod \"e2e-same-id-a-0d586241-kserve-677fd49dc-gb2qv\" (UID: \"f3089cc6-9e87-4674-8a41-ce1b1fe21838\") " pod="llm/e2e-same-id-a-0d586241-kserve-677fd49dc-gb2qv"
Sep 19 01:50:11.109178 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:50:11.109072    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3089cc6-9e87-4674-8a41-ce1b1fe21838-model-cache\") pod \"e2e-same-id-a-0d586241-kserve-677fd49dc-gb2qv\" (UID: \"f3089cc6-9e87-4674-8a41-ce1b1fe21838\") " pod="llm/e2e-same-id-a-0d586241-kserve-677fd49dc-gb2qv"
Sep 19 01:50:11.109178 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:50:11.109098    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3089cc6-9e87-4674-8a41-ce1b1fe21838-tls-certs\") pod \"e2e-same-id-a-0d586241-kserve-677fd49dc-gb2qv\" (UID: \"f3089cc6-9e87-4674-8a41-ce1b1fe21838\") " pod="llm/e2e-same-id-a-0d586241-kserve-677fd49dc-gb2qv"
Sep 19 01:50:11.109287 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:50:11.109224    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3089cc6-9e87-4674-8a41-ce1b1fe21838-home\") pod \"e2e-same-id-a-0d586241-kserve-677fd49dc-gb2qv\" (UID: \"f3089cc6-9e87-4674-8a41-ce1b1fe21838\") " pod="llm/e2e-same-id-a-0d586241-kserve-677fd49dc-gb2qv"
Sep 19 01:50:11.109287 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:50:11.109267    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3089cc6-9e87-4674-8a41-ce1b1fe21838-dshm\") pod \"e2e-same-id-a-0d586241-kserve-677fd49dc-gb2qv\" (UID: \"f3089cc6-9e87-4674-8a41-ce1b1fe21838\") " pod="llm/e2e-same-id-a-0d586241-kserve-677fd49dc-gb2qv"
Sep 19 01:50:11.109397 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:50:11.109323    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9nld\" (UniqueName: \"kubernetes.io/projected/f3089cc6-9e87-4674-8a41-ce1b1fe21838-kube-api-access-j9nld\") pod \"e2e-same-id-a-0d586241-kserve-677fd49dc-gb2qv\" (UID: \"f3089cc6-9e87-4674-8a41-ce1b1fe21838\") " pod="llm/e2e-same-id-a-0d586241-kserve-677fd49dc-gb2qv"
Sep 19 01:50:11.109515 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:50:11.109493    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3089cc6-9e87-4674-8a41-ce1b1fe21838-tmp-dir\") pod \"e2e-same-id-a-0d586241-kserve-677fd49dc-gb2qv\" (UID: \"f3089cc6-9e87-4674-8a41-ce1b1fe21838\") " pod="llm/e2e-same-id-a-0d586241-kserve-677fd49dc-gb2qv"
Sep 19 01:50:11.109582 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:50:11.109526    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3089cc6-9e87-4674-8a41-ce1b1fe21838-model-cache\") pod \"e2e-same-id-a-0d586241-kserve-677fd49dc-gb2qv\" (UID: \"f3089cc6-9e87-4674-8a41-ce1b1fe21838\") " pod="llm/e2e-same-id-a-0d586241-kserve-677fd49dc-gb2qv"
Sep 19 01:50:11.109582 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:50:11.109564    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3089cc6-9e87-4674-8a41-ce1b1fe21838-home\") pod \"e2e-same-id-a-0d586241-kserve-677fd49dc-gb2qv\" (UID: \"f3089cc6-9e87-4674-8a41-ce1b1fe21838\") " pod="llm/e2e-same-id-a-0d586241-kserve-677fd49dc-gb2qv"
Sep 19 01:50:11.111636 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:50:11.111616    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3089cc6-9e87-4674-8a41-ce1b1fe21838-dshm\") pod \"e2e-same-id-a-0d586241-kserve-677fd49dc-gb2qv\" (UID: \"f3089cc6-9e87-4674-8a41-ce1b1fe21838\") " pod="llm/e2e-same-id-a-0d586241-kserve-677fd49dc-gb2qv"
Sep 19 01:50:11.111731 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:50:11.111704    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3089cc6-9e87-4674-8a41-ce1b1fe21838-tls-certs\") pod \"e2e-same-id-a-0d586241-kserve-677fd49dc-gb2qv\" (UID: \"f3089cc6-9e87-4674-8a41-ce1b1fe21838\") " pod="llm/e2e-same-id-a-0d586241-kserve-677fd49dc-gb2qv"
Sep 19 01:50:11.118360 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:50:11.118334    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9nld\" (UniqueName: \"kubernetes.io/projected/f3089cc6-9e87-4674-8a41-ce1b1fe21838-kube-api-access-j9nld\") pod \"e2e-same-id-a-0d586241-kserve-677fd49dc-gb2qv\" (UID: \"f3089cc6-9e87-4674-8a41-ce1b1fe21838\") " pod="llm/e2e-same-id-a-0d586241-kserve-677fd49dc-gb2qv"
Sep 19 01:50:11.201042 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:50:11.200967    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-0d586241-kserve-677fd49dc-gb2qv"
Sep 19 01:50:11.472083 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:50:11.471972    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-0d586241-kserve-677fd49dc-gb2qv"]
Sep 19 01:50:11.475823 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:50:11.475796    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf3089cc6_9e87_4674_8a41_ce1b1fe21838.slice/crio-e2f8ae3fd9122f095e53fa87d5be4486926c025de8884394bdf0903e7553e031 WatchSource:0}: Error finding container e2f8ae3fd9122f095e53fa87d5be4486926c025de8884394bdf0903e7553e031: Status 404 returned error can't find the container with id e2f8ae3fd9122f095e53fa87d5be4486926c025de8884394bdf0903e7553e031
Sep 19 01:50:11.603673 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:50:11.603640    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-0d586241-kserve-677fd49dc-gb2qv" event={"ID":"f3089cc6-9e87-4674-8a41-ce1b1fe21838","Type":"ContainerStarted","Data":"b593cee4242a1982c9e6f217bfb992ee70fb3b89096cf35ac95bab550480a2eb"}
Sep 19 01:50:11.603797 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:50:11.603685    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-0d586241-kserve-677fd49dc-gb2qv" event={"ID":"f3089cc6-9e87-4674-8a41-ce1b1fe21838","Type":"ContainerStarted","Data":"e2f8ae3fd9122f095e53fa87d5be4486926c025de8884394bdf0903e7553e031"}
Sep 19 01:50:11.620931 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:50:11.620849    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-0d586241-kserve-677fd49dc-gb2qv" podStartSLOduration=1.620833417 podStartE2EDuration="1.620833417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 01:50:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 01:50:11.61959229 +0000 UTC m=+2262.507261378" watchObservedRunningTime="2026-09-19 01:50:11.620833417 +0000 UTC m=+2262.508502527"
Sep 19 01:50:13.694746 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:50:13.694709    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c9nsz"]
Sep 19 01:50:21.201764 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:50:21.201725    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-0d586241-kserve-677fd49dc-gb2qv"
Sep 19 01:50:21.202274 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:50:21.201781    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-0d586241-kserve-677fd49dc-gb2qv"
Sep 19 01:50:21.214087 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:50:21.214065    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-0d586241-kserve-677fd49dc-gb2qv"
Sep 19 01:50:21.663879 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:50:21.663835    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-0d586241-kserve-677fd49dc-gb2qv"
Sep 19 01:50:24.110175 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:50:24.110137    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c9nsz"]
Sep 19 01:50:31.776474 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:50:31.776440    2581 scope.go:117] "RemoveContainer" containerID="3b45cfbf9ac33102cf5f92c576a335182a2d05ba9d1a27686c0d778cee080bc5"
Sep 19 01:50:39.754990 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:50:39.754957    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-0d586241-kserve-677fd49dc-gb2qv"]
Sep 19 01:50:41.891051 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:50:41.891014    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c9nsz"]
Sep 19 01:50:51.207053 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:50:51.207022    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c9nsz"]
Sep 19 01:50:54.773999 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:50:54.773958    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-0d586241-kserve-677fd49dc-gb2qv" podUID="f3089cc6-9e87-4674-8a41-ce1b1fe21838" containerName="main" containerID="cri-o://b593cee4242a1982c9e6f217bfb992ee70fb3b89096cf35ac95bab550480a2eb" gracePeriod=45
Sep 19 01:50:55.017742 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:50:55.017720    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-0d586241-kserve-677fd49dc-gb2qv"
Sep 19 01:50:55.040361 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:50:55.040291    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3089cc6-9e87-4674-8a41-ce1b1fe21838-dshm\") pod \"f3089cc6-9e87-4674-8a41-ce1b1fe21838\" (UID: \"f3089cc6-9e87-4674-8a41-ce1b1fe21838\") "
Sep 19 01:50:55.040361 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:50:55.040332    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3089cc6-9e87-4674-8a41-ce1b1fe21838-tls-certs\") pod \"f3089cc6-9e87-4674-8a41-ce1b1fe21838\" (UID: \"f3089cc6-9e87-4674-8a41-ce1b1fe21838\") "
Sep 19 01:50:55.040548 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:50:55.040395    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3089cc6-9e87-4674-8a41-ce1b1fe21838-model-cache\") pod \"f3089cc6-9e87-4674-8a41-ce1b1fe21838\" (UID: \"f3089cc6-9e87-4674-8a41-ce1b1fe21838\") "
Sep 19 01:50:55.040608 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:50:55.040536    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j9nld\" (UniqueName: \"kubernetes.io/projected/f3089cc6-9e87-4674-8a41-ce1b1fe21838-kube-api-access-j9nld\") pod \"f3089cc6-9e87-4674-8a41-ce1b1fe21838\" (UID: \"f3089cc6-9e87-4674-8a41-ce1b1fe21838\") "
Sep 19 01:50:55.040608 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:50:55.040602    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3089cc6-9e87-4674-8a41-ce1b1fe21838-tmp-dir\") pod \"f3089cc6-9e87-4674-8a41-ce1b1fe21838\" (UID: \"f3089cc6-9e87-4674-8a41-ce1b1fe21838\") "
Sep 19 01:50:55.040715 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:50:55.040648    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3089cc6-9e87-4674-8a41-ce1b1fe21838-home\") pod \"f3089cc6-9e87-4674-8a41-ce1b1fe21838\" (UID: \"f3089cc6-9e87-4674-8a41-ce1b1fe21838\") "
Sep 19 01:50:55.040715 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:50:55.040648    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3089cc6-9e87-4674-8a41-ce1b1fe21838-model-cache" (OuterVolumeSpecName: "model-cache") pod "f3089cc6-9e87-4674-8a41-ce1b1fe21838" (UID: "f3089cc6-9e87-4674-8a41-ce1b1fe21838"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 01:50:55.040969 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:50:55.040946    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3089cc6-9e87-4674-8a41-ce1b1fe21838-model-cache\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:50:55.040969 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:50:55.040948    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3089cc6-9e87-4674-8a41-ce1b1fe21838-home" (OuterVolumeSpecName: "home") pod "f3089cc6-9e87-4674-8a41-ce1b1fe21838" (UID: "f3089cc6-9e87-4674-8a41-ce1b1fe21838"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 01:50:55.041181 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:50:55.040988    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3089cc6-9e87-4674-8a41-ce1b1fe21838-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f3089cc6-9e87-4674-8a41-ce1b1fe21838" (UID: "f3089cc6-9e87-4674-8a41-ce1b1fe21838"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 01:50:55.042790 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:50:55.042763    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3089cc6-9e87-4674-8a41-ce1b1fe21838-dshm" (OuterVolumeSpecName: "dshm") pod "f3089cc6-9e87-4674-8a41-ce1b1fe21838" (UID: "f3089cc6-9e87-4674-8a41-ce1b1fe21838"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 01:50:55.042790 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:50:55.042767    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3089cc6-9e87-4674-8a41-ce1b1fe21838-kube-api-access-j9nld" (OuterVolumeSpecName: "kube-api-access-j9nld") pod "f3089cc6-9e87-4674-8a41-ce1b1fe21838" (UID: "f3089cc6-9e87-4674-8a41-ce1b1fe21838"). InnerVolumeSpecName "kube-api-access-j9nld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 01:50:55.042972 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:50:55.042895    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3089cc6-9e87-4674-8a41-ce1b1fe21838-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f3089cc6-9e87-4674-8a41-ce1b1fe21838" (UID: "f3089cc6-9e87-4674-8a41-ce1b1fe21838"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 01:50:55.141521 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:50:55.141498    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9nld\" (UniqueName: \"kubernetes.io/projected/f3089cc6-9e87-4674-8a41-ce1b1fe21838-kube-api-access-j9nld\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:50:55.141521 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:50:55.141519    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3089cc6-9e87-4674-8a41-ce1b1fe21838-tmp-dir\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:50:55.141658 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:50:55.141530    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3089cc6-9e87-4674-8a41-ce1b1fe21838-home\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:50:55.141658 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:50:55.141537    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3089cc6-9e87-4674-8a41-ce1b1fe21838-dshm\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:50:55.141658 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:50:55.141545    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3089cc6-9e87-4674-8a41-ce1b1fe21838-tls-certs\") on node \"ip-10-0-129-113.ec2.internal\" DevicePath \"\""
Sep 19 01:50:55.794497 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:50:55.794462    2581 generic.go:356] "Generic (PLEG): container finished" podID="f3089cc6-9e87-4674-8a41-ce1b1fe21838" containerID="b593cee4242a1982c9e6f217bfb992ee70fb3b89096cf35ac95bab550480a2eb" exitCode=0
Sep 19 01:50:55.794913 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:50:55.794503    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-0d586241-kserve-677fd49dc-gb2qv"
Sep 19 01:50:55.794913 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:50:55.794543    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-0d586241-kserve-677fd49dc-gb2qv" event={"ID":"f3089cc6-9e87-4674-8a41-ce1b1fe21838","Type":"ContainerDied","Data":"b593cee4242a1982c9e6f217bfb992ee70fb3b89096cf35ac95bab550480a2eb"}
Sep 19 01:50:55.794913 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:50:55.794593    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-0d586241-kserve-677fd49dc-gb2qv" event={"ID":"f3089cc6-9e87-4674-8a41-ce1b1fe21838","Type":"ContainerDied","Data":"e2f8ae3fd9122f095e53fa87d5be4486926c025de8884394bdf0903e7553e031"}
Sep 19 01:50:55.794913 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:50:55.794612    2581 scope.go:117] "RemoveContainer" containerID="b593cee4242a1982c9e6f217bfb992ee70fb3b89096cf35ac95bab550480a2eb"
Sep 19 01:50:55.803020 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:50:55.802997    2581 scope.go:117] "RemoveContainer" containerID="b593cee4242a1982c9e6f217bfb992ee70fb3b89096cf35ac95bab550480a2eb"
Sep 19 01:50:55.803284 ip-10-0-129-113 kubenswrapper[2581]: E0919 01:50:55.803265    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b593cee4242a1982c9e6f217bfb992ee70fb3b89096cf35ac95bab550480a2eb\": container with ID starting with b593cee4242a1982c9e6f217bfb992ee70fb3b89096cf35ac95bab550480a2eb not found: ID does not exist" containerID="b593cee4242a1982c9e6f217bfb992ee70fb3b89096cf35ac95bab550480a2eb"
Sep 19 01:50:55.803342 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:50:55.803293    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b593cee4242a1982c9e6f217bfb992ee70fb3b89096cf35ac95bab550480a2eb"} err="failed to get container status \"b593cee4242a1982c9e6f217bfb992ee70fb3b89096cf35ac95bab550480a2eb\": rpc error: code = NotFound desc = could not find container \"b593cee4242a1982c9e6f217bfb992ee70fb3b89096cf35ac95bab550480a2eb\": container with ID starting with b593cee4242a1982c9e6f217bfb992ee70fb3b89096cf35ac95bab550480a2eb not found: ID does not exist"
Sep 19 01:50:55.810763 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:50:55.810738    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-0d586241-kserve-677fd49dc-gb2qv"]
Sep 19 01:50:55.815129 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:50:55.815108    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-0d586241-kserve-677fd49dc-gb2qv"]
Sep 19 01:50:57.276985 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:50:57.276938    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3089cc6-9e87-4674-8a41-ce1b1fe21838" path="/var/lib/kubelet/pods/f3089cc6-9e87-4674-8a41-ce1b1fe21838/volumes"
Sep 19 01:51:00.322209 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:51:00.322175    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c9nsz"]
Sep 19 01:51:08.299584 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:51:08.299548    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c9nsz"]
Sep 19 01:51:19.695771 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:51:19.695694    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c9nsz"]
Sep 19 01:51:25.693789 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:51:25.693759    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c9nsz"]
Sep 19 01:51:31.794179 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:51:31.794151    2581 scope.go:117] "RemoveContainer" containerID="28be2d6aa666916f7b95e4c5b3cf41d317da06beec5b83eb46a8e1e6f1c00a70"
Sep 19 01:51:31.801810 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:51:31.801790    2581 scope.go:117] "RemoveContainer" containerID="b3c374bdf2d6437d4feb9d67035f4130cb9d06d7fcf315f4161158337812ed31"
Sep 19 01:51:32.890197 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:51:32.890156    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-c9nsz"]
Sep 19 01:51:42.125101 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:51:42.125067    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-748475db69-bw4wd_53e08af2-a376-4c3e-a17f-efabef25eca8/copy-manifests/0.log"
Sep 19 01:51:42.131771 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:51:42.131745    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-748475db69-bw4wd_53e08af2-a376-4c3e-a17f-efabef25eca8/manager/0.log"
Sep 19 01:51:42.860673 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:51:42.860636    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29829690-lw8pt_e69b83de-6d45-4d82-8ba0-364cc2ca6bdd/cleanup/0.log"
Sep 19 01:51:42.974945 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:51:42.974919    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29829705-b2hc7_24a134dc-0f1b-4486-944a-371247762b9f/cleanup/0.log"
Sep 19 01:51:44.259729 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:51:44.259701    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-8swpc_028bd24e-116f-4004-a0e2-de3b4943146f/kuadrant-console-plugin/0.log"
Sep 19 01:51:44.523886 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:51:44.523789    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-c9nsz_4a21d745-92cb-4219-986d-d7bb9fb324d3/limitador/0.log"
Sep 19 01:51:45.247284 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:51:45.247257    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5c6796f956-5wjrr_88237eb3-d014-4bf8-9d7f-05ea6cdc1768/kube-auth-proxy/0.log"
Sep 19 01:51:46.333000 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:51:46.332969    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-dnrsw_7847b3fe-4b9a-4416-93ec-5c5ecbce7687/main/0.log"
Sep 19 01:51:46.554227 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:51:46.554192    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-9j4jn_e4520d6a-eb0f-44ac-9005-b19e6b54fc55/main/0.log"
Sep 19 01:51:46.800490 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:51:46.800466    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-hjzvw_ea0f8eff-42f5-468d-a817-bd481182721f/main/0.log"
Sep 19 01:51:46.935764 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:51:46.935737    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-xsc74_680cad6a-5113-4b67-b0b1-80d04e4697eb/main/0.log"
Sep 19 01:51:55.724124 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:51:55.724086    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-hrgqz_2ff7ef56-5c24-40a5-84d6-7259cf83a29c/global-pull-secret-syncer/0.log"
Sep 19 01:51:55.905595 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:51:55.905566    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-8ndtv_51c7d851-c313-4247-a65c-45f9403ddb9c/konnectivity-agent/0.log"
Sep 19 01:51:55.972288 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:51:55.972242    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-113.ec2.internal_bf17d6182a2ab91b4c163d92cc1c844e/haproxy/0.log"
Sep 19 01:52:00.995500 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:52:00.995472    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-8swpc_028bd24e-116f-4004-a0e2-de3b4943146f/kuadrant-console-plugin/0.log"
Sep 19 01:52:01.123883 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:52:01.123839    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-c9nsz_4a21d745-92cb-4219-986d-d7bb9fb324d3/limitador/0.log"
Sep 19 01:52:03.376778 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:52:03.376751    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6497c8758-jhscm_c4bbe6b9-a10c-442c-baed-8f3a102c36df/metrics-server/0.log"
Sep 19 01:52:03.408451 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:52:03.408430    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5984459ff8-dtn4x_55f739af-85c0-4d5a-87cc-e104fcbd1a86/monitoring-plugin/0.log"
Sep 19 01:52:03.539348 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:52:03.539327    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cdprj_8e1d7cd1-1fa0-422e-a512-fef514857934/node-exporter/0.log"
Sep 19 01:52:03.560392 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:52:03.560374    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cdprj_8e1d7cd1-1fa0-422e-a512-fef514857934/kube-rbac-proxy/0.log"
Sep 19 01:52:03.583310 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:52:03.583293    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cdprj_8e1d7cd1-1fa0-422e-a512-fef514857934/init-textfile/0.log"
Sep 19 01:52:05.741593 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:52:05.741553    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-vc5k7/perf-node-gather-daemonset-4kb7r"]
Sep 19 01:52:05.742077 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:52:05.741993    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3089cc6-9e87-4674-8a41-ce1b1fe21838" containerName="main"
Sep 19 01:52:05.742077 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:52:05.742011    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3089cc6-9e87-4674-8a41-ce1b1fe21838" containerName="main"
Sep 19 01:52:05.742192 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:52:05.742109    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3089cc6-9e87-4674-8a41-ce1b1fe21838" containerName="main"
Sep 19 01:52:05.745040 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:52:05.745020    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vc5k7/perf-node-gather-daemonset-4kb7r"
Sep 19 01:52:05.747395 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:52:05.747369    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-vc5k7\"/\"default-dockercfg-gbbnr\""
Sep 19 01:52:05.748091 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:52:05.748073    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vc5k7\"/\"kube-root-ca.crt\""
Sep 19 01:52:05.748183 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:52:05.748077    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vc5k7\"/\"openshift-service-ca.crt\""
Sep 19 01:52:05.754298 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:52:05.754277    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vc5k7/perf-node-gather-daemonset-4kb7r"]
Sep 19 01:52:05.777796 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:52:05.777775    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/96266cdf-4f3c-4a91-a8b8-07bca4d18429-lib-modules\") pod \"perf-node-gather-daemonset-4kb7r\" (UID: \"96266cdf-4f3c-4a91-a8b8-07bca4d18429\") " pod="openshift-must-gather-vc5k7/perf-node-gather-daemonset-4kb7r"
Sep 19 01:52:05.777944 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:52:05.777816    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4xw9\" (UniqueName: \"kubernetes.io/projected/96266cdf-4f3c-4a91-a8b8-07bca4d18429-kube-api-access-f4xw9\") pod \"perf-node-gather-daemonset-4kb7r\" (UID: \"96266cdf-4f3c-4a91-a8b8-07bca4d18429\") " pod="openshift-must-gather-vc5k7/perf-node-gather-daemonset-4kb7r"
Sep 19 01:52:05.777944 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:52:05.777848    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/96266cdf-4f3c-4a91-a8b8-07bca4d18429-sys\") pod \"perf-node-gather-daemonset-4kb7r\" (UID: \"96266cdf-4f3c-4a91-a8b8-07bca4d18429\") " pod="openshift-must-gather-vc5k7/perf-node-gather-daemonset-4kb7r"
Sep 19 01:52:05.777944 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:52:05.777893    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/96266cdf-4f3c-4a91-a8b8-07bca4d18429-proc\") pod \"perf-node-gather-daemonset-4kb7r\" (UID: \"96266cdf-4f3c-4a91-a8b8-07bca4d18429\") " pod="openshift-must-gather-vc5k7/perf-node-gather-daemonset-4kb7r"
Sep 19 01:52:05.777944 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:52:05.777932    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/96266cdf-4f3c-4a91-a8b8-07bca4d18429-podres\") pod \"perf-node-gather-daemonset-4kb7r\" (UID: \"96266cdf-4f3c-4a91-a8b8-07bca4d18429\") " pod="openshift-must-gather-vc5k7/perf-node-gather-daemonset-4kb7r"
Sep 19 01:52:05.879063 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:52:05.879038    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/96266cdf-4f3c-4a91-a8b8-07bca4d18429-sys\") pod \"perf-node-gather-daemonset-4kb7r\" (UID: \"96266cdf-4f3c-4a91-a8b8-07bca4d18429\") " pod="openshift-must-gather-vc5k7/perf-node-gather-daemonset-4kb7r"
Sep 19 01:52:05.879063 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:52:05.879065    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/96266cdf-4f3c-4a91-a8b8-07bca4d18429-proc\") pod \"perf-node-gather-daemonset-4kb7r\" (UID: \"96266cdf-4f3c-4a91-a8b8-07bca4d18429\") " pod="openshift-must-gather-vc5k7/perf-node-gather-daemonset-4kb7r"
Sep 19 01:52:05.879205 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:52:05.879096    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/96266cdf-4f3c-4a91-a8b8-07bca4d18429-podres\") pod \"perf-node-gather-daemonset-4kb7r\" (UID: \"96266cdf-4f3c-4a91-a8b8-07bca4d18429\") " pod="openshift-must-gather-vc5k7/perf-node-gather-daemonset-4kb7r"
Sep 19 01:52:05.879205 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:52:05.879152    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/96266cdf-4f3c-4a91-a8b8-07bca4d18429-sys\") pod \"perf-node-gather-daemonset-4kb7r\" (UID: \"96266cdf-4f3c-4a91-a8b8-07bca4d18429\") " pod="openshift-must-gather-vc5k7/perf-node-gather-daemonset-4kb7r"
Sep 19 01:52:05.879205 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:52:05.879158    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/96266cdf-4f3c-4a91-a8b8-07bca4d18429-proc\") pod \"perf-node-gather-daemonset-4kb7r\" (UID: \"96266cdf-4f3c-4a91-a8b8-07bca4d18429\") " pod="openshift-must-gather-vc5k7/perf-node-gather-daemonset-4kb7r"
Sep 19 01:52:05.879205 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:52:05.879167    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/96266cdf-4f3c-4a91-a8b8-07bca4d18429-lib-modules\") pod \"perf-node-gather-daemonset-4kb7r\" (UID: \"96266cdf-4f3c-4a91-a8b8-07bca4d18429\") " pod="openshift-must-gather-vc5k7/perf-node-gather-daemonset-4kb7r"
Sep 19 01:52:05.879205 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:52:05.879200    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4xw9\" (UniqueName: \"kubernetes.io/projected/96266cdf-4f3c-4a91-a8b8-07bca4d18429-kube-api-access-f4xw9\") pod \"perf-node-gather-daemonset-4kb7r\" (UID: \"96266cdf-4f3c-4a91-a8b8-07bca4d18429\") " pod="openshift-must-gather-vc5k7/perf-node-gather-daemonset-4kb7r"
Sep 19 01:52:05.879406 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:52:05.879253    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/96266cdf-4f3c-4a91-a8b8-07bca4d18429-podres\") pod \"perf-node-gather-daemonset-4kb7r\" (UID: \"96266cdf-4f3c-4a91-a8b8-07bca4d18429\") " pod="openshift-must-gather-vc5k7/perf-node-gather-daemonset-4kb7r"
Sep 19 01:52:05.879406 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:52:05.879330    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/96266cdf-4f3c-4a91-a8b8-07bca4d18429-lib-modules\") pod \"perf-node-gather-daemonset-4kb7r\" (UID: \"96266cdf-4f3c-4a91-a8b8-07bca4d18429\") " pod="openshift-must-gather-vc5k7/perf-node-gather-daemonset-4kb7r"
Sep 19 01:52:05.887478 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:52:05.887452    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4xw9\" (UniqueName: \"kubernetes.io/projected/96266cdf-4f3c-4a91-a8b8-07bca4d18429-kube-api-access-f4xw9\") pod \"perf-node-gather-daemonset-4kb7r\" (UID: \"96266cdf-4f3c-4a91-a8b8-07bca4d18429\") " pod="openshift-must-gather-vc5k7/perf-node-gather-daemonset-4kb7r"
Sep 19 01:52:06.054744 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:52:06.054679    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vc5k7/perf-node-gather-daemonset-4kb7r"
Sep 19 01:52:06.323084 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:52:06.323008    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vc5k7/perf-node-gather-daemonset-4kb7r"]
Sep 19 01:52:06.326361 ip-10-0-129-113 kubenswrapper[2581]: W0919 01:52:06.326333    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod96266cdf_4f3c_4a91_a8b8_07bca4d18429.slice/crio-28939298c9700e4db12dd67d79e52c732a9bcd3b5772aa7f9ad69e47cfa8aabf WatchSource:0}: Error finding container 28939298c9700e4db12dd67d79e52c732a9bcd3b5772aa7f9ad69e47cfa8aabf: Status 404 returned error can't find the container with id 28939298c9700e4db12dd67d79e52c732a9bcd3b5772aa7f9ad69e47cfa8aabf
Sep 19 01:52:07.096573 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:52:07.096537    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vc5k7/perf-node-gather-daemonset-4kb7r" event={"ID":"96266cdf-4f3c-4a91-a8b8-07bca4d18429","Type":"ContainerStarted","Data":"cef4f921f2df9664d1ea81b94ebc5b4a1e9580e02fe9ee3285b26cfed0587096"}
Sep 19 01:52:07.096958 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:52:07.096597    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vc5k7/perf-node-gather-daemonset-4kb7r" event={"ID":"96266cdf-4f3c-4a91-a8b8-07bca4d18429","Type":"ContainerStarted","Data":"28939298c9700e4db12dd67d79e52c732a9bcd3b5772aa7f9ad69e47cfa8aabf"}
Sep 19 01:52:07.096958 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:52:07.096683    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-vc5k7/perf-node-gather-daemonset-4kb7r"
Sep 19 01:52:07.116193 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:52:07.116141    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-vc5k7/perf-node-gather-daemonset-4kb7r" podStartSLOduration=2.11612845 podStartE2EDuration="2.11612845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 01:52:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 01:52:07.114087808 +0000 UTC m=+2378.001756885" watchObservedRunningTime="2026-09-19 01:52:07.11612845 +0000 UTC m=+2378.003797564"
Sep 19 01:52:08.480513 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:52:08.480482    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-v6mxp_a4338a4a-7257-495a-8e73-952e81106c45/dns/0.log"
Sep 19 01:52:08.501397 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:52:08.501366    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-v6mxp_a4338a4a-7257-495a-8e73-952e81106c45/kube-rbac-proxy/0.log"
Sep 19 01:52:08.660814 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:52:08.660786    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-nx2sn_be4767d8-9458-4c5a-bc2b-d033a86f7017/dns-node-resolver/0.log"
Sep 19 01:52:09.321813 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:52:09.321779    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vrlkw_ff802e32-5c32-466d-b76f-252275c6fb82/node-ca/0.log"
Sep 19 01:52:11.192194 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:52:11.192163    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5c6796f956-5wjrr_88237eb3-d014-4bf8-9d7f-05ea6cdc1768/kube-auth-proxy/0.log"
Sep 19 01:52:12.298469 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:52:12.298439    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9wqc6_eef21c42-fb10-46ba-9ba9-fff960f461a6/serve-healthcheck-canary/0.log"
Sep 19 01:52:12.931042 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:52:12.931012    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7nbtf_da1df6c8-9dc8-44c2-9093-66d5f87ffd35/kube-rbac-proxy/0.log"
Sep 19 01:52:12.950851 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:52:12.950826    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7nbtf_da1df6c8-9dc8-44c2-9093-66d5f87ffd35/exporter/0.log"
Sep 19 01:52:12.972551 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:52:12.972528    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7nbtf_da1df6c8-9dc8-44c2-9093-66d5f87ffd35/extractor/0.log"
Sep 19 01:52:13.110704 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:52:13.110682    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-vc5k7/perf-node-gather-daemonset-4kb7r"
Sep 19 01:52:16.364525 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:52:16.364494    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-748475db69-bw4wd_53e08af2-a376-4c3e-a17f-efabef25eca8/manager/0.log"
Sep 19 01:52:16.386848 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:52:16.386825    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-748475db69-bw4wd_53e08af2-a376-4c3e-a17f-efabef25eca8/copy-manifests/0.log"
Sep 19 01:52:16.936053 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:52:16.936030    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5c6796f956-5wjrr_88237eb3-d014-4bf8-9d7f-05ea6cdc1768/kube-auth-proxy/0.log"
Sep 19 01:52:17.933474 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:52:17.933445    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-74c67d8b6f-tt6jk_d18b558d-3914-4240-a629-4df992413e22/manager/0.log"
Sep 19 01:52:19.468665 ip-10-0-129-113 kubenswrapper[2581]: I0919 01:52:19.468637    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5c6796f956-5wjrr_88237eb3-d014-4bf8-9d7f-05ea6cdc1768/kube-auth-proxy/0.log"