Sep 15 12:00:16.236188 ip-10-0-140-17 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 15 12:00:16.236198 ip-10-0-140-17 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 15 12:00:16.236208 ip-10-0-140-17 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 15 12:00:16.236397 ip-10-0-140-17 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 15 12:00:26.268363 ip-10-0-140-17 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 15 12:00:26.268385 ip-10-0-140-17 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 5e48b1ce9c0e4eae961c63d38930b824 --
Sep 15 12:02:23.101633 ip-10-0-140-17 systemd[1]: Starting Kubernetes Kubelet...
Sep 15 12:02:23.523127 ip-10-0-140-17 kubenswrapper[2555]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 15 12:02:23.523127 ip-10-0-140-17 kubenswrapper[2555]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 15 12:02:23.523127 ip-10-0-140-17 kubenswrapper[2555]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 15 12:02:23.523127 ip-10-0-140-17 kubenswrapper[2555]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 15 12:02:23.523127 ip-10-0-140-17 kubenswrapper[2555]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 15 12:02:23.524290 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.524189    2555 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 15 12:02:23.532278 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.532235    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 12:02:23.532278 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.532284    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 12:02:23.532424 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.532291    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 12:02:23.532424 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.532295    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 12:02:23.532424 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.532299    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 12:02:23.532424 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.532304    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 12:02:23.532424 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.532309    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 12:02:23.532424 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.532313    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 12:02:23.532424 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.532317    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 12:02:23.532424 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.532321    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 12:02:23.532424 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.532325    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 12:02:23.532424 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.532329    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 12:02:23.532424 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.532334    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 12:02:23.532424 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.532337    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 12:02:23.532424 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.532342    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 12:02:23.532424 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.532345    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 12:02:23.532424 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.532349    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 12:02:23.532424 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.532353    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 12:02:23.532424 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.532356    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 12:02:23.532424 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.532368    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 12:02:23.532424 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.532373    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 12:02:23.532885 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.532378    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 12:02:23.532885 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.532384    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 12:02:23.532885 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.532389    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 12:02:23.532885 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.532394    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 12:02:23.532885 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.532398    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 12:02:23.532885 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.532402    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 12:02:23.532885 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.532406    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 12:02:23.532885 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.532410    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 12:02:23.532885 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.532414    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 12:02:23.532885 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.532417    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 12:02:23.532885 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.532422    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 12:02:23.532885 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.532426    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 12:02:23.532885 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.532433    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 12:02:23.532885 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.532439    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 12:02:23.532885 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.532444    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 12:02:23.532885 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.532449    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 12:02:23.532885 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.532453    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 12:02:23.532885 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.532456    2555 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 12:02:23.532885 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.532460    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 12:02:23.533404 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.532463    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 12:02:23.533404 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.532468    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 12:02:23.533404 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.532471    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 12:02:23.533404 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.532475    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 12:02:23.533404 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.532478    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 12:02:23.533404 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.532482    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 12:02:23.533404 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.532486    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 12:02:23.533404 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.532490    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 12:02:23.533404 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.532494    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 12:02:23.533404 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.532498    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 12:02:23.533404 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.532503    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 12:02:23.533404 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.532507    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 12:02:23.533404 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.532510    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 12:02:23.533404 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.532515    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 12:02:23.533404 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.532519    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 12:02:23.533404 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.532523    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 12:02:23.533404 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.532526    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 12:02:23.533404 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.532531    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 12:02:23.533404 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.532535    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 12:02:23.533404 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.532538    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 12:02:23.533889 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.532541    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 12:02:23.533889 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.532545    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 12:02:23.533889 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.532551    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 12:02:23.533889 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.532554    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 12:02:23.533889 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.532558    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 12:02:23.533889 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.532562    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 12:02:23.533889 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.532565    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 12:02:23.533889 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.532569    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 12:02:23.533889 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.532573    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 12:02:23.533889 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.532577    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 12:02:23.533889 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.532581    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 12:02:23.533889 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.532584    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 12:02:23.533889 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.532588    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 12:02:23.533889 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.532592    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 12:02:23.533889 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.532596    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 12:02:23.533889 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.532599    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 12:02:23.533889 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.532603    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 12:02:23.533889 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.532606    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 12:02:23.533889 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.532611    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 12:02:23.533889 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.532614    2555 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 12:02:23.534392 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.532618    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 12:02:23.534392 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.532622    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 12:02:23.534392 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.532627    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 12:02:23.534392 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.532631    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 12:02:23.534392 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.532635    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 12:02:23.534392 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.532641    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 12:02:23.534392 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.532648    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 12:02:23.534392 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.532659    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 12:02:23.534392 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.532663    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 12:02:23.534392 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.532667    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 12:02:23.534392 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.532671    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 12:02:23.534392 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.532674    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 12:02:23.534392 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.532678    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 12:02:23.534392 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.532682    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 12:02:23.534392 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.532686    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 12:02:23.534392 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.532689    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 12:02:23.534392 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.532694    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 12:02:23.534392 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.532698    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 12:02:23.534392 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.532701    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 12:02:23.534392 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.533739    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 12:02:23.534886 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.533748    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 12:02:23.534886 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.533753    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 12:02:23.534886 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.533757    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 12:02:23.534886 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.533760    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 12:02:23.534886 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.533765    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 12:02:23.534886 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.533769    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 12:02:23.534886 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.533773    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 12:02:23.534886 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.533776    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 12:02:23.534886 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.533780    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 12:02:23.534886 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.533785    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 12:02:23.534886 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.533789    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 12:02:23.534886 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.533792    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 12:02:23.534886 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.533796    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 12:02:23.534886 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.533800    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 12:02:23.534886 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.533805    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 12:02:23.534886 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.533809    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 12:02:23.534886 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.533819    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 12:02:23.534886 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.533826    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 12:02:23.534886 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.533833    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 12:02:23.535384 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.533839    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 12:02:23.535384 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.533843    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 12:02:23.535384 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.533848    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 12:02:23.535384 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.533852    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 12:02:23.535384 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.533856    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 12:02:23.535384 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.533860    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 12:02:23.535384 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.533864    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 12:02:23.535384 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.533868    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 12:02:23.535384 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.533872    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 12:02:23.535384 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.533876    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 12:02:23.535384 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.533880    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 12:02:23.535384 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.533884    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 12:02:23.535384 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.533888    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 12:02:23.535384 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.533892    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 12:02:23.535384 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.533896    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 12:02:23.535384 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.533900    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 12:02:23.535384 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.533904    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 12:02:23.535384 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.533908    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 12:02:23.535384 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.533912    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 12:02:23.535384 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.533916    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 12:02:23.535873 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.533919    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 12:02:23.535873 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.533923    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 12:02:23.535873 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.533926    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 12:02:23.535873 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.533930    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 12:02:23.535873 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.533934    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 12:02:23.535873 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.533937    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 12:02:23.535873 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.533940    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 12:02:23.535873 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.533944    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 12:02:23.535873 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.533948    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 12:02:23.535873 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.533951    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 12:02:23.535873 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.533957    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 12:02:23.535873 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.533961    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 12:02:23.535873 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.533965    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 12:02:23.535873 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.533968    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 12:02:23.535873 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.533973    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 12:02:23.535873 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.533977    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 12:02:23.535873 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.533981    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 12:02:23.535873 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.533985    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 12:02:23.535873 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.533988    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 12:02:23.536377 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.533992    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 12:02:23.536377 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.533996    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 12:02:23.536377 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534000    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 12:02:23.536377 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534003    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 12:02:23.536377 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534008    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 12:02:23.536377 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534012    2555 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 12:02:23.536377 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534016    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 12:02:23.536377 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534019    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 12:02:23.536377 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534024    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 12:02:23.536377 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534027    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 12:02:23.536377 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534031    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 12:02:23.536377 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534034    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 12:02:23.536377 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534038    2555 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 12:02:23.536377 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534042    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 12:02:23.536377 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534046    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 12:02:23.536377 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534049    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 12:02:23.536377 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534054    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 12:02:23.536377 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534058    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 12:02:23.536377 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534061    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 12:02:23.536377 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534065    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 12:02:23.536882 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534068    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 12:02:23.536882 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534072    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 12:02:23.536882 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534076    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 12:02:23.536882 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534080    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 12:02:23.536882 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534086    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 12:02:23.536882 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534090    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 12:02:23.536882 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534094    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 12:02:23.536882 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534098    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 12:02:23.536882 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534103    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 12:02:23.536882 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534108    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 12:02:23.536882 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534113    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 12:02:23.536882 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534117    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 12:02:23.536882 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534120    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 12:02:23.536882 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534124    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 12:02:23.536882 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534128    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 12:02:23.536882 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534132    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 12:02:23.536882 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534136    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 12:02:23.536882 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534140    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 12:02:23.536882 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534144    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 12:02:23.537365 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534147    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 12:02:23.537365 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534714    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 12:02:23.537365 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534721    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 12:02:23.537365 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534726    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 12:02:23.537365 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534731    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 12:02:23.537365 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534735    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 12:02:23.537365 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534739    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 12:02:23.537365 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534743    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 12:02:23.537365 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534747    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 12:02:23.537365 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534751    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 12:02:23.537365 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534755    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 12:02:23.537365 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534758    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 12:02:23.537365 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534763    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 12:02:23.537365 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534766    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 12:02:23.537365 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534770    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 12:02:23.537365 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534774    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 12:02:23.537365 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534779    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 12:02:23.537365 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534784    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 12:02:23.537365 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534788    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 12:02:23.537365 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534792    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 12:02:23.537856 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534796    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 12:02:23.537856 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534799    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 12:02:23.537856 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534803    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 12:02:23.537856 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534807    2555 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 12:02:23.537856 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534811    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 12:02:23.537856 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534815    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 12:02:23.537856 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534818    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 12:02:23.537856 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534822    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 12:02:23.537856 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534826    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 12:02:23.537856 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534829    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 12:02:23.537856 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534833    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 12:02:23.537856 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534837    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 12:02:23.537856 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534841    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 12:02:23.537856 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534845    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 12:02:23.537856 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534849    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 12:02:23.537856 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534854    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 12:02:23.537856 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534858    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 12:02:23.537856 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534862    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 12:02:23.537856 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534866    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 12:02:23.537856 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534869    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 12:02:23.538355 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534873    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 12:02:23.538355 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534877    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 12:02:23.538355 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534880    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 12:02:23.538355 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534885    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 12:02:23.538355 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534889    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 12:02:23.538355 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534893    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 12:02:23.538355 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534897    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 12:02:23.538355 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534900    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 12:02:23.538355 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534904    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 12:02:23.538355 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534909    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 12:02:23.538355 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534913    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 12:02:23.538355 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534917    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 12:02:23.538355 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534921    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 12:02:23.538355 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534925    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 12:02:23.538355 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534928    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 12:02:23.538355 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534932    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 12:02:23.538355 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534936    2555 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 12:02:23.538355 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534940    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 12:02:23.538355 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534943    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 12:02:23.538355 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534947    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 12:02:23.538959 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534950    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 12:02:23.538959 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534954    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 12:02:23.538959 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534958    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 12:02:23.538959 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534961    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 12:02:23.538959 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534965    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 12:02:23.538959 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534969    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 12:02:23.538959 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534973    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 12:02:23.538959 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534978    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 12:02:23.538959 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534981    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 12:02:23.538959 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534985    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 12:02:23.538959 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534988    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 12:02:23.538959 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534992    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 12:02:23.538959 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.534995    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 12:02:23.538959 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.535004    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 12:02:23.538959 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.535009    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 12:02:23.538959 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.535013    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 12:02:23.538959 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.535017    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 12:02:23.538959 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.535023    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 12:02:23.538959 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.535028    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 12:02:23.539432 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.535033    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 12:02:23.539432 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.535039    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 12:02:23.539432 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.535044    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 12:02:23.539432 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.535048    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 12:02:23.539432 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.535053    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 12:02:23.539432 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.535057    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 12:02:23.539432 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.535061    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 12:02:23.539432 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.535065    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 12:02:23.539432 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.535069    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 12:02:23.539432 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.535073    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 12:02:23.539432 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.535077    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 12:02:23.539432 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.535081    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 12:02:23.539432 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.535085    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 12:02:23.539432 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.535089    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 12:02:23.539432 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.535092    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 12:02:23.539432 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.535096    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 12:02:23.539432 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.535100    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 12:02:23.539432 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.535104    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 12:02:23.539432 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.535107    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 12:02:23.539893 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.535111    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 12:02:23.539893 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.535115    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 12:02:23.539893 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535213    2555 flags.go:64] FLAG: --address="0.0.0.0"
Sep 15 12:02:23.539893 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535275    2555 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 15 12:02:23.539893 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535284    2555 flags.go:64] FLAG: --anonymous-auth="true"
Sep 15 12:02:23.539893 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535291    2555 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 15 12:02:23.539893 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535297    2555 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 15 12:02:23.539893 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535302    2555 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 15 12:02:23.539893 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535308    2555 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 15 12:02:23.539893 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535315    2555 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 15 12:02:23.539893 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535319    2555 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 15 12:02:23.539893 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535324    2555 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 15 12:02:23.539893 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535329    2555 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 15 12:02:23.539893 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535333    2555 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 15 12:02:23.539893 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535339    2555 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 15 12:02:23.539893 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535343    2555 flags.go:64] FLAG: --cgroup-root=""
Sep 15 12:02:23.539893 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535347    2555 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 15 12:02:23.539893 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535351    2555 flags.go:64] FLAG: --client-ca-file=""
Sep 15 12:02:23.539893 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535357    2555 flags.go:64] FLAG: --cloud-provider="external"
Sep 15 12:02:23.539893 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535362    2555 flags.go:64] FLAG: --cluster-dns="[]"
Sep 15 12:02:23.539893 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535369    2555 flags.go:64] FLAG: --cluster-domain=""
Sep 15 12:02:23.539893 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535373    2555 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 15 12:02:23.539893 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535378    2555 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 15 12:02:23.540461 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535382    2555 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 15 12:02:23.540461 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535387    2555 flags.go:64] FLAG: --container-log-max-files="5"
Sep 15 12:02:23.540461 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535393    2555 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 15 12:02:23.540461 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535398    2555 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 15 12:02:23.540461 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535403    2555 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 15 12:02:23.540461 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535408    2555 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 15 12:02:23.540461 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535412    2555 flags.go:64] FLAG: --contention-profiling="false"
Sep 15 12:02:23.540461 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535417    2555 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 15 12:02:23.540461 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535421    2555 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 15 12:02:23.540461 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535428    2555 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 15 12:02:23.540461 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535433    2555 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 15 12:02:23.540461 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535439    2555 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 15 12:02:23.540461 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535443    2555 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 15 12:02:23.540461 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535447    2555 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 15 12:02:23.540461 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535452    2555 flags.go:64] FLAG: --enable-load-reader="false"
Sep 15 12:02:23.540461 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535457    2555 flags.go:64] FLAG: --enable-server="true"
Sep 15 12:02:23.540461 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535461    2555 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 15 12:02:23.540461 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535468    2555 flags.go:64] FLAG: --event-burst="100"
Sep 15 12:02:23.540461 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535473    2555 flags.go:64] FLAG: --event-qps="50"
Sep 15 12:02:23.540461 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535477    2555 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 15 12:02:23.540461 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535481    2555 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 15 12:02:23.540461 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535486    2555 flags.go:64] FLAG: --eviction-hard=""
Sep 15 12:02:23.540461 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535491    2555 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 15 12:02:23.540461 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535495    2555 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 15 12:02:23.541044 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535500    2555 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 15 12:02:23.541044 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535504    2555 flags.go:64] FLAG: --eviction-soft=""
Sep 15 12:02:23.541044 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535508    2555 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 15 12:02:23.541044 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535513    2555 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 15 12:02:23.541044 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535518    2555 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 15 12:02:23.541044 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535523    2555 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 15 12:02:23.541044 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535527    2555 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 15 12:02:23.541044 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535531    2555 flags.go:64] FLAG: --fail-swap-on="true"
Sep 15 12:02:23.541044 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535535    2555 flags.go:64] FLAG: --feature-gates=""
Sep 15 12:02:23.541044 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535542    2555 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 15 12:02:23.541044 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535546    2555 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 15 12:02:23.541044 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535551    2555 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 15 12:02:23.541044 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535555    2555 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 15 12:02:23.541044 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535560    2555 flags.go:64] FLAG: --healthz-port="10248"
Sep 15 12:02:23.541044 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535564    2555 flags.go:64] FLAG: --help="false"
Sep 15 12:02:23.541044 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535568    2555 flags.go:64] FLAG: --hostname-override="ip-10-0-140-17.ec2.internal"
Sep 15 12:02:23.541044 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535573    2555 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 15 12:02:23.541044 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535579    2555 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 15 12:02:23.541044 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535584    2555 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 15 12:02:23.541044 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535590    2555 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 15 12:02:23.541044 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535595    2555 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 15 12:02:23.541044 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535599    2555 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 15 12:02:23.541044 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535604    2555 flags.go:64] FLAG: --image-service-endpoint=""
Sep 15 12:02:23.541648 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535608    2555 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 15 12:02:23.541648 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535612    2555 flags.go:64] FLAG: --kube-api-burst="100"
Sep 15 12:02:23.541648 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535616    2555 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 15 12:02:23.541648 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535621    2555 flags.go:64] FLAG: --kube-api-qps="50"
Sep 15 12:02:23.541648 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535625    2555 flags.go:64] FLAG: --kube-reserved=""
Sep 15 12:02:23.541648 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535630    2555 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 15 12:02:23.541648 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535634    2555 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 15 12:02:23.541648 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535639    2555 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 15 12:02:23.541648 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535643    2555 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 15 12:02:23.541648 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535647    2555 flags.go:64] FLAG: --lock-file=""
Sep 15 12:02:23.541648 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535652    2555 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 15 12:02:23.541648 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535656    2555 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 15 12:02:23.541648 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535660    2555 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 15 12:02:23.541648 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535668    2555 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 15 12:02:23.541648 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535674    2555 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 15 12:02:23.541648 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535678    2555 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 15 12:02:23.541648 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535682    2555 flags.go:64] FLAG: --logging-format="text"
Sep 15 12:02:23.541648 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535687    2555 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 15 12:02:23.541648 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535692    2555 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 15 12:02:23.541648 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535696    2555 flags.go:64] FLAG: --manifest-url=""
Sep 15 12:02:23.541648 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535701    2555 flags.go:64] FLAG: --manifest-url-header=""
Sep 15 12:02:23.541648 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535707    2555 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 15 12:02:23.541648 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535711    2555 flags.go:64] FLAG: --max-open-files="1000000"
Sep 15 12:02:23.541648 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535718    2555 flags.go:64] FLAG: --max-pods="110"
Sep 15 12:02:23.541648 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535722    2555 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 15 12:02:23.542287 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535727    2555 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 15 12:02:23.542287 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535737    2555 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 15 12:02:23.542287 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535741    2555 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 15 12:02:23.542287 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535745    2555 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 15 12:02:23.542287 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535749    2555 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 15 12:02:23.542287 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535754    2555 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 15 12:02:23.542287 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535769    2555 flags.go:64] FLAG: --node-status-max-images="50"
Sep 15 12:02:23.542287 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535774    2555 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 15 12:02:23.542287 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535779    2555 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 15 12:02:23.542287 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535783    2555 flags.go:64] FLAG: --pod-cidr=""
Sep 15 12:02:23.542287 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535788    2555 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 15 12:02:23.542287 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535796    2555 flags.go:64] FLAG: --pod-manifest-path=""
Sep 15 12:02:23.542287 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535800    2555 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 15 12:02:23.542287 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535805    2555 flags.go:64] FLAG: --pods-per-core="0"
Sep 15 12:02:23.542287 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535809    2555 flags.go:64] FLAG: --port="10250"
Sep 15 12:02:23.542287 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535817    2555 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 15 12:02:23.542287 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535821    2555 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0932d2b0eabc18848"
Sep 15 12:02:23.542287 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535826    2555 flags.go:64] FLAG: --qos-reserved=""
Sep 15 12:02:23.542287 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535831    2555 flags.go:64] FLAG: --read-only-port="10255"
Sep 15 12:02:23.542287 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535836    2555 flags.go:64] FLAG: --register-node="true"
Sep 15 12:02:23.542287 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535840    2555 flags.go:64] FLAG: --register-with-taints=""
Sep 15 12:02:23.542287 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535845    2555 flags.go:64] FLAG: --registry-burst="10"
Sep 15 12:02:23.542287 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535850    2555 flags.go:64] FLAG: --registry-qps="5"
Sep 15 12:02:23.542287 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535855    2555 flags.go:64] FLAG: --reserved-cpus=""
Sep 15 12:02:23.542879 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535859    2555 flags.go:64] FLAG: --reserved-memory=""
Sep 15 12:02:23.542879 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535865    2555 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 15 12:02:23.542879 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535869    2555 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 15 12:02:23.542879 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535874    2555 flags.go:64] FLAG: --rotate-certificates="false"
Sep 15 12:02:23.542879 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535879    2555 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 15 12:02:23.542879 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535883    2555 flags.go:64] FLAG: --runonce="false"
Sep 15 12:02:23.542879 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535887    2555 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 15 12:02:23.542879 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535892    2555 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 15 12:02:23.542879 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535896    2555 flags.go:64] FLAG: --seccomp-default="false"
Sep 15 12:02:23.542879 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535903    2555 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 15 12:02:23.542879 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535907    2555 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 15 12:02:23.542879 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535912    2555 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 15 12:02:23.542879 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535916    2555 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 15 12:02:23.542879 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535920    2555 flags.go:64] FLAG: --storage-driver-password="root"
Sep 15 12:02:23.542879 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535925    2555 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 15 12:02:23.542879 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535929    2555 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 15 12:02:23.542879 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535933    2555 flags.go:64] FLAG: --storage-driver-user="root"
Sep 15 12:02:23.542879 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535937    2555 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 15 12:02:23.542879 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535942    2555 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 15 12:02:23.542879 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535947    2555 flags.go:64] FLAG: --system-cgroups=""
Sep 15 12:02:23.542879 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535951    2555 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 15 12:02:23.542879 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535958    2555 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 15 12:02:23.542879 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535963    2555 flags.go:64] FLAG: --tls-cert-file=""
Sep 15 12:02:23.542879 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535968    2555 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 15 12:02:23.542879 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535975    2555 flags.go:64] FLAG: --tls-min-version=""
Sep 15 12:02:23.543520 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535979    2555 flags.go:64] FLAG: --tls-private-key-file=""
Sep 15 12:02:23.543520 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535983    2555 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 15 12:02:23.543520 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535988    2555 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 15 12:02:23.543520 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535992    2555 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 15 12:02:23.543520 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.535997    2555 flags.go:64] FLAG: --v="2"
Sep 15 12:02:23.543520 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.536003    2555 flags.go:64] FLAG: --version="false"
Sep 15 12:02:23.543520 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.536010    2555 flags.go:64] FLAG: --vmodule=""
Sep 15 12:02:23.543520 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.536016    2555 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 15 12:02:23.543520 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.536021    2555 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 15 12:02:23.543520 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.536184    2555 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 12:02:23.543520 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.536190    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 12:02:23.543520 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.536194    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 12:02:23.543520 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.536199    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 12:02:23.543520 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.536203    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 12:02:23.543520 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.536207    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 12:02:23.543520 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.536211    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 12:02:23.543520 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.536217    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 12:02:23.543520 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.536221    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 12:02:23.543520 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.536225    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 12:02:23.543520 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.536230    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 12:02:23.543520 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.536234    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 12:02:23.543520 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.536238    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 12:02:23.544125 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.536242    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 12:02:23.544125 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.536246    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 12:02:23.544125 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.536250    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 12:02:23.544125 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.536254    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 12:02:23.544125 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.536271    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 12:02:23.544125 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.536276    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 12:02:23.544125 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.536280    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 12:02:23.544125 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.536284    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 12:02:23.544125 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.536290    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 12:02:23.544125 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.536294    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 12:02:23.544125 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.536297    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 12:02:23.544125 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.536301    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 12:02:23.544125 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.536305    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 12:02:23.544125 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.536310    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 12:02:23.544125 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.536313    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 12:02:23.544125 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.536317    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 12:02:23.544125 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.536321    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 12:02:23.544125 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.536326    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 12:02:23.544125 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.536329    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 12:02:23.544125 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.536333    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 12:02:23.544640 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.536337    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 12:02:23.544640 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.536341    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 12:02:23.544640 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.536345    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 12:02:23.544640 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.536349    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 12:02:23.544640 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.536352    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 12:02:23.544640 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.536356    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 12:02:23.544640 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.536361    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 12:02:23.544640 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.536366    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 12:02:23.544640 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.536370    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 12:02:23.544640 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.536374    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 12:02:23.544640 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.536377    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 12:02:23.544640 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.536381    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 12:02:23.544640 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.536386    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 12:02:23.544640 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.536390    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 12:02:23.544640 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.536394    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 12:02:23.544640 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.536399    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 12:02:23.544640 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.536403    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 12:02:23.544640 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.536406    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 12:02:23.544640 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.536410    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 12:02:23.545121 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.536414    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 12:02:23.545121 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.536420    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 12:02:23.545121 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.536423    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 12:02:23.545121 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.536427    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 12:02:23.545121 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.536431    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 12:02:23.545121 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.536435    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 12:02:23.545121 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.536439    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 12:02:23.545121 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.536443    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 12:02:23.545121 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.536449    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 12:02:23.545121 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.536454    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 12:02:23.545121 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.536458    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 12:02:23.545121 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.536463    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 12:02:23.545121 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.536466    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 12:02:23.545121 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.536470    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 12:02:23.545121 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.536473    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 12:02:23.545121 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.536477    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 12:02:23.545121 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.536481    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 12:02:23.545121 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.536485    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 12:02:23.545121 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.536488    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 12:02:23.545121 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.536493    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 12:02:23.545664 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.536497    2555 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 12:02:23.545664 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.536502    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 12:02:23.545664 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.536505    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 12:02:23.545664 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.536508    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 12:02:23.545664 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.536512    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 12:02:23.545664 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.536518    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 12:02:23.545664 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.536523    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 12:02:23.545664 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.536528    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 12:02:23.545664 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.536533    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 12:02:23.545664 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.536537    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 12:02:23.545664 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.536541    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 12:02:23.545664 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.536546    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 12:02:23.545664 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.536550    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 12:02:23.545664 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.536556    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 12:02:23.545664 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.536560    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 12:02:23.545664 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.536564    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 12:02:23.545664 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.536567    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 12:02:23.545664 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.536571    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 12:02:23.545664 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.536575    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 12:02:23.546127 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.536579    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 12:02:23.546127 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.536583    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 12:02:23.546127 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.536587    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 12:02:23.546127 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.536591    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 12:02:23.546127 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.536595    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 12:02:23.546127 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.536599    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 12:02:23.546127 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.536603    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 12:02:23.546127 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.536607    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 12:02:23.546127 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.537370    2555 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 15 12:02:23.547031 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.547004    2555 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 15 12:02:23.547065 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.547033    2555 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 15 12:02:23.547065 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.547051    2555 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 15 12:02:23.547065 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.547059    2555 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 15 12:02:23.547145 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547139    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 12:02:23.547174 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547146    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 12:02:23.547174 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547150    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 12:02:23.547174 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547155    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 12:02:23.547174 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547160    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 12:02:23.547174 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547164    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 12:02:23.547174 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547168    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 12:02:23.547174 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547171    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 12:02:23.547174 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547175    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 12:02:23.547382 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547179    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 12:02:23.547382 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547184    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 12:02:23.547382 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547188    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 12:02:23.547382 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547191    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 12:02:23.547382 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547196    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 12:02:23.547382 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547200    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 12:02:23.547382 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547204    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 12:02:23.547382 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547208    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 12:02:23.547382 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547211    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 12:02:23.547382 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547216    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 12:02:23.547382 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547219    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 12:02:23.547382 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547223    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 12:02:23.547382 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547226    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 12:02:23.547382 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547230    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 12:02:23.547382 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547234    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 12:02:23.547382 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547237    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 12:02:23.547382 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547241    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 12:02:23.547382 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547244    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 12:02:23.547382 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547248    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 12:02:23.547382 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547252    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 12:02:23.547875 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547256    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 12:02:23.547875 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547279    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 12:02:23.547875 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547283    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 12:02:23.547875 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547287    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 12:02:23.547875 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547290    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 12:02:23.547875 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547294    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 12:02:23.547875 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547298    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 12:02:23.547875 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547302    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 12:02:23.547875 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547305    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 12:02:23.547875 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547309    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 12:02:23.547875 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547313    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 12:02:23.547875 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547317    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 12:02:23.547875 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547320    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 12:02:23.547875 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547326    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 12:02:23.547875 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547330    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 12:02:23.547875 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547334    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 12:02:23.547875 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547338    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 12:02:23.547875 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547342    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 12:02:23.547875 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547346    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 12:02:23.548440 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547349    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 12:02:23.548440 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547352    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 12:02:23.548440 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547357    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 12:02:23.548440 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547361    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 12:02:23.548440 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547387    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 12:02:23.548440 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547392    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 12:02:23.548440 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547397    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 12:02:23.548440 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547402    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 12:02:23.548440 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547406    2555 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 12:02:23.548440 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547410    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 12:02:23.548440 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547415    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 12:02:23.548440 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547420    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 12:02:23.548440 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547424    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 12:02:23.548440 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547428    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 12:02:23.548440 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547432    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 12:02:23.548440 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547436    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 12:02:23.548440 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547440    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 12:02:23.548440 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547444    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 12:02:23.548440 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547448    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 12:02:23.548440 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547452    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 12:02:23.548948 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547456    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 12:02:23.548948 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547459    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 12:02:23.548948 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547463    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 12:02:23.548948 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547466    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 12:02:23.548948 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547470    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 12:02:23.548948 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547476    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 12:02:23.548948 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547484    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 12:02:23.548948 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547489    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 12:02:23.548948 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547493    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 12:02:23.548948 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547498    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 12:02:23.548948 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547502    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 12:02:23.548948 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547506    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 12:02:23.548948 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547510    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 12:02:23.548948 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547513    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 12:02:23.548948 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547518    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 12:02:23.548948 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547522    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 12:02:23.548948 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547526    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 12:02:23.548948 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547529    2555 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 12:02:23.548948 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547533    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 12:02:23.549419 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547538    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 12:02:23.549419 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547543    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 12:02:23.549419 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547547    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 12:02:23.549419 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547551    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 12:02:23.549419 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547555    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 12:02:23.549419 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547559    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 12:02:23.549419 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547562    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 12:02:23.549419 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547566    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 12:02:23.549419 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547571    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 12:02:23.549419 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547575    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 12:02:23.549419 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547578    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 12:02:23.549419 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547582    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 12:02:23.549419 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.547589    2555 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 15 12:02:23.549419 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547769    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 12:02:23.549419 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547776    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 12:02:23.549824 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547780    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 12:02:23.549824 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547786    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 12:02:23.549824 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547792    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 12:02:23.549824 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547796    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 12:02:23.549824 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547802    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 12:02:23.549824 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547806    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 12:02:23.549824 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547810    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 12:02:23.549824 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547815    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 12:02:23.549824 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547819    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 12:02:23.549824 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547823    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 12:02:23.549824 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547826    2555 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 12:02:23.549824 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547830    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 12:02:23.549824 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547835    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 12:02:23.549824 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547839    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 12:02:23.549824 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547842    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 12:02:23.549824 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547847    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 12:02:23.549824 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547851    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 12:02:23.549824 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547855    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 12:02:23.549824 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547859    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 12:02:23.550290 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547863    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 12:02:23.550290 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547867    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 12:02:23.550290 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547870    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 12:02:23.550290 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547874    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 12:02:23.550290 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547877    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 12:02:23.550290 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547882    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 12:02:23.550290 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547886    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 12:02:23.550290 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547890    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 12:02:23.550290 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547894    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 12:02:23.550290 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547898    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 12:02:23.550290 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547902    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 12:02:23.550290 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547906    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 12:02:23.550290 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547910    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 12:02:23.550290 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547913    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 12:02:23.550290 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547917    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 12:02:23.550290 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547921    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 12:02:23.550290 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547924    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 12:02:23.550290 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547928    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 12:02:23.550290 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547933    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 12:02:23.550290 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547936    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 12:02:23.550789 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547940    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 12:02:23.550789 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547944    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 12:02:23.550789 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547948    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 12:02:23.550789 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547951    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 12:02:23.550789 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547955    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 12:02:23.550789 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547958    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 12:02:23.550789 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547962    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 12:02:23.550789 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547966    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 12:02:23.550789 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547970    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 12:02:23.550789 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547974    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 12:02:23.550789 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547977    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 12:02:23.550789 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547981    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 12:02:23.550789 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547986    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 12:02:23.550789 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547991    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 12:02:23.550789 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.547996    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 12:02:23.550789 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.548000    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 12:02:23.550789 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.548003    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 12:02:23.550789 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.548007    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 12:02:23.550789 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.548011    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 12:02:23.551256 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.548015    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 12:02:23.551256 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.548021    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 12:02:23.551256 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.548026    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 12:02:23.551256 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.548031    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 12:02:23.551256 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.548036    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 12:02:23.551256 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.548040    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 12:02:23.551256 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.548045    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 12:02:23.551256 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.548049    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 12:02:23.551256 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.548052    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 12:02:23.551256 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.548055    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 12:02:23.551256 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.548060    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 12:02:23.551256 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.548065    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 12:02:23.551256 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.548069    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 12:02:23.551256 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.548073    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 12:02:23.551256 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.548076    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 12:02:23.551256 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.548080    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 12:02:23.551256 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.548084    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 12:02:23.551256 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.548087    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 12:02:23.551256 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.548091    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 12:02:23.551722 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.548095    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 12:02:23.551722 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.548099    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 12:02:23.551722 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.548103    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 12:02:23.551722 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.548106    2555 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 12:02:23.551722 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.548110    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 12:02:23.551722 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.548114    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 12:02:23.551722 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.548118    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 12:02:23.551722 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.548121    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 12:02:23.551722 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.548125    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 12:02:23.551722 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.548129    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 12:02:23.551722 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.548133    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 12:02:23.551722 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.548136    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 12:02:23.551722 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.548140    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 12:02:23.551722 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.548144    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 12:02:23.551722 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.548149    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 12:02:23.551722 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.548153    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 12:02:23.551722 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.548156    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 12:02:23.551722 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.548160    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 12:02:23.551722 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.548164    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 12:02:23.551722 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:23.548168    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 12:02:23.552203 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.548175    2555 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 15 12:02:23.552203 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.548375    2555 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 15 12:02:23.637688 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.637654    2555 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 15 12:02:23.638876 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.638863    2555 server.go:1019] "Starting client certificate rotation"
Sep 15 12:02:23.638975 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.638955    2555 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 15 12:02:23.639016 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.638995    2555 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 15 12:02:23.641733 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.641718    2555 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 15 12:02:23.650730 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.650702    2555 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 15 12:02:23.661705 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.661671    2555 log.go:25] "Validated CRI v1 runtime API"
Sep 15 12:02:23.665704 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.665676    2555 log.go:25] "Validated CRI v1 image API"
Sep 15 12:02:23.666637 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.666607    2555 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 15 12:02:23.670783 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.670763    2555 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 15 12:02:23.670857 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.670818    2555 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 e85c7a02-794f-4060-993e-e3f2080d486d:/dev/nvme0n1p4 f5429cfd-bd30-426e-8a0f-e882c4ce6a32:/dev/nvme0n1p3]
Sep 15 12:02:23.670898 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.670854    2555 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 15 12:02:23.676238 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.676116    2555 manager.go:216] Machine: {Timestamp:2026-09-15 12:02:23.674640161 +0000 UTC m=+0.465562674 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3149543 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28f62f559c63902896cd92c0d0a694 SystemUUID:ec28f62f-559c-6390-2896-cd92c0d0a694 BootID:5e48b1ce-9c0e-4eae-961c-63d38930b824 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d8:01:7c:ff Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d8:01:7c:ff Speed:0 Mtu:9001} {Name:ovs-system MacAddress:5a:e0:fc:67:50:13 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 15 12:02:23.676238 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.676228    2555 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 15 12:02:23.676372 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.676336    2555 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 15 12:02:23.680094 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.680066    2555 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 15 12:02:23.680249 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.680097    2555 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-17.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 15 12:02:23.680981 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.680970    2555 topology_manager.go:138] "Creating topology manager with none policy"
Sep 15 12:02:23.681022 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.680983    2555 container_manager_linux.go:309] "Creating device plugin manager"
Sep 15 12:02:23.681022 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.680996    2555 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 15 12:02:23.681824 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.681814    2555 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 15 12:02:23.683195 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.683185    2555 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 15 12:02:23.687902 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.687890    2555 state_mem.go:36] "Initialized new in-memory state store"
Sep 15 12:02:23.688031 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.688022    2555 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 15 12:02:23.688067 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.688053    2555 kubelet.go:486] "Attempting to sync node with API server"
Sep 15 12:02:23.688067 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.688063    2555 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 15 12:02:23.688124 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.688076    2555 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 15 12:02:23.688124 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.688087    2555 kubelet.go:398] "Adding apiserver pod source"
Sep 15 12:02:23.688124 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.688096    2555 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 15 12:02:23.688472 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.688449    2555 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 15 12:02:23.688472 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.688473    2555 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 15 12:02:23.690959 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.690941    2555 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 15 12:02:23.693214 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.693184    2555 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nfc6h"
Sep 15 12:02:23.693346 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.693333    2555 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 15 12:02:23.693415 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.693374    2555 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 15 12:02:23.695436 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.695422    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 15 12:02:23.695436 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.695443    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 15 12:02:23.695558 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.695452    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 15 12:02:23.695558 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.695460    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 15 12:02:23.695558 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.695472    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 15 12:02:23.695558 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.695480    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 15 12:02:23.695558 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.695499    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 15 12:02:23.695558 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.695508    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 15 12:02:23.695757 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.695561    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 15 12:02:23.695757 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.695573    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 15 12:02:23.695757 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.695586    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 15 12:02:23.695757 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.695595    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 15 12:02:23.696487 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.696476    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 15 12:02:23.696548 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.696489    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 15 12:02:23.700399 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.700379    2555 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nfc6h"
Sep 15 12:02:23.700502 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.700489    2555 server.go:1268] "Started kubelet"
Sep 15 12:02:23.700613 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.700583    2555 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 15 12:02:23.700718 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.700623    2555 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 15 12:02:23.700781 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.700730    2555 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 15 12:02:23.700971 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.700954    2555 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-17.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 15 12:02:23.701047 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:23.701028    2555 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-17.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 15 12:02:23.701205 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:23.701184    2555 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 15 12:02:23.701491 ip-10-0-140-17 systemd[1]: Started Kubernetes Kubelet.
Sep 15 12:02:23.706762 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.706729    2555 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 15 12:02:23.706860 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.706749    2555 server.go:310] "Adding debug handlers to kubelet server"
Sep 15 12:02:23.707547 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:23.707500    2555 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 15 12:02:23.711656 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:23.710510    2555 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-17.ec2.internal.18d57ce38c2771b3  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-17.ec2.internal,UID:ip-10-0-140-17.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-17.ec2.internal,},FirstTimestamp:2026-09-15 12:02:23.700455859 +0000 UTC m=+0.491378365,LastTimestamp:2026-09-15 12:02:23.700455859 +0000 UTC m=+0.491378365,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-17.ec2.internal,}"
Sep 15 12:02:23.715343 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.715313    2555 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 15 12:02:23.716007 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.715992    2555 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 15 12:02:23.716111 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:23.716095    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-17.ec2.internal\" not found"
Sep 15 12:02:23.716999 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.716980    2555 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 15 12:02:23.717125 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.716992    2555 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 15 12:02:23.717188 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.717128    2555 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 15 12:02:23.717188 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.717163    2555 factory.go:153] Registering CRI-O factory
Sep 15 12:02:23.717188 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.717177    2555 factory.go:223] Registration of the crio container factory successfully
Sep 15 12:02:23.717285 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.717232    2555 reconstruct.go:97] "Volume reconstruction finished"
Sep 15 12:02:23.717285 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.717239    2555 reconciler.go:29] "Reconciler: start to sync state"
Sep 15 12:02:23.717285 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.717237    2555 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 15 12:02:23.717285 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.717250    2555 factory.go:55] Registering systemd factory
Sep 15 12:02:23.717285 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.717257    2555 factory.go:223] Registration of the systemd container factory successfully
Sep 15 12:02:23.717285 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.717289    2555 factory.go:103] Registering Raw factory
Sep 15 12:02:23.717562 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.717299    2555 manager.go:1196] Started watching for new ooms in manager
Sep 15 12:02:23.717688 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.717676    2555 manager.go:349] Starting recovery of all containers
Sep 15 12:02:23.718587 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.718564    2555 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 12:02:23.720832 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:23.720633    2555 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-140-17.ec2.internal\" not found" node="ip-10-0-140-17.ec2.internal"
Sep 15 12:02:23.728614 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.728596    2555 manager.go:354] Recovery completed
Sep 15 12:02:23.732606 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.732592    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 12:02:23.734542 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.734525    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-17.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 12:02:23.734600 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.734558    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-17.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 12:02:23.734600 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.734572    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-17.ec2.internal" event="NodeHasSufficientPID"
Sep 15 12:02:23.735142 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.735128    2555 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 15 12:02:23.735142 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.735140    2555 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 15 12:02:23.735241 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.735158    2555 state_mem.go:36] "Initialized new in-memory state store"
Sep 15 12:02:23.735241 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.735160    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 12:02:23.735991 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.735972    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-17.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 12:02:23.736049 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.736001    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-17.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 12:02:23.736049 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.736010    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-17.ec2.internal" event="NodeHasSufficientPID"
Sep 15 12:02:23.737365 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.737351    2555 policy_none.go:49] "None policy: Start"
Sep 15 12:02:23.737424 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.737369    2555 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 15 12:02:23.737424 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.737379    2555 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 15 12:02:23.739405 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.739392    2555 policy_none.go:47] "Start"
Sep 15 12:02:23.782050 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.781966    2555 manager.go:337] "Starting Device Plugin manager"
Sep 15 12:02:23.782050 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:23.782005    2555 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 15 12:02:23.782050 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.782015    2555 server.go:87] "Starting device plugin registration server"
Sep 15 12:02:23.794110 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.782418    2555 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 15 12:02:23.794110 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.782429    2555 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 15 12:02:23.794110 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.782534    2555 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 15 12:02:23.794110 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.782898    2555 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 15 12:02:23.794110 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.782913    2555 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 15 12:02:23.794110 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:23.785300    2555 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 15 12:02:23.794110 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:23.785352    2555 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-17.ec2.internal\" not found"
Sep 15 12:02:23.809151 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.809122    2555 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 15 12:02:23.810396 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.810379    2555 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 15 12:02:23.810468 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.810441    2555 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 15 12:02:23.810528 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.810469    2555 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 15 12:02:23.810528 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:23.810521    2555 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 15 12:02:23.812172 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.812149    2555 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 12:02:23.883410 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.883381    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 12:02:23.884496 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.884480    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-17.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 12:02:23.884564 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.884513    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-17.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 12:02:23.884564 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.884524    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-17.ec2.internal" event="NodeHasSufficientPID"
Sep 15 12:02:23.884564 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.884547    2555 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-17.ec2.internal"
Sep 15 12:02:23.903729 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.903705    2555 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-17.ec2.internal"
Sep 15 12:02:23.903729 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:23.903732    2555 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-17.ec2.internal\": node \"ip-10-0-140-17.ec2.internal\" not found"
Sep 15 12:02:23.911175 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.911154    2555 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-17.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-17.ec2.internal"]
Sep 15 12:02:23.911234 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.911222    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 12:02:23.912767 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.912743    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-17.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 12:02:23.912868 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.912782    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-17.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 12:02:23.912868 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.912798    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-17.ec2.internal" event="NodeHasSufficientPID"
Sep 15 12:02:23.915130 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.915116    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 12:02:23.915247 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.915230    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-17.ec2.internal"
Sep 15 12:02:23.915313 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.915275    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 12:02:23.915901 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.915885    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-17.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 12:02:23.915901 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.915895    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-17.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 12:02:23.916031 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.915913    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-17.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 12:02:23.916031 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.915921    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-17.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 12:02:23.916031 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.915939    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-17.ec2.internal" event="NodeHasSufficientPID"
Sep 15 12:02:23.916031 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.915923    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-17.ec2.internal" event="NodeHasSufficientPID"
Sep 15 12:02:23.918283 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.918256    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-17.ec2.internal"
Sep 15 12:02:23.918360 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.918292    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 12:02:23.918419 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.918400    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/99788892a4247a0a40d9fb25d8f545f5-config\") pod \"kube-apiserver-proxy-ip-10-0-140-17.ec2.internal\" (UID: \"99788892a4247a0a40d9fb25d8f545f5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-17.ec2.internal"
Sep 15 12:02:23.918466 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.918435    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/81882ce609db62ac71292b6338f62ad1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-17.ec2.internal\" (UID: \"81882ce609db62ac71292b6338f62ad1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-17.ec2.internal"
Sep 15 12:02:23.918466 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.918454    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/81882ce609db62ac71292b6338f62ad1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-17.ec2.internal\" (UID: \"81882ce609db62ac71292b6338f62ad1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-17.ec2.internal"
Sep 15 12:02:23.919100 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.919083    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-17.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 12:02:23.919176 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.919109    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-17.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 12:02:23.919176 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:23.919122    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-17.ec2.internal" event="NodeHasSufficientPID"
Sep 15 12:02:23.921029 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:23.921014    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-17.ec2.internal\" not found"
Sep 15 12:02:23.941069 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:23.941048    2555 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-17.ec2.internal\" not found" node="ip-10-0-140-17.ec2.internal"
Sep 15 12:02:23.945588 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:23.945570    2555 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-17.ec2.internal\" not found" node="ip-10-0-140-17.ec2.internal"
Sep 15 12:02:24.019301 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:24.019246    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/99788892a4247a0a40d9fb25d8f545f5-config\") pod \"kube-apiserver-proxy-ip-10-0-140-17.ec2.internal\" (UID: \"99788892a4247a0a40d9fb25d8f545f5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-17.ec2.internal"
Sep 15 12:02:24.019514 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:24.019303    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/99788892a4247a0a40d9fb25d8f545f5-config\") pod \"kube-apiserver-proxy-ip-10-0-140-17.ec2.internal\" (UID: \"99788892a4247a0a40d9fb25d8f545f5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-17.ec2.internal"
Sep 15 12:02:24.019514 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:24.019349    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/81882ce609db62ac71292b6338f62ad1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-17.ec2.internal\" (UID: \"81882ce609db62ac71292b6338f62ad1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-17.ec2.internal"
Sep 15 12:02:24.019514 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:24.019311    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/81882ce609db62ac71292b6338f62ad1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-17.ec2.internal\" (UID: \"81882ce609db62ac71292b6338f62ad1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-17.ec2.internal"
Sep 15 12:02:24.019514 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:24.019391    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/81882ce609db62ac71292b6338f62ad1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-17.ec2.internal\" (UID: \"81882ce609db62ac71292b6338f62ad1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-17.ec2.internal"
Sep 15 12:02:24.019514 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:24.019427    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/81882ce609db62ac71292b6338f62ad1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-17.ec2.internal\" (UID: \"81882ce609db62ac71292b6338f62ad1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-17.ec2.internal"
Sep 15 12:02:24.021163 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:24.021136    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-17.ec2.internal\" not found"
Sep 15 12:02:24.122366 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:24.122294    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-17.ec2.internal\" not found"
Sep 15 12:02:24.222412 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:24.222370    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-17.ec2.internal\" not found"
Sep 15 12:02:24.244202 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:24.244164    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-17.ec2.internal"
Sep 15 12:02:24.247898 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:24.247873    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-17.ec2.internal"
Sep 15 12:02:24.323592 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:24.323542    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-17.ec2.internal\" not found"
Sep 15 12:02:24.424114 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:24.424048    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-17.ec2.internal\" not found"
Sep 15 12:02:24.524584 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:24.524531    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-17.ec2.internal\" not found"
Sep 15 12:02:24.575071 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:24.575034    2555 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 12:02:24.624937 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:24.624887    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-17.ec2.internal\" not found"
Sep 15 12:02:24.639094 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:24.639067    2555 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 15 12:02:24.639244 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:24.639217    2555 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 15 12:02:24.639311 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:24.639244    2555 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 15 12:02:24.639311 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:24.639250    2555 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 15 12:02:24.710230 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:24.710138    2555 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-14 11:57:24 +0000 UTC" deadline="2028-06-03 10:43:38.096291122 +0000 UTC"
Sep 15 12:02:24.710230 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:24.710169    2555 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15046h41m13.386125989s"
Sep 15 12:02:24.725345 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:24.725301    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-17.ec2.internal\" not found"
Sep 15 12:02:24.825687 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:24.825652    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-17.ec2.internal\" not found"
Sep 15 12:02:24.827844 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:24.827821    2555 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 12:02:24.916422 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:24.916395    2555 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-17.ec2.internal"
Sep 15 12:02:24.925836 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:24.925800    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81882ce609db62ac71292b6338f62ad1.slice/crio-ddd647f94677071151b9854892a2dec4ae42bd5e7471fb590adb108dca11d594 WatchSource:0}: Error finding container ddd647f94677071151b9854892a2dec4ae42bd5e7471fb590adb108dca11d594: Status 404 returned error can't find the container with id ddd647f94677071151b9854892a2dec4ae42bd5e7471fb590adb108dca11d594
Sep 15 12:02:24.928458 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:24.928429    2555 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 15 12:02:24.929673 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:24.929586    2555 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-17.ec2.internal"
Sep 15 12:02:24.931062 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:24.931048    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 12:02:24.936896 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:24.936876    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99788892a4247a0a40d9fb25d8f545f5.slice/crio-deb47426d6ee0094fdb470b4adeba582c38aae8ff6fb83f72df90273cdace3f1 WatchSource:0}: Error finding container deb47426d6ee0094fdb470b4adeba582c38aae8ff6fb83f72df90273cdace3f1: Status 404 returned error can't find the container with id deb47426d6ee0094fdb470b4adeba582c38aae8ff6fb83f72df90273cdace3f1
Sep 15 12:02:24.941495 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:24.941477    2555 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 15 12:02:25.588409 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.588375    2555 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 12:02:25.603277 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.603225    2555 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 12:02:25.689194 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.688909    2555 apiserver.go:52] "Watching apiserver"
Sep 15 12:02:25.700319 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.699832    2555 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 15 12:02:25.702371 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.702330    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-mkck2","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-17.ec2.internal","openshift-multus/multus-additional-cni-plugins-p4rvp","openshift-network-operator/iptables-alerter-7fbc7","openshift-ovn-kubernetes/ovnkube-node-k62zz","kube-system/global-pull-secret-syncer-g25rd","kube-system/konnectivity-agent-c47md","openshift-cluster-node-tuning-operator/tuned-tnbfp","openshift-multus/multus-2wkff","openshift-multus/network-metrics-daemon-bdjwd","openshift-network-diagnostics/network-check-target-wrxv2","kube-system/kube-apiserver-proxy-ip-10-0-140-17.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qjwgl","openshift-dns/node-resolver-kgjkj"]
Sep 15 12:02:25.705235 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.705210    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-c47md"
Sep 15 12:02:25.709615 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.708681    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qjwgl"
Sep 15 12:02:25.710401 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.709897    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 15 12:02:25.710401 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.709957    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-p4rvp"
Sep 15 12:02:25.710401 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.710138    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 15 12:02:25.711830 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.711806    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-n2b8l\""
Sep 15 12:02:25.712859 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.712830    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 15 12:02:25.717036 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.713182    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 15 12:02:25.717036 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.713773    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-nhc29\""
Sep 15 12:02:25.717036 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.714267    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 15 12:02:25.717036 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.714840    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 15 12:02:25.717036 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.715427    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 15 12:02:25.717036 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.716311    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 15 12:02:25.717426 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.717328    2555 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 15 12:02:25.718981 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.718955    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-sf4bj\""
Sep 15 12:02:25.719084 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.719059    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 15 12:02:25.719353 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.719326    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7fbc7"
Sep 15 12:02:25.719752 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.719712    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-k62zz"
Sep 15 12:02:25.719836 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.719730    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 15 12:02:25.721476 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.721336    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g25rd"
Sep 15 12:02:25.721476 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:25.721439    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g25rd" podUID="07f9ddc4-14d7-4b04-9797-67507d4063bc"
Sep 15 12:02:25.722239 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.722222    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-tftdz\""
Sep 15 12:02:25.724732 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.722363    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 15 12:02:25.724837 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.722509    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 15 12:02:25.724910 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.722528    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 15 12:02:25.724972 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.724235    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 15 12:02:25.725049 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.724304    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 15 12:02:25.725049 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.724325    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 15 12:02:25.725172 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.724394    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 15 12:02:25.725172 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.724492    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 15 12:02:25.725279 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.724554    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 15 12:02:25.725925 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.724433    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-cbhtj\""
Sep 15 12:02:25.727446 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.726951    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2wkff"
Sep 15 12:02:25.729525 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.729467    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 15 12:02:25.730296 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.729758    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mkck2"
Sep 15 12:02:25.730296 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.729876    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-wrvpq\""
Sep 15 12:02:25.733257 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.731427    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0fa9c45a-2974-45d2-82c2-23a63b9f0662-device-dir\") pod \"aws-ebs-csi-driver-node-qjwgl\" (UID: \"0fa9c45a-2974-45d2-82c2-23a63b9f0662\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qjwgl"
Sep 15 12:02:25.733257 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.731469    2555 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 15 12:02:25.733257 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.731479    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0fa9c45a-2974-45d2-82c2-23a63b9f0662-sys-fs\") pod \"aws-ebs-csi-driver-node-qjwgl\" (UID: \"0fa9c45a-2974-45d2-82c2-23a63b9f0662\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qjwgl"
Sep 15 12:02:25.733257 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.731505    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b9b05b4f-57d5-454e-b739-2778ddcab5fb-cni-binary-copy\") pod \"multus-additional-cni-plugins-p4rvp\" (UID: \"b9b05b4f-57d5-454e-b739-2778ddcab5fb\") " pod="openshift-multus/multus-additional-cni-plugins-p4rvp"
Sep 15 12:02:25.733257 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.731529    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b9b05b4f-57d5-454e-b739-2778ddcab5fb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-p4rvp\" (UID: \"b9b05b4f-57d5-454e-b739-2778ddcab5fb\") " pod="openshift-multus/multus-additional-cni-plugins-p4rvp"
Sep 15 12:02:25.733257 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.731554    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/95528bd1-4100-43e9-bc0a-2e56670bc84f-systemd-units\") pod \"ovnkube-node-k62zz\" (UID: \"95528bd1-4100-43e9-bc0a-2e56670bc84f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k62zz"
Sep 15 12:02:25.733257 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.731593    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/95528bd1-4100-43e9-bc0a-2e56670bc84f-host-slash\") pod \"ovnkube-node-k62zz\" (UID: \"95528bd1-4100-43e9-bc0a-2e56670bc84f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k62zz"
Sep 15 12:02:25.733257 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.731618    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/95528bd1-4100-43e9-bc0a-2e56670bc84f-log-socket\") pod \"ovnkube-node-k62zz\" (UID: \"95528bd1-4100-43e9-bc0a-2e56670bc84f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k62zz"
Sep 15 12:02:25.733257 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.731642    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jx9sp\" (UniqueName: \"kubernetes.io/projected/95528bd1-4100-43e9-bc0a-2e56670bc84f-kube-api-access-jx9sp\") pod \"ovnkube-node-k62zz\" (UID: \"95528bd1-4100-43e9-bc0a-2e56670bc84f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k62zz"
Sep 15 12:02:25.733257 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.731666    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/95528bd1-4100-43e9-bc0a-2e56670bc84f-run-ovn\") pod \"ovnkube-node-k62zz\" (UID: \"95528bd1-4100-43e9-bc0a-2e56670bc84f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k62zz"
Sep 15 12:02:25.733257 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.731695    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/95528bd1-4100-43e9-bc0a-2e56670bc84f-host-run-ovn-kubernetes\") pod \"ovnkube-node-k62zz\" (UID: \"95528bd1-4100-43e9-bc0a-2e56670bc84f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k62zz"
Sep 15 12:02:25.733257 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.731727    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/95528bd1-4100-43e9-bc0a-2e56670bc84f-host-kubelet\") pod \"ovnkube-node-k62zz\" (UID: \"95528bd1-4100-43e9-bc0a-2e56670bc84f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k62zz"
Sep 15 12:02:25.733257 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.731764    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/95528bd1-4100-43e9-bc0a-2e56670bc84f-host-run-netns\") pod \"ovnkube-node-k62zz\" (UID: \"95528bd1-4100-43e9-bc0a-2e56670bc84f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k62zz"
Sep 15 12:02:25.733257 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.731795    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/95528bd1-4100-43e9-bc0a-2e56670bc84f-ovnkube-script-lib\") pod \"ovnkube-node-k62zz\" (UID: \"95528bd1-4100-43e9-bc0a-2e56670bc84f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k62zz"
Sep 15 12:02:25.733257 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.731839    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/25dcd72f-7a70-4cd8-ab3c-964f525bd756-host-slash\") pod \"iptables-alerter-7fbc7\" (UID: \"25dcd72f-7a70-4cd8-ab3c-964f525bd756\") " pod="openshift-network-operator/iptables-alerter-7fbc7"
Sep 15 12:02:25.733257 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.731872    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/95528bd1-4100-43e9-bc0a-2e56670bc84f-host-cni-netd\") pod \"ovnkube-node-k62zz\" (UID: \"95528bd1-4100-43e9-bc0a-2e56670bc84f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k62zz"
Sep 15 12:02:25.734142 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.731908    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/95528bd1-4100-43e9-bc0a-2e56670bc84f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k62zz\" (UID: \"95528bd1-4100-43e9-bc0a-2e56670bc84f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k62zz"
Sep 15 12:02:25.734142 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.731934    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5be458c0-7237-4003-b4b9-1218ac635869-agent-certs\") pod \"konnectivity-agent-c47md\" (UID: \"5be458c0-7237-4003-b4b9-1218ac635869\") " pod="kube-system/konnectivity-agent-c47md"
Sep 15 12:02:25.734142 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.731977    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0fa9c45a-2974-45d2-82c2-23a63b9f0662-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qjwgl\" (UID: \"0fa9c45a-2974-45d2-82c2-23a63b9f0662\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qjwgl"
Sep 15 12:02:25.734142 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.732001    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s54nn\" (UniqueName: \"kubernetes.io/projected/0fa9c45a-2974-45d2-82c2-23a63b9f0662-kube-api-access-s54nn\") pod \"aws-ebs-csi-driver-node-qjwgl\" (UID: \"0fa9c45a-2974-45d2-82c2-23a63b9f0662\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qjwgl"
Sep 15 12:02:25.734142 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.732033    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/95528bd1-4100-43e9-bc0a-2e56670bc84f-var-lib-openvswitch\") pod \"ovnkube-node-k62zz\" (UID: \"95528bd1-4100-43e9-bc0a-2e56670bc84f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k62zz"
Sep 15 12:02:25.734142 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.732057    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/95528bd1-4100-43e9-bc0a-2e56670bc84f-ovn-node-metrics-cert\") pod \"ovnkube-node-k62zz\" (UID: \"95528bd1-4100-43e9-bc0a-2e56670bc84f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k62zz"
Sep 15 12:02:25.734142 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.732084    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b9b05b4f-57d5-454e-b739-2778ddcab5fb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-p4rvp\" (UID: \"b9b05b4f-57d5-454e-b739-2778ddcab5fb\") " pod="openshift-multus/multus-additional-cni-plugins-p4rvp"
Sep 15 12:02:25.734142 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.732139    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/95528bd1-4100-43e9-bc0a-2e56670bc84f-node-log\") pod \"ovnkube-node-k62zz\" (UID: \"95528bd1-4100-43e9-bc0a-2e56670bc84f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k62zz"
Sep 15 12:02:25.734142 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.732164    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0fa9c45a-2974-45d2-82c2-23a63b9f0662-socket-dir\") pod \"aws-ebs-csi-driver-node-qjwgl\" (UID: \"0fa9c45a-2974-45d2-82c2-23a63b9f0662\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qjwgl"
Sep 15 12:02:25.734142 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.732188    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/95528bd1-4100-43e9-bc0a-2e56670bc84f-run-openvswitch\") pod \"ovnkube-node-k62zz\" (UID: \"95528bd1-4100-43e9-bc0a-2e56670bc84f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k62zz"
Sep 15 12:02:25.734142 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.732211    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/95528bd1-4100-43e9-bc0a-2e56670bc84f-host-cni-bin\") pod \"ovnkube-node-k62zz\" (UID: \"95528bd1-4100-43e9-bc0a-2e56670bc84f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k62zz"
Sep 15 12:02:25.734142 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.732252    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b9b05b4f-57d5-454e-b739-2778ddcab5fb-system-cni-dir\") pod \"multus-additional-cni-plugins-p4rvp\" (UID: \"b9b05b4f-57d5-454e-b739-2778ddcab5fb\") " pod="openshift-multus/multus-additional-cni-plugins-p4rvp"
Sep 15 12:02:25.734142 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.732278    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b9b05b4f-57d5-454e-b739-2778ddcab5fb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-p4rvp\" (UID: \"b9b05b4f-57d5-454e-b739-2778ddcab5fb\") " pod="openshift-multus/multus-additional-cni-plugins-p4rvp"
Sep 15 12:02:25.734142 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.732305    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqwmx\" (UniqueName: \"kubernetes.io/projected/25dcd72f-7a70-4cd8-ab3c-964f525bd756-kube-api-access-xqwmx\") pod \"iptables-alerter-7fbc7\" (UID: \"25dcd72f-7a70-4cd8-ab3c-964f525bd756\") " pod="openshift-network-operator/iptables-alerter-7fbc7"
Sep 15 12:02:25.734142 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.732417    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/95528bd1-4100-43e9-bc0a-2e56670bc84f-run-systemd\") pod \"ovnkube-node-k62zz\" (UID: \"95528bd1-4100-43e9-bc0a-2e56670bc84f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k62zz"
Sep 15 12:02:25.734142 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.732455    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 15 12:02:25.734142 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.732573    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-kgjkj"
Sep 15 12:02:25.735008 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.732445    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/95528bd1-4100-43e9-bc0a-2e56670bc84f-ovnkube-config\") pod \"ovnkube-node-k62zz\" (UID: \"95528bd1-4100-43e9-bc0a-2e56670bc84f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k62zz"
Sep 15 12:02:25.735008 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.732792    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b9b05b4f-57d5-454e-b739-2778ddcab5fb-cnibin\") pod \"multus-additional-cni-plugins-p4rvp\" (UID: \"b9b05b4f-57d5-454e-b739-2778ddcab5fb\") " pod="openshift-multus/multus-additional-cni-plugins-p4rvp"
Sep 15 12:02:25.735008 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.732826    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jm5zm\" (UniqueName: \"kubernetes.io/projected/b9b05b4f-57d5-454e-b739-2778ddcab5fb-kube-api-access-jm5zm\") pod \"multus-additional-cni-plugins-p4rvp\" (UID: \"b9b05b4f-57d5-454e-b739-2778ddcab5fb\") " pod="openshift-multus/multus-additional-cni-plugins-p4rvp"
Sep 15 12:02:25.735008 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.732855    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/95528bd1-4100-43e9-bc0a-2e56670bc84f-etc-openvswitch\") pod \"ovnkube-node-k62zz\" (UID: \"95528bd1-4100-43e9-bc0a-2e56670bc84f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k62zz"
Sep 15 12:02:25.735008 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.732882    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/95528bd1-4100-43e9-bc0a-2e56670bc84f-env-overrides\") pod \"ovnkube-node-k62zz\" (UID: \"95528bd1-4100-43e9-bc0a-2e56670bc84f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k62zz"
Sep 15 12:02:25.735008 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.732906    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5be458c0-7237-4003-b4b9-1218ac635869-konnectivity-ca\") pod \"konnectivity-agent-c47md\" (UID: \"5be458c0-7237-4003-b4b9-1218ac635869\") " pod="kube-system/konnectivity-agent-c47md"
Sep 15 12:02:25.735008 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.732935    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0fa9c45a-2974-45d2-82c2-23a63b9f0662-etc-selinux\") pod \"aws-ebs-csi-driver-node-qjwgl\" (UID: \"0fa9c45a-2974-45d2-82c2-23a63b9f0662\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qjwgl"
Sep 15 12:02:25.735008 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.732962    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/25dcd72f-7a70-4cd8-ab3c-964f525bd756-iptables-alerter-script\") pod \"iptables-alerter-7fbc7\" (UID: \"25dcd72f-7a70-4cd8-ab3c-964f525bd756\") " pod="openshift-network-operator/iptables-alerter-7fbc7"
Sep 15 12:02:25.735008 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.732988    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0fa9c45a-2974-45d2-82c2-23a63b9f0662-registration-dir\") pod \"aws-ebs-csi-driver-node-qjwgl\" (UID: \"0fa9c45a-2974-45d2-82c2-23a63b9f0662\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qjwgl"
Sep 15 12:02:25.735008 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.733370    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 15 12:02:25.735008 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.733385    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-b6dc2\""
Sep 15 12:02:25.735008 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.733569    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 15 12:02:25.735984 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.735960    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdjwd"
Sep 15 12:02:25.736070 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:25.736038    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdjwd" podUID="f9c68d51-9107-4cd4-a57d-6f76c0762106"
Sep 15 12:02:25.736499 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.736478    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-bzc5t\""
Sep 15 12:02:25.736582 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.736517    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 15 12:02:25.736582 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.736550    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 15 12:02:25.738482 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.738458    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wrxv2"
Sep 15 12:02:25.738582 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:25.738535    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wrxv2" podUID="090605d8-b2cc-4a2f-9f09-c22538254886"
Sep 15 12:02:25.740601 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.740573    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-tnbfp"
Sep 15 12:02:25.743548 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.743530    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 15 12:02:25.743911 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.743880    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7wff7\""
Sep 15 12:02:25.744286 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.744259    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 15 12:02:25.777524 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.777494    2555 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-6hj9t"
Sep 15 12:02:25.784617 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.784592    2555 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-6hj9t"
Sep 15 12:02:25.818242 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.818217    2555 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 15 12:02:25.819610 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.819525    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-17.ec2.internal" event={"ID":"99788892a4247a0a40d9fb25d8f545f5","Type":"ContainerStarted","Data":"deb47426d6ee0094fdb470b4adeba582c38aae8ff6fb83f72df90273cdace3f1"}
Sep 15 12:02:25.821711 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.821680    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-17.ec2.internal" event={"ID":"81882ce609db62ac71292b6338f62ad1","Type":"ContainerStarted","Data":"ddd647f94677071151b9854892a2dec4ae42bd5e7471fb590adb108dca11d594"}
Sep 15 12:02:25.833334 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.833293    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6p4hh\" (UniqueName: \"kubernetes.io/projected/01c26b58-ebc2-4b0f-96dd-14f8c373b0b5-kube-api-access-6p4hh\") pod \"tuned-tnbfp\" (UID: \"01c26b58-ebc2-4b0f-96dd-14f8c373b0b5\") " pod="openshift-cluster-node-tuning-operator/tuned-tnbfp"
Sep 15 12:02:25.833466 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.833349    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e246fd0a-f899-465f-aaeb-661d2a429d06-host-run-multus-certs\") pod \"multus-2wkff\" (UID: \"e246fd0a-f899-465f-aaeb-661d2a429d06\") " pod="openshift-multus/multus-2wkff"
Sep 15 12:02:25.833466 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.833383    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/95528bd1-4100-43e9-bc0a-2e56670bc84f-host-cni-bin\") pod \"ovnkube-node-k62zz\" (UID: \"95528bd1-4100-43e9-bc0a-2e56670bc84f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k62zz"
Sep 15 12:02:25.833466 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.833412    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b9b05b4f-57d5-454e-b739-2778ddcab5fb-system-cni-dir\") pod \"multus-additional-cni-plugins-p4rvp\" (UID: \"b9b05b4f-57d5-454e-b739-2778ddcab5fb\") " pod="openshift-multus/multus-additional-cni-plugins-p4rvp"
Sep 15 12:02:25.833466 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.833432    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b9b05b4f-57d5-454e-b739-2778ddcab5fb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-p4rvp\" (UID: \"b9b05b4f-57d5-454e-b739-2778ddcab5fb\") " pod="openshift-multus/multus-additional-cni-plugins-p4rvp"
Sep 15 12:02:25.833466 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.833449    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqwmx\" (UniqueName: \"kubernetes.io/projected/25dcd72f-7a70-4cd8-ab3c-964f525bd756-kube-api-access-xqwmx\") pod \"iptables-alerter-7fbc7\" (UID: \"25dcd72f-7a70-4cd8-ab3c-964f525bd756\") " pod="openshift-network-operator/iptables-alerter-7fbc7"
Sep 15 12:02:25.833757 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.833518    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b9b05b4f-57d5-454e-b739-2778ddcab5fb-system-cni-dir\") pod \"multus-additional-cni-plugins-p4rvp\" (UID: \"b9b05b4f-57d5-454e-b739-2778ddcab5fb\") " pod="openshift-multus/multus-additional-cni-plugins-p4rvp"
Sep 15 12:02:25.833757 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.833518    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/95528bd1-4100-43e9-bc0a-2e56670bc84f-host-cni-bin\") pod \"ovnkube-node-k62zz\" (UID: \"95528bd1-4100-43e9-bc0a-2e56670bc84f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k62zz"
Sep 15 12:02:25.833757 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.833548    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/95528bd1-4100-43e9-bc0a-2e56670bc84f-ovnkube-config\") pod \"ovnkube-node-k62zz\" (UID: \"95528bd1-4100-43e9-bc0a-2e56670bc84f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k62zz"
Sep 15 12:02:25.833757 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.833591    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm5zm\" (UniqueName: \"kubernetes.io/projected/b9b05b4f-57d5-454e-b739-2778ddcab5fb-kube-api-access-jm5zm\") pod \"multus-additional-cni-plugins-p4rvp\" (UID: \"b9b05b4f-57d5-454e-b739-2778ddcab5fb\") " pod="openshift-multus/multus-additional-cni-plugins-p4rvp"
Sep 15 12:02:25.833757 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.833624    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0fa9c45a-2974-45d2-82c2-23a63b9f0662-etc-selinux\") pod \"aws-ebs-csi-driver-node-qjwgl\" (UID: \"0fa9c45a-2974-45d2-82c2-23a63b9f0662\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qjwgl"
Sep 15 12:02:25.833757 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.833652    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/01c26b58-ebc2-4b0f-96dd-14f8c373b0b5-etc-systemd\") pod \"tuned-tnbfp\" (UID: \"01c26b58-ebc2-4b0f-96dd-14f8c373b0b5\") " pod="openshift-cluster-node-tuning-operator/tuned-tnbfp"
Sep 15 12:02:25.833757 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.833678    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/25dcd72f-7a70-4cd8-ab3c-964f525bd756-iptables-alerter-script\") pod \"iptables-alerter-7fbc7\" (UID: \"25dcd72f-7a70-4cd8-ab3c-964f525bd756\") " pod="openshift-network-operator/iptables-alerter-7fbc7"
Sep 15 12:02:25.833757 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.833705    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0fa9c45a-2974-45d2-82c2-23a63b9f0662-sys-fs\") pod \"aws-ebs-csi-driver-node-qjwgl\" (UID: \"0fa9c45a-2974-45d2-82c2-23a63b9f0662\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qjwgl"
Sep 15 12:02:25.833757 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.833734    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e246fd0a-f899-465f-aaeb-661d2a429d06-multus-cni-dir\") pod \"multus-2wkff\" (UID: \"e246fd0a-f899-465f-aaeb-661d2a429d06\") " pod="openshift-multus/multus-2wkff"
Sep 15 12:02:25.833757 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.833759    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/01c26b58-ebc2-4b0f-96dd-14f8c373b0b5-etc-sysctl-conf\") pod \"tuned-tnbfp\" (UID: \"01c26b58-ebc2-4b0f-96dd-14f8c373b0b5\") " pod="openshift-cluster-node-tuning-operator/tuned-tnbfp"
Sep 15 12:02:25.834270 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.833783    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e246fd0a-f899-465f-aaeb-661d2a429d06-system-cni-dir\") pod \"multus-2wkff\" (UID: \"e246fd0a-f899-465f-aaeb-661d2a429d06\") " pod="openshift-multus/multus-2wkff"
Sep 15 12:02:25.834270 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.833810    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b9b05b4f-57d5-454e-b739-2778ddcab5fb-cni-binary-copy\") pod \"multus-additional-cni-plugins-p4rvp\" (UID: \"b9b05b4f-57d5-454e-b739-2778ddcab5fb\") " pod="openshift-multus/multus-additional-cni-plugins-p4rvp"
Sep 15 12:02:25.834270 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.833834    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/95528bd1-4100-43e9-bc0a-2e56670bc84f-systemd-units\") pod \"ovnkube-node-k62zz\" (UID: \"95528bd1-4100-43e9-bc0a-2e56670bc84f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k62zz"
Sep 15 12:02:25.834270 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.833858    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/95528bd1-4100-43e9-bc0a-2e56670bc84f-host-slash\") pod \"ovnkube-node-k62zz\" (UID: \"95528bd1-4100-43e9-bc0a-2e56670bc84f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k62zz"
Sep 15 12:02:25.834270 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.833883    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e246fd0a-f899-465f-aaeb-661d2a429d06-host-run-netns\") pod \"multus-2wkff\" (UID: \"e246fd0a-f899-465f-aaeb-661d2a429d06\") " pod="openshift-multus/multus-2wkff"
Sep 15 12:02:25.834270 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.833911    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e246fd0a-f899-465f-aaeb-661d2a429d06-host-var-lib-cni-multus\") pod \"multus-2wkff\" (UID: \"e246fd0a-f899-465f-aaeb-661d2a429d06\") " pod="openshift-multus/multus-2wkff"
Sep 15 12:02:25.834270 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.833936    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wsgh\" (UniqueName: \"kubernetes.io/projected/f9c68d51-9107-4cd4-a57d-6f76c0762106-kube-api-access-8wsgh\") pod \"network-metrics-daemon-bdjwd\" (UID: \"f9c68d51-9107-4cd4-a57d-6f76c0762106\") " pod="openshift-multus/network-metrics-daemon-bdjwd"
Sep 15 12:02:25.834270 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.833954    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0fa9c45a-2974-45d2-82c2-23a63b9f0662-sys-fs\") pod \"aws-ebs-csi-driver-node-qjwgl\" (UID: \"0fa9c45a-2974-45d2-82c2-23a63b9f0662\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qjwgl"
Sep 15 12:02:25.834270 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.833962    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/365587ef-e477-48b4-aa6d-4e284b0b5951-tmp-dir\") pod \"node-resolver-kgjkj\" (UID: \"365587ef-e477-48b4-aa6d-4e284b0b5951\") " pod="openshift-dns/node-resolver-kgjkj"
Sep 15 12:02:25.834270 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.834024    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/95528bd1-4100-43e9-bc0a-2e56670bc84f-host-slash\") pod \"ovnkube-node-k62zz\" (UID: \"95528bd1-4100-43e9-bc0a-2e56670bc84f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k62zz"
Sep 15 12:02:25.834270 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.834071    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/95528bd1-4100-43e9-bc0a-2e56670bc84f-systemd-units\") pod \"ovnkube-node-k62zz\" (UID: \"95528bd1-4100-43e9-bc0a-2e56670bc84f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k62zz"
Sep 15 12:02:25.834270 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.834118    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b9b05b4f-57d5-454e-b739-2778ddcab5fb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-p4rvp\" (UID: \"b9b05b4f-57d5-454e-b739-2778ddcab5fb\") " pod="openshift-multus/multus-additional-cni-plugins-p4rvp"
Sep 15 12:02:25.834270 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.834135    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/95528bd1-4100-43e9-bc0a-2e56670bc84f-ovnkube-config\") pod \"ovnkube-node-k62zz\" (UID: \"95528bd1-4100-43e9-bc0a-2e56670bc84f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k62zz"
Sep 15 12:02:25.834270 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.834192    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/07f9ddc4-14d7-4b04-9797-67507d4063bc-dbus\") pod \"global-pull-secret-syncer-g25rd\" (UID: \"07f9ddc4-14d7-4b04-9797-67507d4063bc\") " pod="kube-system/global-pull-secret-syncer-g25rd"
Sep 15 12:02:25.834270 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.834228    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/95528bd1-4100-43e9-bc0a-2e56670bc84f-run-ovn\") pod \"ovnkube-node-k62zz\" (UID: \"95528bd1-4100-43e9-bc0a-2e56670bc84f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k62zz"
Sep 15 12:02:25.835064 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.834286    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/95528bd1-4100-43e9-bc0a-2e56670bc84f-host-kubelet\") pod \"ovnkube-node-k62zz\" (UID: \"95528bd1-4100-43e9-bc0a-2e56670bc84f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k62zz"
Sep 15 12:02:25.835064 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.834314    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/95528bd1-4100-43e9-bc0a-2e56670bc84f-host-run-netns\") pod \"ovnkube-node-k62zz\" (UID: \"95528bd1-4100-43e9-bc0a-2e56670bc84f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k62zz"
Sep 15 12:02:25.835064 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.834342    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0fa9c45a-2974-45d2-82c2-23a63b9f0662-etc-selinux\") pod \"aws-ebs-csi-driver-node-qjwgl\" (UID: \"0fa9c45a-2974-45d2-82c2-23a63b9f0662\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qjwgl"
Sep 15 12:02:25.835064 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.834379    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ee282972-ad6d-4a25-a3f5-218dfe607d6a-host\") pod \"node-ca-mkck2\" (UID: \"ee282972-ad6d-4a25-a3f5-218dfe607d6a\") " pod="openshift-image-registry/node-ca-mkck2"
Sep 15 12:02:25.835064 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.834411    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/365587ef-e477-48b4-aa6d-4e284b0b5951-hosts-file\") pod \"node-resolver-kgjkj\" (UID: \"365587ef-e477-48b4-aa6d-4e284b0b5951\") " pod="openshift-dns/node-resolver-kgjkj"
Sep 15 12:02:25.835064 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.834419    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/95528bd1-4100-43e9-bc0a-2e56670bc84f-host-run-netns\") pod \"ovnkube-node-k62zz\" (UID: \"95528bd1-4100-43e9-bc0a-2e56670bc84f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k62zz"
Sep 15 12:02:25.835064 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.834381    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/95528bd1-4100-43e9-bc0a-2e56670bc84f-run-ovn\") pod \"ovnkube-node-k62zz\" (UID: \"95528bd1-4100-43e9-bc0a-2e56670bc84f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k62zz"
Sep 15 12:02:25.835064 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.834465    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/95528bd1-4100-43e9-bc0a-2e56670bc84f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k62zz\" (UID: \"95528bd1-4100-43e9-bc0a-2e56670bc84f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k62zz"
Sep 15 12:02:25.835064 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.834471    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/95528bd1-4100-43e9-bc0a-2e56670bc84f-host-kubelet\") pod \"ovnkube-node-k62zz\" (UID: \"95528bd1-4100-43e9-bc0a-2e56670bc84f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k62zz"
Sep 15 12:02:25.835064 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.834497    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5be458c0-7237-4003-b4b9-1218ac635869-agent-certs\") pod \"konnectivity-agent-c47md\" (UID: \"5be458c0-7237-4003-b4b9-1218ac635869\") " pod="kube-system/konnectivity-agent-c47md"
Sep 15 12:02:25.835064 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.834514    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/95528bd1-4100-43e9-bc0a-2e56670bc84f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k62zz\" (UID: \"95528bd1-4100-43e9-bc0a-2e56670bc84f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k62zz"
Sep 15 12:02:25.835064 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.834547    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2mdt\" (UniqueName: \"kubernetes.io/projected/365587ef-e477-48b4-aa6d-4e284b0b5951-kube-api-access-z2mdt\") pod \"node-resolver-kgjkj\" (UID: \"365587ef-e477-48b4-aa6d-4e284b0b5951\") " pod="openshift-dns/node-resolver-kgjkj"
Sep 15 12:02:25.835064 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.834585    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e246fd0a-f899-465f-aaeb-661d2a429d06-hostroot\") pod \"multus-2wkff\" (UID: \"e246fd0a-f899-465f-aaeb-661d2a429d06\") " pod="openshift-multus/multus-2wkff"
Sep 15 12:02:25.835064 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.834610    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e246fd0a-f899-465f-aaeb-661d2a429d06-multus-daemon-config\") pod \"multus-2wkff\" (UID: \"e246fd0a-f899-465f-aaeb-661d2a429d06\") " pod="openshift-multus/multus-2wkff"
Sep 15 12:02:25.835064 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.834611    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b9b05b4f-57d5-454e-b739-2778ddcab5fb-cni-binary-copy\") pod \"multus-additional-cni-plugins-p4rvp\" (UID: \"b9b05b4f-57d5-454e-b739-2778ddcab5fb\") " pod="openshift-multus/multus-additional-cni-plugins-p4rvp"
Sep 15 12:02:25.835064 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.834761    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e246fd0a-f899-465f-aaeb-661d2a429d06-etc-kubernetes\") pod \"multus-2wkff\" (UID: \"e246fd0a-f899-465f-aaeb-661d2a429d06\") " pod="openshift-multus/multus-2wkff"
Sep 15 12:02:25.835064 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.834794    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5vth\" (UniqueName: \"kubernetes.io/projected/e246fd0a-f899-465f-aaeb-661d2a429d06-kube-api-access-k5vth\") pod \"multus-2wkff\" (UID: \"e246fd0a-f899-465f-aaeb-661d2a429d06\") " pod="openshift-multus/multus-2wkff"
Sep 15 12:02:25.835954 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.834838    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/95528bd1-4100-43e9-bc0a-2e56670bc84f-ovn-node-metrics-cert\") pod \"ovnkube-node-k62zz\" (UID: \"95528bd1-4100-43e9-bc0a-2e56670bc84f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k62zz"
Sep 15 12:02:25.835954 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.834866    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e246fd0a-f899-465f-aaeb-661d2a429d06-host-var-lib-cni-bin\") pod \"multus-2wkff\" (UID: \"e246fd0a-f899-465f-aaeb-661d2a429d06\") " pod="openshift-multus/multus-2wkff"
Sep 15 12:02:25.835954 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.834904    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e246fd0a-f899-465f-aaeb-661d2a429d06-multus-conf-dir\") pod \"multus-2wkff\" (UID: \"e246fd0a-f899-465f-aaeb-661d2a429d06\") " pod="openshift-multus/multus-2wkff"
Sep 15 12:02:25.835954 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.834943    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9c68d51-9107-4cd4-a57d-6f76c0762106-metrics-certs\") pod \"network-metrics-daemon-bdjwd\" (UID: \"f9c68d51-9107-4cd4-a57d-6f76c0762106\") " pod="openshift-multus/network-metrics-daemon-bdjwd"
Sep 15 12:02:25.835954 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.834968    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/01c26b58-ebc2-4b0f-96dd-14f8c373b0b5-etc-sysconfig\") pod \"tuned-tnbfp\" (UID: \"01c26b58-ebc2-4b0f-96dd-14f8c373b0b5\") " pod="openshift-cluster-node-tuning-operator/tuned-tnbfp"
Sep 15 12:02:25.835954 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.834999    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/01c26b58-ebc2-4b0f-96dd-14f8c373b0b5-run\") pod \"tuned-tnbfp\" (UID: \"01c26b58-ebc2-4b0f-96dd-14f8c373b0b5\") " pod="openshift-cluster-node-tuning-operator/tuned-tnbfp"
Sep 15 12:02:25.835954 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.835026    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b9b05b4f-57d5-454e-b739-2778ddcab5fb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-p4rvp\" (UID: \"b9b05b4f-57d5-454e-b739-2778ddcab5fb\") " pod="openshift-multus/multus-additional-cni-plugins-p4rvp"
Sep 15 12:02:25.835954 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.835053    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/95528bd1-4100-43e9-bc0a-2e56670bc84f-node-log\") pod \"ovnkube-node-k62zz\" (UID: \"95528bd1-4100-43e9-bc0a-2e56670bc84f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k62zz"
Sep 15 12:02:25.835954 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.835085    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0fa9c45a-2974-45d2-82c2-23a63b9f0662-socket-dir\") pod \"aws-ebs-csi-driver-node-qjwgl\" (UID: \"0fa9c45a-2974-45d2-82c2-23a63b9f0662\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qjwgl"
Sep 15 12:02:25.835954 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.835139    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/01c26b58-ebc2-4b0f-96dd-14f8c373b0b5-lib-modules\") pod \"tuned-tnbfp\" (UID: \"01c26b58-ebc2-4b0f-96dd-14f8c373b0b5\") " pod="openshift-cluster-node-tuning-operator/tuned-tnbfp"
Sep 15 12:02:25.835954 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.835165    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/95528bd1-4100-43e9-bc0a-2e56670bc84f-run-openvswitch\") pod \"ovnkube-node-k62zz\" (UID: \"95528bd1-4100-43e9-bc0a-2e56670bc84f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k62zz"
Sep 15 12:02:25.835954 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.835193    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/95528bd1-4100-43e9-bc0a-2e56670bc84f-run-systemd\") pod \"ovnkube-node-k62zz\" (UID: \"95528bd1-4100-43e9-bc0a-2e56670bc84f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k62zz"
Sep 15 12:02:25.835954 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.835216    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ee282972-ad6d-4a25-a3f5-218dfe607d6a-serviceca\") pod \"node-ca-mkck2\" (UID: \"ee282972-ad6d-4a25-a3f5-218dfe607d6a\") " pod="openshift-image-registry/node-ca-mkck2"
Sep 15 12:02:25.835954 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.835241    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/07f9ddc4-14d7-4b04-9797-67507d4063bc-original-pull-secret\") pod \"global-pull-secret-syncer-g25rd\" (UID: \"07f9ddc4-14d7-4b04-9797-67507d4063bc\") " pod="kube-system/global-pull-secret-syncer-g25rd"
Sep 15 12:02:25.835954 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.835283    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e246fd0a-f899-465f-aaeb-661d2a429d06-multus-socket-dir-parent\") pod \"multus-2wkff\" (UID: \"e246fd0a-f899-465f-aaeb-661d2a429d06\") " pod="openshift-multus/multus-2wkff"
Sep 15 12:02:25.835954 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.835349    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e246fd0a-f899-465f-aaeb-661d2a429d06-host-var-lib-kubelet\") pod \"multus-2wkff\" (UID: \"e246fd0a-f899-465f-aaeb-661d2a429d06\") " pod="openshift-multus/multus-2wkff"
Sep 15 12:02:25.835954 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.835376    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/01c26b58-ebc2-4b0f-96dd-14f8c373b0b5-etc-modprobe-d\") pod \"tuned-tnbfp\" (UID: \"01c26b58-ebc2-4b0f-96dd-14f8c373b0b5\") " pod="openshift-cluster-node-tuning-operator/tuned-tnbfp"
Sep 15 12:02:25.836875 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.835404    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b9b05b4f-57d5-454e-b739-2778ddcab5fb-cnibin\") pod \"multus-additional-cni-plugins-p4rvp\" (UID: \"b9b05b4f-57d5-454e-b739-2778ddcab5fb\") " pod="openshift-multus/multus-additional-cni-plugins-p4rvp"
Sep 15 12:02:25.836875 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.835445    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/25dcd72f-7a70-4cd8-ab3c-964f525bd756-iptables-alerter-script\") pod \"iptables-alerter-7fbc7\" (UID: \"25dcd72f-7a70-4cd8-ab3c-964f525bd756\") " pod="openshift-network-operator/iptables-alerter-7fbc7"
Sep 15 12:02:25.836875 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.835449    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/95528bd1-4100-43e9-bc0a-2e56670bc84f-etc-openvswitch\") pod \"ovnkube-node-k62zz\" (UID: \"95528bd1-4100-43e9-bc0a-2e56670bc84f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k62zz"
Sep 15 12:02:25.836875 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.835463    2555 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 15 12:02:25.836875 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.835498    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0fa9c45a-2974-45d2-82c2-23a63b9f0662-socket-dir\") pod \"aws-ebs-csi-driver-node-qjwgl\" (UID: \"0fa9c45a-2974-45d2-82c2-23a63b9f0662\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qjwgl"
Sep 15 12:02:25.836875 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.835550    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/95528bd1-4100-43e9-bc0a-2e56670bc84f-run-openvswitch\") pod \"ovnkube-node-k62zz\" (UID: \"95528bd1-4100-43e9-bc0a-2e56670bc84f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k62zz"
Sep 15 12:02:25.836875 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.835551    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/95528bd1-4100-43e9-bc0a-2e56670bc84f-env-overrides\") pod \"ovnkube-node-k62zz\" (UID: \"95528bd1-4100-43e9-bc0a-2e56670bc84f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k62zz"
Sep 15 12:02:25.836875 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.835599    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/95528bd1-4100-43e9-bc0a-2e56670bc84f-etc-openvswitch\") pod \"ovnkube-node-k62zz\" (UID: \"95528bd1-4100-43e9-bc0a-2e56670bc84f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k62zz"
Sep 15 12:02:25.836875 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.835599    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5be458c0-7237-4003-b4b9-1218ac635869-konnectivity-ca\") pod \"konnectivity-agent-c47md\" (UID: \"5be458c0-7237-4003-b4b9-1218ac635869\") " pod="kube-system/konnectivity-agent-c47md"
Sep 15 12:02:25.836875 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.835614    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/95528bd1-4100-43e9-bc0a-2e56670bc84f-node-log\") pod \"ovnkube-node-k62zz\" (UID: \"95528bd1-4100-43e9-bc0a-2e56670bc84f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k62zz"
Sep 15 12:02:25.836875 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.835642    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e246fd0a-f899-465f-aaeb-661d2a429d06-cni-binary-copy\") pod \"multus-2wkff\" (UID: \"e246fd0a-f899-465f-aaeb-661d2a429d06\") " pod="openshift-multus/multus-2wkff"
Sep 15 12:02:25.836875 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.835645    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b9b05b4f-57d5-454e-b739-2778ddcab5fb-cnibin\") pod \"multus-additional-cni-plugins-p4rvp\" (UID: \"b9b05b4f-57d5-454e-b739-2778ddcab5fb\") " pod="openshift-multus/multus-additional-cni-plugins-p4rvp"
Sep 15 12:02:25.836875 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.835560    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/95528bd1-4100-43e9-bc0a-2e56670bc84f-run-systemd\") pod \"ovnkube-node-k62zz\" (UID: \"95528bd1-4100-43e9-bc0a-2e56670bc84f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k62zz"
Sep 15 12:02:25.836875 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.835992    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/01c26b58-ebc2-4b0f-96dd-14f8c373b0b5-etc-tuned\") pod \"tuned-tnbfp\" (UID: \"01c26b58-ebc2-4b0f-96dd-14f8c373b0b5\") " pod="openshift-cluster-node-tuning-operator/tuned-tnbfp"
Sep 15 12:02:25.836875 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.836032    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/95528bd1-4100-43e9-bc0a-2e56670bc84f-env-overrides\") pod \"ovnkube-node-k62zz\" (UID: \"95528bd1-4100-43e9-bc0a-2e56670bc84f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k62zz"
Sep 15 12:02:25.836875 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.836033    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0fa9c45a-2974-45d2-82c2-23a63b9f0662-registration-dir\") pod \"aws-ebs-csi-driver-node-qjwgl\" (UID: \"0fa9c45a-2974-45d2-82c2-23a63b9f0662\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qjwgl"
Sep 15 12:02:25.836875 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.836083    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0fa9c45a-2974-45d2-82c2-23a63b9f0662-device-dir\") pod \"aws-ebs-csi-driver-node-qjwgl\" (UID: \"0fa9c45a-2974-45d2-82c2-23a63b9f0662\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qjwgl"
Sep 15 12:02:25.837771 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.836086    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0fa9c45a-2974-45d2-82c2-23a63b9f0662-registration-dir\") pod \"aws-ebs-csi-driver-node-qjwgl\" (UID: \"0fa9c45a-2974-45d2-82c2-23a63b9f0662\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qjwgl"
Sep 15 12:02:25.837771 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.836118    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b9b05b4f-57d5-454e-b739-2778ddcab5fb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-p4rvp\" (UID: \"b9b05b4f-57d5-454e-b739-2778ddcab5fb\") " pod="openshift-multus/multus-additional-cni-plugins-p4rvp"
Sep 15 12:02:25.837771 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.836147    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/95528bd1-4100-43e9-bc0a-2e56670bc84f-log-socket\") pod \"ovnkube-node-k62zz\" (UID: \"95528bd1-4100-43e9-bc0a-2e56670bc84f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k62zz"
Sep 15 12:02:25.837771 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.836153    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0fa9c45a-2974-45d2-82c2-23a63b9f0662-device-dir\") pod \"aws-ebs-csi-driver-node-qjwgl\" (UID: \"0fa9c45a-2974-45d2-82c2-23a63b9f0662\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qjwgl"
Sep 15 12:02:25.837771 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.836163    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b9b05b4f-57d5-454e-b739-2778ddcab5fb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-p4rvp\" (UID: \"b9b05b4f-57d5-454e-b739-2778ddcab5fb\") " pod="openshift-multus/multus-additional-cni-plugins-p4rvp"
Sep 15 12:02:25.837771 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.836182    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jx9sp\" (UniqueName: \"kubernetes.io/projected/95528bd1-4100-43e9-bc0a-2e56670bc84f-kube-api-access-jx9sp\") pod \"ovnkube-node-k62zz\" (UID: \"95528bd1-4100-43e9-bc0a-2e56670bc84f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k62zz"
Sep 15 12:02:25.837771 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.836197    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/95528bd1-4100-43e9-bc0a-2e56670bc84f-log-socket\") pod \"ovnkube-node-k62zz\" (UID: \"95528bd1-4100-43e9-bc0a-2e56670bc84f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k62zz"
Sep 15 12:02:25.837771 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.836214    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xnbb\" (UniqueName: \"kubernetes.io/projected/ee282972-ad6d-4a25-a3f5-218dfe607d6a-kube-api-access-2xnbb\") pod \"node-ca-mkck2\" (UID: \"ee282972-ad6d-4a25-a3f5-218dfe607d6a\") " pod="openshift-image-registry/node-ca-mkck2"
Sep 15 12:02:25.837771 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.836337    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm9js\" (UniqueName: \"kubernetes.io/projected/090605d8-b2cc-4a2f-9f09-c22538254886-kube-api-access-zm9js\") pod \"network-check-target-wrxv2\" (UID: \"090605d8-b2cc-4a2f-9f09-c22538254886\") " pod="openshift-network-diagnostics/network-check-target-wrxv2"
Sep 15 12:02:25.837771 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.836363    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b9b05b4f-57d5-454e-b739-2778ddcab5fb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-p4rvp\" (UID: \"b9b05b4f-57d5-454e-b739-2778ddcab5fb\") " pod="openshift-multus/multus-additional-cni-plugins-p4rvp"
Sep 15 12:02:25.837771 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.836368    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/01c26b58-ebc2-4b0f-96dd-14f8c373b0b5-sys\") pod \"tuned-tnbfp\" (UID: \"01c26b58-ebc2-4b0f-96dd-14f8c373b0b5\") " pod="openshift-cluster-node-tuning-operator/tuned-tnbfp"
Sep 15 12:02:25.837771 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.836420    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/01c26b58-ebc2-4b0f-96dd-14f8c373b0b5-var-lib-kubelet\") pod \"tuned-tnbfp\" (UID: \"01c26b58-ebc2-4b0f-96dd-14f8c373b0b5\") " pod="openshift-cluster-node-tuning-operator/tuned-tnbfp"
Sep 15 12:02:25.837771 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.836412    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5be458c0-7237-4003-b4b9-1218ac635869-konnectivity-ca\") pod \"konnectivity-agent-c47md\" (UID: \"5be458c0-7237-4003-b4b9-1218ac635869\") " pod="kube-system/konnectivity-agent-c47md"
Sep 15 12:02:25.837771 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.836452    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/01c26b58-ebc2-4b0f-96dd-14f8c373b0b5-host\") pod \"tuned-tnbfp\" (UID: \"01c26b58-ebc2-4b0f-96dd-14f8c373b0b5\") " pod="openshift-cluster-node-tuning-operator/tuned-tnbfp"
Sep 15 12:02:25.837771 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.836486    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/95528bd1-4100-43e9-bc0a-2e56670bc84f-host-run-ovn-kubernetes\") pod \"ovnkube-node-k62zz\" (UID: \"95528bd1-4100-43e9-bc0a-2e56670bc84f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k62zz"
Sep 15 12:02:25.837771 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.836515    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/95528bd1-4100-43e9-bc0a-2e56670bc84f-ovnkube-script-lib\") pod \"ovnkube-node-k62zz\" (UID: \"95528bd1-4100-43e9-bc0a-2e56670bc84f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k62zz"
Sep 15 12:02:25.837771 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.836542    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e246fd0a-f899-465f-aaeb-661d2a429d06-host-run-k8s-cni-cncf-io\") pod \"multus-2wkff\" (UID: \"e246fd0a-f899-465f-aaeb-661d2a429d06\") " pod="openshift-multus/multus-2wkff"
Sep 15 12:02:25.838684 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.836583    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/25dcd72f-7a70-4cd8-ab3c-964f525bd756-host-slash\") pod \"iptables-alerter-7fbc7\" (UID: \"25dcd72f-7a70-4cd8-ab3c-964f525bd756\") " pod="openshift-network-operator/iptables-alerter-7fbc7"
Sep 15 12:02:25.838684 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.836615    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/95528bd1-4100-43e9-bc0a-2e56670bc84f-host-cni-netd\") pod \"ovnkube-node-k62zz\" (UID: \"95528bd1-4100-43e9-bc0a-2e56670bc84f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k62zz"
Sep 15 12:02:25.838684 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.836642    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0fa9c45a-2974-45d2-82c2-23a63b9f0662-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qjwgl\" (UID: \"0fa9c45a-2974-45d2-82c2-23a63b9f0662\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qjwgl"
Sep 15 12:02:25.838684 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.836683    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s54nn\" (UniqueName: \"kubernetes.io/projected/0fa9c45a-2974-45d2-82c2-23a63b9f0662-kube-api-access-s54nn\") pod \"aws-ebs-csi-driver-node-qjwgl\" (UID: \"0fa9c45a-2974-45d2-82c2-23a63b9f0662\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qjwgl"
Sep 15 12:02:25.838684 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.836711    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/01c26b58-ebc2-4b0f-96dd-14f8c373b0b5-etc-sysctl-d\") pod \"tuned-tnbfp\" (UID: \"01c26b58-ebc2-4b0f-96dd-14f8c373b0b5\") " pod="openshift-cluster-node-tuning-operator/tuned-tnbfp"
Sep 15 12:02:25.838684 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.836739    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/01c26b58-ebc2-4b0f-96dd-14f8c373b0b5-tmp\") pod \"tuned-tnbfp\" (UID: \"01c26b58-ebc2-4b0f-96dd-14f8c373b0b5\") " pod="openshift-cluster-node-tuning-operator/tuned-tnbfp"
Sep 15 12:02:25.838684 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.836766    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/95528bd1-4100-43e9-bc0a-2e56670bc84f-var-lib-openvswitch\") pod \"ovnkube-node-k62zz\" (UID: \"95528bd1-4100-43e9-bc0a-2e56670bc84f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k62zz"
Sep 15 12:02:25.838684 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.836791    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/07f9ddc4-14d7-4b04-9797-67507d4063bc-kubelet-config\") pod \"global-pull-secret-syncer-g25rd\" (UID: \"07f9ddc4-14d7-4b04-9797-67507d4063bc\") " pod="kube-system/global-pull-secret-syncer-g25rd"
Sep 15 12:02:25.838684 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.836817    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e246fd0a-f899-465f-aaeb-661d2a429d06-cnibin\") pod \"multus-2wkff\" (UID: \"e246fd0a-f899-465f-aaeb-661d2a429d06\") " pod="openshift-multus/multus-2wkff"
Sep 15 12:02:25.838684 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.836846    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/01c26b58-ebc2-4b0f-96dd-14f8c373b0b5-etc-kubernetes\") pod \"tuned-tnbfp\" (UID: \"01c26b58-ebc2-4b0f-96dd-14f8c373b0b5\") " pod="openshift-cluster-node-tuning-operator/tuned-tnbfp"
Sep 15 12:02:25.838684 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.836583    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/95528bd1-4100-43e9-bc0a-2e56670bc84f-host-run-ovn-kubernetes\") pod \"ovnkube-node-k62zz\" (UID: \"95528bd1-4100-43e9-bc0a-2e56670bc84f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k62zz"
Sep 15 12:02:25.838684 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.836963    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/25dcd72f-7a70-4cd8-ab3c-964f525bd756-host-slash\") pod \"iptables-alerter-7fbc7\" (UID: \"25dcd72f-7a70-4cd8-ab3c-964f525bd756\") " pod="openshift-network-operator/iptables-alerter-7fbc7"
Sep 15 12:02:25.838684 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.837013    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/95528bd1-4100-43e9-bc0a-2e56670bc84f-host-cni-netd\") pod \"ovnkube-node-k62zz\" (UID: \"95528bd1-4100-43e9-bc0a-2e56670bc84f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k62zz"
Sep 15 12:02:25.838684 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.837018    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/95528bd1-4100-43e9-bc0a-2e56670bc84f-var-lib-openvswitch\") pod \"ovnkube-node-k62zz\" (UID: \"95528bd1-4100-43e9-bc0a-2e56670bc84f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k62zz"
Sep 15 12:02:25.838684 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.837076    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0fa9c45a-2974-45d2-82c2-23a63b9f0662-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qjwgl\" (UID: \"0fa9c45a-2974-45d2-82c2-23a63b9f0662\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qjwgl"
Sep 15 12:02:25.838684 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.837128    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/95528bd1-4100-43e9-bc0a-2e56670bc84f-ovnkube-script-lib\") pod \"ovnkube-node-k62zz\" (UID: \"95528bd1-4100-43e9-bc0a-2e56670bc84f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k62zz"
Sep 15 12:02:25.841508 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.841150    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/95528bd1-4100-43e9-bc0a-2e56670bc84f-ovn-node-metrics-cert\") pod \"ovnkube-node-k62zz\" (UID: \"95528bd1-4100-43e9-bc0a-2e56670bc84f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k62zz"
Sep 15 12:02:25.841508 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.841342    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5be458c0-7237-4003-b4b9-1218ac635869-agent-certs\") pod \"konnectivity-agent-c47md\" (UID: \"5be458c0-7237-4003-b4b9-1218ac635869\") " pod="kube-system/konnectivity-agent-c47md"
Sep 15 12:02:25.851240 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.851164    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s54nn\" (UniqueName: \"kubernetes.io/projected/0fa9c45a-2974-45d2-82c2-23a63b9f0662-kube-api-access-s54nn\") pod \"aws-ebs-csi-driver-node-qjwgl\" (UID: \"0fa9c45a-2974-45d2-82c2-23a63b9f0662\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qjwgl"
Sep 15 12:02:25.854118 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.854066    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqwmx\" (UniqueName: \"kubernetes.io/projected/25dcd72f-7a70-4cd8-ab3c-964f525bd756-kube-api-access-xqwmx\") pod \"iptables-alerter-7fbc7\" (UID: \"25dcd72f-7a70-4cd8-ab3c-964f525bd756\") " pod="openshift-network-operator/iptables-alerter-7fbc7"
Sep 15 12:02:25.854745 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.854694    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm5zm\" (UniqueName: \"kubernetes.io/projected/b9b05b4f-57d5-454e-b739-2778ddcab5fb-kube-api-access-jm5zm\") pod \"multus-additional-cni-plugins-p4rvp\" (UID: \"b9b05b4f-57d5-454e-b739-2778ddcab5fb\") " pod="openshift-multus/multus-additional-cni-plugins-p4rvp"
Sep 15 12:02:25.856717 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.856695    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jx9sp\" (UniqueName: \"kubernetes.io/projected/95528bd1-4100-43e9-bc0a-2e56670bc84f-kube-api-access-jx9sp\") pod \"ovnkube-node-k62zz\" (UID: \"95528bd1-4100-43e9-bc0a-2e56670bc84f\") " pod="openshift-ovn-kubernetes/ovnkube-node-k62zz"
Sep 15 12:02:25.937202 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.937165    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e246fd0a-f899-465f-aaeb-661d2a429d06-host-run-netns\") pod \"multus-2wkff\" (UID: \"e246fd0a-f899-465f-aaeb-661d2a429d06\") " pod="openshift-multus/multus-2wkff"
Sep 15 12:02:25.937202 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.937206    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e246fd0a-f899-465f-aaeb-661d2a429d06-host-var-lib-cni-multus\") pod \"multus-2wkff\" (UID: \"e246fd0a-f899-465f-aaeb-661d2a429d06\") " pod="openshift-multus/multus-2wkff"
Sep 15 12:02:25.937479 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.937223    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wsgh\" (UniqueName: \"kubernetes.io/projected/f9c68d51-9107-4cd4-a57d-6f76c0762106-kube-api-access-8wsgh\") pod \"network-metrics-daemon-bdjwd\" (UID: \"f9c68d51-9107-4cd4-a57d-6f76c0762106\") " pod="openshift-multus/network-metrics-daemon-bdjwd"
Sep 15 12:02:25.937479 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.937248    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/365587ef-e477-48b4-aa6d-4e284b0b5951-tmp-dir\") pod \"node-resolver-kgjkj\" (UID: \"365587ef-e477-48b4-aa6d-4e284b0b5951\") " pod="openshift-dns/node-resolver-kgjkj"
Sep 15 12:02:25.937479 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.937304    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e246fd0a-f899-465f-aaeb-661d2a429d06-host-run-netns\") pod \"multus-2wkff\" (UID: \"e246fd0a-f899-465f-aaeb-661d2a429d06\") " pod="openshift-multus/multus-2wkff"
Sep 15 12:02:25.937479 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.937299    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/07f9ddc4-14d7-4b04-9797-67507d4063bc-dbus\") pod \"global-pull-secret-syncer-g25rd\" (UID: \"07f9ddc4-14d7-4b04-9797-67507d4063bc\") " pod="kube-system/global-pull-secret-syncer-g25rd"
Sep 15 12:02:25.937479 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.937310    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e246fd0a-f899-465f-aaeb-661d2a429d06-host-var-lib-cni-multus\") pod \"multus-2wkff\" (UID: \"e246fd0a-f899-465f-aaeb-661d2a429d06\") " pod="openshift-multus/multus-2wkff"
Sep 15 12:02:25.937479 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.937372    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ee282972-ad6d-4a25-a3f5-218dfe607d6a-host\") pod \"node-ca-mkck2\" (UID: \"ee282972-ad6d-4a25-a3f5-218dfe607d6a\") " pod="openshift-image-registry/node-ca-mkck2"
Sep 15 12:02:25.937479 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.937418    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ee282972-ad6d-4a25-a3f5-218dfe607d6a-host\") pod \"node-ca-mkck2\" (UID: \"ee282972-ad6d-4a25-a3f5-218dfe607d6a\") " pod="openshift-image-registry/node-ca-mkck2"
Sep 15 12:02:25.937479 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.937459    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/365587ef-e477-48b4-aa6d-4e284b0b5951-hosts-file\") pod \"node-resolver-kgjkj\" (UID: \"365587ef-e477-48b4-aa6d-4e284b0b5951\") " pod="openshift-dns/node-resolver-kgjkj"
Sep 15 12:02:25.937891 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.937493    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/07f9ddc4-14d7-4b04-9797-67507d4063bc-dbus\") pod \"global-pull-secret-syncer-g25rd\" (UID: \"07f9ddc4-14d7-4b04-9797-67507d4063bc\") " pod="kube-system/global-pull-secret-syncer-g25rd"
Sep 15 12:02:25.937891 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.937546    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/365587ef-e477-48b4-aa6d-4e284b0b5951-hosts-file\") pod \"node-resolver-kgjkj\" (UID: \"365587ef-e477-48b4-aa6d-4e284b0b5951\") " pod="openshift-dns/node-resolver-kgjkj"
Sep 15 12:02:25.937891 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.937548    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2mdt\" (UniqueName: \"kubernetes.io/projected/365587ef-e477-48b4-aa6d-4e284b0b5951-kube-api-access-z2mdt\") pod \"node-resolver-kgjkj\" (UID: \"365587ef-e477-48b4-aa6d-4e284b0b5951\") " pod="openshift-dns/node-resolver-kgjkj"
Sep 15 12:02:25.937891 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.937577    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e246fd0a-f899-465f-aaeb-661d2a429d06-hostroot\") pod \"multus-2wkff\" (UID: \"e246fd0a-f899-465f-aaeb-661d2a429d06\") " pod="openshift-multus/multus-2wkff"
Sep 15 12:02:25.937891 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.937602    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e246fd0a-f899-465f-aaeb-661d2a429d06-multus-daemon-config\") pod \"multus-2wkff\" (UID: \"e246fd0a-f899-465f-aaeb-661d2a429d06\") " pod="openshift-multus/multus-2wkff"
Sep 15 12:02:25.937891 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.937624    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e246fd0a-f899-465f-aaeb-661d2a429d06-etc-kubernetes\") pod \"multus-2wkff\" (UID: \"e246fd0a-f899-465f-aaeb-661d2a429d06\") " pod="openshift-multus/multus-2wkff"
Sep 15 12:02:25.937891 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.937639    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5vth\" (UniqueName: \"kubernetes.io/projected/e246fd0a-f899-465f-aaeb-661d2a429d06-kube-api-access-k5vth\") pod \"multus-2wkff\" (UID: \"e246fd0a-f899-465f-aaeb-661d2a429d06\") " pod="openshift-multus/multus-2wkff"
Sep 15 12:02:25.937891 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.937655    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e246fd0a-f899-465f-aaeb-661d2a429d06-host-var-lib-cni-bin\") pod \"multus-2wkff\" (UID: \"e246fd0a-f899-465f-aaeb-661d2a429d06\") " pod="openshift-multus/multus-2wkff"
Sep 15 12:02:25.937891 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.937706    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e246fd0a-f899-465f-aaeb-661d2a429d06-etc-kubernetes\") pod \"multus-2wkff\" (UID: \"e246fd0a-f899-465f-aaeb-661d2a429d06\") " pod="openshift-multus/multus-2wkff"
Sep 15 12:02:25.937891 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.937712    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e246fd0a-f899-465f-aaeb-661d2a429d06-multus-conf-dir\") pod \"multus-2wkff\" (UID: \"e246fd0a-f899-465f-aaeb-661d2a429d06\") " pod="openshift-multus/multus-2wkff"
Sep 15 12:02:25.937891 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.937733    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e246fd0a-f899-465f-aaeb-661d2a429d06-hostroot\") pod \"multus-2wkff\" (UID: \"e246fd0a-f899-465f-aaeb-661d2a429d06\") " pod="openshift-multus/multus-2wkff"
Sep 15 12:02:25.937891 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.937745    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9c68d51-9107-4cd4-a57d-6f76c0762106-metrics-certs\") pod \"network-metrics-daemon-bdjwd\" (UID: \"f9c68d51-9107-4cd4-a57d-6f76c0762106\") " pod="openshift-multus/network-metrics-daemon-bdjwd"
Sep 15 12:02:25.937891 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.937758    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/365587ef-e477-48b4-aa6d-4e284b0b5951-tmp-dir\") pod \"node-resolver-kgjkj\" (UID: \"365587ef-e477-48b4-aa6d-4e284b0b5951\") " pod="openshift-dns/node-resolver-kgjkj"
Sep 15 12:02:25.937891 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.937775    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/01c26b58-ebc2-4b0f-96dd-14f8c373b0b5-etc-sysconfig\") pod \"tuned-tnbfp\" (UID: \"01c26b58-ebc2-4b0f-96dd-14f8c373b0b5\") " pod="openshift-cluster-node-tuning-operator/tuned-tnbfp"
Sep 15 12:02:25.937891 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.937801    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/01c26b58-ebc2-4b0f-96dd-14f8c373b0b5-run\") pod \"tuned-tnbfp\" (UID: \"01c26b58-ebc2-4b0f-96dd-14f8c373b0b5\") " pod="openshift-cluster-node-tuning-operator/tuned-tnbfp"
Sep 15 12:02:25.937891 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.937828    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/01c26b58-ebc2-4b0f-96dd-14f8c373b0b5-lib-modules\") pod \"tuned-tnbfp\" (UID: \"01c26b58-ebc2-4b0f-96dd-14f8c373b0b5\") " pod="openshift-cluster-node-tuning-operator/tuned-tnbfp"
Sep 15 12:02:25.937891 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.937835    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e246fd0a-f899-465f-aaeb-661d2a429d06-multus-conf-dir\") pod \"multus-2wkff\" (UID: \"e246fd0a-f899-465f-aaeb-661d2a429d06\") " pod="openshift-multus/multus-2wkff"
Sep 15 12:02:25.937891 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.937853    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ee282972-ad6d-4a25-a3f5-218dfe607d6a-serviceca\") pod \"node-ca-mkck2\" (UID: \"ee282972-ad6d-4a25-a3f5-218dfe607d6a\") " pod="openshift-image-registry/node-ca-mkck2"
Sep 15 12:02:25.937891 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:25.937876    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 12:02:25.938918 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.937886    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e246fd0a-f899-465f-aaeb-661d2a429d06-host-var-lib-cni-bin\") pod \"multus-2wkff\" (UID: \"e246fd0a-f899-465f-aaeb-661d2a429d06\") " pod="openshift-multus/multus-2wkff"
Sep 15 12:02:25.938918 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.937893    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/01c26b58-ebc2-4b0f-96dd-14f8c373b0b5-run\") pod \"tuned-tnbfp\" (UID: \"01c26b58-ebc2-4b0f-96dd-14f8c373b0b5\") " pod="openshift-cluster-node-tuning-operator/tuned-tnbfp"
Sep 15 12:02:25.938918 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.937879    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/07f9ddc4-14d7-4b04-9797-67507d4063bc-original-pull-secret\") pod \"global-pull-secret-syncer-g25rd\" (UID: \"07f9ddc4-14d7-4b04-9797-67507d4063bc\") " pod="kube-system/global-pull-secret-syncer-g25rd"
Sep 15 12:02:25.938918 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.937946    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/01c26b58-ebc2-4b0f-96dd-14f8c373b0b5-etc-sysconfig\") pod \"tuned-tnbfp\" (UID: \"01c26b58-ebc2-4b0f-96dd-14f8c373b0b5\") " pod="openshift-cluster-node-tuning-operator/tuned-tnbfp"
Sep 15 12:02:25.938918 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:25.937961    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9c68d51-9107-4cd4-a57d-6f76c0762106-metrics-certs podName:f9c68d51-9107-4cd4-a57d-6f76c0762106 nodeName:}" failed. No retries permitted until 2026-09-15 12:02:26.437935783 +0000 UTC m=+3.228858275 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9c68d51-9107-4cd4-a57d-6f76c0762106-metrics-certs") pod "network-metrics-daemon-bdjwd" (UID: "f9c68d51-9107-4cd4-a57d-6f76c0762106") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 12:02:25.938918 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.938000    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e246fd0a-f899-465f-aaeb-661d2a429d06-multus-socket-dir-parent\") pod \"multus-2wkff\" (UID: \"e246fd0a-f899-465f-aaeb-661d2a429d06\") " pod="openshift-multus/multus-2wkff"
Sep 15 12:02:25.938918 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.938029    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e246fd0a-f899-465f-aaeb-661d2a429d06-host-var-lib-kubelet\") pod \"multus-2wkff\" (UID: \"e246fd0a-f899-465f-aaeb-661d2a429d06\") " pod="openshift-multus/multus-2wkff"
Sep 15 12:02:25.938918 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.938039    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/01c26b58-ebc2-4b0f-96dd-14f8c373b0b5-lib-modules\") pod \"tuned-tnbfp\" (UID: \"01c26b58-ebc2-4b0f-96dd-14f8c373b0b5\") " pod="openshift-cluster-node-tuning-operator/tuned-tnbfp"
Sep 15 12:02:25.938918 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.938056    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/01c26b58-ebc2-4b0f-96dd-14f8c373b0b5-etc-modprobe-d\") pod \"tuned-tnbfp\" (UID: \"01c26b58-ebc2-4b0f-96dd-14f8c373b0b5\") " pod="openshift-cluster-node-tuning-operator/tuned-tnbfp"
Sep 15 12:02:25.938918 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.938087    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e246fd0a-f899-465f-aaeb-661d2a429d06-cni-binary-copy\") pod \"multus-2wkff\" (UID: \"e246fd0a-f899-465f-aaeb-661d2a429d06\") " pod="openshift-multus/multus-2wkff"
Sep 15 12:02:25.938918 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.938119    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/01c26b58-ebc2-4b0f-96dd-14f8c373b0b5-etc-tuned\") pod \"tuned-tnbfp\" (UID: \"01c26b58-ebc2-4b0f-96dd-14f8c373b0b5\") " pod="openshift-cluster-node-tuning-operator/tuned-tnbfp"
Sep 15 12:02:25.938918 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.938133    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e246fd0a-f899-465f-aaeb-661d2a429d06-host-var-lib-kubelet\") pod \"multus-2wkff\" (UID: \"e246fd0a-f899-465f-aaeb-661d2a429d06\") " pod="openshift-multus/multus-2wkff"
Sep 15 12:02:25.938918 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.938152    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xnbb\" (UniqueName: \"kubernetes.io/projected/ee282972-ad6d-4a25-a3f5-218dfe607d6a-kube-api-access-2xnbb\") pod \"node-ca-mkck2\" (UID: \"ee282972-ad6d-4a25-a3f5-218dfe607d6a\") " pod="openshift-image-registry/node-ca-mkck2"
Sep 15 12:02:25.938918 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:25.938184    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 12:02:25.938918 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.938204    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e246fd0a-f899-465f-aaeb-661d2a429d06-multus-socket-dir-parent\") pod \"multus-2wkff\" (UID: \"e246fd0a-f899-465f-aaeb-661d2a429d06\") " pod="openshift-multus/multus-2wkff"
Sep 15 12:02:25.938918 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:25.938267    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07f9ddc4-14d7-4b04-9797-67507d4063bc-original-pull-secret podName:07f9ddc4-14d7-4b04-9797-67507d4063bc nodeName:}" failed. No retries permitted until 2026-09-15 12:02:26.438248305 +0000 UTC m=+3.229170811 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/07f9ddc4-14d7-4b04-9797-67507d4063bc-original-pull-secret") pod "global-pull-secret-syncer-g25rd" (UID: "07f9ddc4-14d7-4b04-9797-67507d4063bc") : object "kube-system"/"original-pull-secret" not registered
Sep 15 12:02:25.939760 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.938184    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm9js\" (UniqueName: \"kubernetes.io/projected/090605d8-b2cc-4a2f-9f09-c22538254886-kube-api-access-zm9js\") pod \"network-check-target-wrxv2\" (UID: \"090605d8-b2cc-4a2f-9f09-c22538254886\") " pod="openshift-network-diagnostics/network-check-target-wrxv2"
Sep 15 12:02:25.939760 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.938309    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/01c26b58-ebc2-4b0f-96dd-14f8c373b0b5-sys\") pod \"tuned-tnbfp\" (UID: \"01c26b58-ebc2-4b0f-96dd-14f8c373b0b5\") " pod="openshift-cluster-node-tuning-operator/tuned-tnbfp"
Sep 15 12:02:25.939760 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.938339    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ee282972-ad6d-4a25-a3f5-218dfe607d6a-serviceca\") pod \"node-ca-mkck2\" (UID: \"ee282972-ad6d-4a25-a3f5-218dfe607d6a\") " pod="openshift-image-registry/node-ca-mkck2"
Sep 15 12:02:25.939760 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.938351    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/01c26b58-ebc2-4b0f-96dd-14f8c373b0b5-etc-modprobe-d\") pod \"tuned-tnbfp\" (UID: \"01c26b58-ebc2-4b0f-96dd-14f8c373b0b5\") " pod="openshift-cluster-node-tuning-operator/tuned-tnbfp"
Sep 15 12:02:25.939760 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.938353    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/01c26b58-ebc2-4b0f-96dd-14f8c373b0b5-var-lib-kubelet\") pod \"tuned-tnbfp\" (UID: \"01c26b58-ebc2-4b0f-96dd-14f8c373b0b5\") " pod="openshift-cluster-node-tuning-operator/tuned-tnbfp"
Sep 15 12:02:25.939760 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.938359    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e246fd0a-f899-465f-aaeb-661d2a429d06-multus-daemon-config\") pod \"multus-2wkff\" (UID: \"e246fd0a-f899-465f-aaeb-661d2a429d06\") " pod="openshift-multus/multus-2wkff"
Sep 15 12:02:25.939760 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.938395    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/01c26b58-ebc2-4b0f-96dd-14f8c373b0b5-host\") pod \"tuned-tnbfp\" (UID: \"01c26b58-ebc2-4b0f-96dd-14f8c373b0b5\") " pod="openshift-cluster-node-tuning-operator/tuned-tnbfp"
Sep 15 12:02:25.939760 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.938408    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/01c26b58-ebc2-4b0f-96dd-14f8c373b0b5-sys\") pod \"tuned-tnbfp\" (UID: \"01c26b58-ebc2-4b0f-96dd-14f8c373b0b5\") " pod="openshift-cluster-node-tuning-operator/tuned-tnbfp"
Sep 15 12:02:25.939760 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.938407    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/01c26b58-ebc2-4b0f-96dd-14f8c373b0b5-var-lib-kubelet\") pod \"tuned-tnbfp\" (UID: \"01c26b58-ebc2-4b0f-96dd-14f8c373b0b5\") " pod="openshift-cluster-node-tuning-operator/tuned-tnbfp"
Sep 15 12:02:25.939760 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.938437    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e246fd0a-f899-465f-aaeb-661d2a429d06-host-run-k8s-cni-cncf-io\") pod \"multus-2wkff\" (UID: \"e246fd0a-f899-465f-aaeb-661d2a429d06\") " pod="openshift-multus/multus-2wkff"
Sep 15 12:02:25.939760 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.938463    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/01c26b58-ebc2-4b0f-96dd-14f8c373b0b5-etc-sysctl-d\") pod \"tuned-tnbfp\" (UID: \"01c26b58-ebc2-4b0f-96dd-14f8c373b0b5\") " pod="openshift-cluster-node-tuning-operator/tuned-tnbfp"
Sep 15 12:02:25.939760 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.938478    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/01c26b58-ebc2-4b0f-96dd-14f8c373b0b5-tmp\") pod \"tuned-tnbfp\" (UID: \"01c26b58-ebc2-4b0f-96dd-14f8c373b0b5\") " pod="openshift-cluster-node-tuning-operator/tuned-tnbfp"
Sep 15 12:02:25.939760 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.938479    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/01c26b58-ebc2-4b0f-96dd-14f8c373b0b5-host\") pod \"tuned-tnbfp\" (UID: \"01c26b58-ebc2-4b0f-96dd-14f8c373b0b5\") " pod="openshift-cluster-node-tuning-operator/tuned-tnbfp"
Sep 15 12:02:25.939760 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.938506    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/07f9ddc4-14d7-4b04-9797-67507d4063bc-kubelet-config\") pod \"global-pull-secret-syncer-g25rd\" (UID: \"07f9ddc4-14d7-4b04-9797-67507d4063bc\") " pod="kube-system/global-pull-secret-syncer-g25rd"
Sep 15 12:02:25.939760 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.938525    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e246fd0a-f899-465f-aaeb-661d2a429d06-host-run-k8s-cni-cncf-io\") pod \"multus-2wkff\" (UID: \"e246fd0a-f899-465f-aaeb-661d2a429d06\") " pod="openshift-multus/multus-2wkff"
Sep 15 12:02:25.939760 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.938531    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e246fd0a-f899-465f-aaeb-661d2a429d06-cnibin\") pod \"multus-2wkff\" (UID: \"e246fd0a-f899-465f-aaeb-661d2a429d06\") " pod="openshift-multus/multus-2wkff"
Sep 15 12:02:25.939760 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.938558    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/01c26b58-ebc2-4b0f-96dd-14f8c373b0b5-etc-kubernetes\") pod \"tuned-tnbfp\" (UID: \"01c26b58-ebc2-4b0f-96dd-14f8c373b0b5\") " pod="openshift-cluster-node-tuning-operator/tuned-tnbfp"
Sep 15 12:02:25.939760 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.938585    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6p4hh\" (UniqueName: \"kubernetes.io/projected/01c26b58-ebc2-4b0f-96dd-14f8c373b0b5-kube-api-access-6p4hh\") pod \"tuned-tnbfp\" (UID: \"01c26b58-ebc2-4b0f-96dd-14f8c373b0b5\") " pod="openshift-cluster-node-tuning-operator/tuned-tnbfp"
Sep 15 12:02:25.940695 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.938612    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e246fd0a-f899-465f-aaeb-661d2a429d06-host-run-multus-certs\") pod \"multus-2wkff\" (UID: \"e246fd0a-f899-465f-aaeb-661d2a429d06\") " pod="openshift-multus/multus-2wkff"
Sep 15 12:02:25.940695 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.938621    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/01c26b58-ebc2-4b0f-96dd-14f8c373b0b5-etc-sysctl-d\") pod \"tuned-tnbfp\" (UID: \"01c26b58-ebc2-4b0f-96dd-14f8c373b0b5\") " pod="openshift-cluster-node-tuning-operator/tuned-tnbfp"
Sep 15 12:02:25.940695 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.938645    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/01c26b58-ebc2-4b0f-96dd-14f8c373b0b5-etc-systemd\") pod \"tuned-tnbfp\" (UID: \"01c26b58-ebc2-4b0f-96dd-14f8c373b0b5\") " pod="openshift-cluster-node-tuning-operator/tuned-tnbfp"
Sep 15 12:02:25.940695 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.938653    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e246fd0a-f899-465f-aaeb-661d2a429d06-cnibin\") pod \"multus-2wkff\" (UID: \"e246fd0a-f899-465f-aaeb-661d2a429d06\") " pod="openshift-multus/multus-2wkff"
Sep 15 12:02:25.940695 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.938668    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/07f9ddc4-14d7-4b04-9797-67507d4063bc-kubelet-config\") pod \"global-pull-secret-syncer-g25rd\" (UID: \"07f9ddc4-14d7-4b04-9797-67507d4063bc\") " pod="kube-system/global-pull-secret-syncer-g25rd"
Sep 15 12:02:25.940695 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.938671    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e246fd0a-f899-465f-aaeb-661d2a429d06-multus-cni-dir\") pod \"multus-2wkff\" (UID: \"e246fd0a-f899-465f-aaeb-661d2a429d06\") " pod="openshift-multus/multus-2wkff"
Sep 15 12:02:25.940695 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.938707    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/01c26b58-ebc2-4b0f-96dd-14f8c373b0b5-etc-sysctl-conf\") pod \"tuned-tnbfp\" (UID: \"01c26b58-ebc2-4b0f-96dd-14f8c373b0b5\") " pod="openshift-cluster-node-tuning-operator/tuned-tnbfp"
Sep 15 12:02:25.940695 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.938733    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/01c26b58-ebc2-4b0f-96dd-14f8c373b0b5-etc-systemd\") pod \"tuned-tnbfp\" (UID: \"01c26b58-ebc2-4b0f-96dd-14f8c373b0b5\") " pod="openshift-cluster-node-tuning-operator/tuned-tnbfp"
Sep 15 12:02:25.940695 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.938745    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e246fd0a-f899-465f-aaeb-661d2a429d06-system-cni-dir\") pod \"multus-2wkff\" (UID: \"e246fd0a-f899-465f-aaeb-661d2a429d06\") " pod="openshift-multus/multus-2wkff"
Sep 15 12:02:25.940695 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.938852    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e246fd0a-f899-465f-aaeb-661d2a429d06-system-cni-dir\") pod \"multus-2wkff\" (UID: \"e246fd0a-f899-465f-aaeb-661d2a429d06\") " pod="openshift-multus/multus-2wkff"
Sep 15 12:02:25.940695 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.938852    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/01c26b58-ebc2-4b0f-96dd-14f8c373b0b5-etc-kubernetes\") pod \"tuned-tnbfp\" (UID: \"01c26b58-ebc2-4b0f-96dd-14f8c373b0b5\") " pod="openshift-cluster-node-tuning-operator/tuned-tnbfp"
Sep 15 12:02:25.940695 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.938896    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e246fd0a-f899-465f-aaeb-661d2a429d06-multus-cni-dir\") pod \"multus-2wkff\" (UID: \"e246fd0a-f899-465f-aaeb-661d2a429d06\") " pod="openshift-multus/multus-2wkff"
Sep 15 12:02:25.940695 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.938925    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e246fd0a-f899-465f-aaeb-661d2a429d06-host-run-multus-certs\") pod \"multus-2wkff\" (UID: \"e246fd0a-f899-465f-aaeb-661d2a429d06\") " pod="openshift-multus/multus-2wkff"
Sep 15 12:02:25.940695 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.938994    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/01c26b58-ebc2-4b0f-96dd-14f8c373b0b5-etc-sysctl-conf\") pod \"tuned-tnbfp\" (UID: \"01c26b58-ebc2-4b0f-96dd-14f8c373b0b5\") " pod="openshift-cluster-node-tuning-operator/tuned-tnbfp"
Sep 15 12:02:25.940695 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.939005    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e246fd0a-f899-465f-aaeb-661d2a429d06-cni-binary-copy\") pod \"multus-2wkff\" (UID: \"e246fd0a-f899-465f-aaeb-661d2a429d06\") " pod="openshift-multus/multus-2wkff"
Sep 15 12:02:25.941435 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.941409    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/01c26b58-ebc2-4b0f-96dd-14f8c373b0b5-tmp\") pod \"tuned-tnbfp\" (UID: \"01c26b58-ebc2-4b0f-96dd-14f8c373b0b5\") " pod="openshift-cluster-node-tuning-operator/tuned-tnbfp"
Sep 15 12:02:25.941511 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.941482    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/01c26b58-ebc2-4b0f-96dd-14f8c373b0b5-etc-tuned\") pod \"tuned-tnbfp\" (UID: \"01c26b58-ebc2-4b0f-96dd-14f8c373b0b5\") " pod="openshift-cluster-node-tuning-operator/tuned-tnbfp"
Sep 15 12:02:25.949145 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:25.948800    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 12:02:25.949145 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:25.948832    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 12:02:25.949145 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:25.948853    2555 projected.go:196] Error preparing data for projected volume kube-api-access-zm9js for pod openshift-network-diagnostics/network-check-target-wrxv2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 12:02:25.949145 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:25.948973    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/090605d8-b2cc-4a2f-9f09-c22538254886-kube-api-access-zm9js podName:090605d8-b2cc-4a2f-9f09-c22538254886 nodeName:}" failed. No retries permitted until 2026-09-15 12:02:26.448926028 +0000 UTC m=+3.239848533 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-zm9js" (UniqueName: "kubernetes.io/projected/090605d8-b2cc-4a2f-9f09-c22538254886-kube-api-access-zm9js") pod "network-check-target-wrxv2" (UID: "090605d8-b2cc-4a2f-9f09-c22538254886") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 12:02:25.952252 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.952220    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xnbb\" (UniqueName: \"kubernetes.io/projected/ee282972-ad6d-4a25-a3f5-218dfe607d6a-kube-api-access-2xnbb\") pod \"node-ca-mkck2\" (UID: \"ee282972-ad6d-4a25-a3f5-218dfe607d6a\") " pod="openshift-image-registry/node-ca-mkck2"
Sep 15 12:02:25.954092 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.954060    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5vth\" (UniqueName: \"kubernetes.io/projected/e246fd0a-f899-465f-aaeb-661d2a429d06-kube-api-access-k5vth\") pod \"multus-2wkff\" (UID: \"e246fd0a-f899-465f-aaeb-661d2a429d06\") " pod="openshift-multus/multus-2wkff"
Sep 15 12:02:25.954208 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.954154    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wsgh\" (UniqueName: \"kubernetes.io/projected/f9c68d51-9107-4cd4-a57d-6f76c0762106-kube-api-access-8wsgh\") pod \"network-metrics-daemon-bdjwd\" (UID: \"f9c68d51-9107-4cd4-a57d-6f76c0762106\") " pod="openshift-multus/network-metrics-daemon-bdjwd"
Sep 15 12:02:25.955463 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.955435    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2mdt\" (UniqueName: \"kubernetes.io/projected/365587ef-e477-48b4-aa6d-4e284b0b5951-kube-api-access-z2mdt\") pod \"node-resolver-kgjkj\" (UID: \"365587ef-e477-48b4-aa6d-4e284b0b5951\") " pod="openshift-dns/node-resolver-kgjkj"
Sep 15 12:02:25.955889 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:25.955794    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6p4hh\" (UniqueName: \"kubernetes.io/projected/01c26b58-ebc2-4b0f-96dd-14f8c373b0b5-kube-api-access-6p4hh\") pod \"tuned-tnbfp\" (UID: \"01c26b58-ebc2-4b0f-96dd-14f8c373b0b5\") " pod="openshift-cluster-node-tuning-operator/tuned-tnbfp"
Sep 15 12:02:26.029164 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:26.029117    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-c47md"
Sep 15 12:02:26.039514 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:26.039477    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qjwgl"
Sep 15 12:02:26.052991 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:26.052961    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-p4rvp"
Sep 15 12:02:26.062078 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:26.062047    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7fbc7"
Sep 15 12:02:26.079138 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:26.079104    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-k62zz"
Sep 15 12:02:26.085444 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:26.085420    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2wkff"
Sep 15 12:02:26.091458 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:26.091427    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mkck2"
Sep 15 12:02:26.103457 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:26.103428    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-kgjkj"
Sep 15 12:02:26.111638 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:26.111602    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-tnbfp"
Sep 15 12:02:26.444147 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:26.444056    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9c68d51-9107-4cd4-a57d-6f76c0762106-metrics-certs\") pod \"network-metrics-daemon-bdjwd\" (UID: \"f9c68d51-9107-4cd4-a57d-6f76c0762106\") " pod="openshift-multus/network-metrics-daemon-bdjwd"
Sep 15 12:02:26.444147 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:26.444099    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/07f9ddc4-14d7-4b04-9797-67507d4063bc-original-pull-secret\") pod \"global-pull-secret-syncer-g25rd\" (UID: \"07f9ddc4-14d7-4b04-9797-67507d4063bc\") " pod="kube-system/global-pull-secret-syncer-g25rd"
Sep 15 12:02:26.444420 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:26.444218    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 12:02:26.444420 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:26.444232    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 12:02:26.444420 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:26.444282    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07f9ddc4-14d7-4b04-9797-67507d4063bc-original-pull-secret podName:07f9ddc4-14d7-4b04-9797-67507d4063bc nodeName:}" failed. No retries permitted until 2026-09-15 12:02:27.444261977 +0000 UTC m=+4.235184479 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/07f9ddc4-14d7-4b04-9797-67507d4063bc-original-pull-secret") pod "global-pull-secret-syncer-g25rd" (UID: "07f9ddc4-14d7-4b04-9797-67507d4063bc") : object "kube-system"/"original-pull-secret" not registered
Sep 15 12:02:26.444420 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:26.444310    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9c68d51-9107-4cd4-a57d-6f76c0762106-metrics-certs podName:f9c68d51-9107-4cd4-a57d-6f76c0762106 nodeName:}" failed. No retries permitted until 2026-09-15 12:02:27.44429065 +0000 UTC m=+4.235213157 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9c68d51-9107-4cd4-a57d-6f76c0762106-metrics-certs") pod "network-metrics-daemon-bdjwd" (UID: "f9c68d51-9107-4cd4-a57d-6f76c0762106") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 12:02:26.544606 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:26.544566    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm9js\" (UniqueName: \"kubernetes.io/projected/090605d8-b2cc-4a2f-9f09-c22538254886-kube-api-access-zm9js\") pod \"network-check-target-wrxv2\" (UID: \"090605d8-b2cc-4a2f-9f09-c22538254886\") " pod="openshift-network-diagnostics/network-check-target-wrxv2"
Sep 15 12:02:26.544802 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:26.544767    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 12:02:26.544802 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:26.544796    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 12:02:26.544899 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:26.544807    2555 projected.go:196] Error preparing data for projected volume kube-api-access-zm9js for pod openshift-network-diagnostics/network-check-target-wrxv2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 12:02:26.544899 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:26.544873    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/090605d8-b2cc-4a2f-9f09-c22538254886-kube-api-access-zm9js podName:090605d8-b2cc-4a2f-9f09-c22538254886 nodeName:}" failed. No retries permitted until 2026-09-15 12:02:27.5448541 +0000 UTC m=+4.335776596 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-zm9js" (UniqueName: "kubernetes.io/projected/090605d8-b2cc-4a2f-9f09-c22538254886-kube-api-access-zm9js") pod "network-check-target-wrxv2" (UID: "090605d8-b2cc-4a2f-9f09-c22538254886") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 12:02:26.788860 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:26.788744    2555 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-14 11:57:26 +0000 UTC" deadline="2028-04-08 08:54:58.8921441 +0000 UTC"
Sep 15 12:02:26.788860 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:26.788804    2555 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13700h52m32.103343819s"
Sep 15 12:02:27.114396 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:27.114145    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode246fd0a_f899_465f_aaeb_661d2a429d06.slice/crio-4bbf399b48b20516d33ab7e541b2b532910c5c65ba82678fa14068198cecba43 WatchSource:0}: Error finding container 4bbf399b48b20516d33ab7e541b2b532910c5c65ba82678fa14068198cecba43: Status 404 returned error can't find the container with id 4bbf399b48b20516d33ab7e541b2b532910c5c65ba82678fa14068198cecba43
Sep 15 12:02:27.118178 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:27.118026    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25dcd72f_7a70_4cd8_ab3c_964f525bd756.slice/crio-38eabb7f3fb298df5bb5225587a02c88dd93b74d2a4c808d5fb4d183c8be8f10 WatchSource:0}: Error finding container 38eabb7f3fb298df5bb5225587a02c88dd93b74d2a4c808d5fb4d183c8be8f10: Status 404 returned error can't find the container with id 38eabb7f3fb298df5bb5225587a02c88dd93b74d2a4c808d5fb4d183c8be8f10
Sep 15 12:02:27.120840 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:27.120766    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95528bd1_4100_43e9_bc0a_2e56670bc84f.slice/crio-0b9c45341afa62738e4487311118f0aeed2c00d01500aa03d02bf238f46ebf8a WatchSource:0}: Error finding container 0b9c45341afa62738e4487311118f0aeed2c00d01500aa03d02bf238f46ebf8a: Status 404 returned error can't find the container with id 0b9c45341afa62738e4487311118f0aeed2c00d01500aa03d02bf238f46ebf8a
Sep 15 12:02:27.120915 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:27.120841    2555 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 12:02:27.122190 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:27.122164    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9b05b4f_57d5_454e_b739_2778ddcab5fb.slice/crio-f24a9f98c9dc33675615b3967f045d580600c30b0b92f7b86ad3b45beabd9f5c WatchSource:0}: Error finding container f24a9f98c9dc33675615b3967f045d580600c30b0b92f7b86ad3b45beabd9f5c: Status 404 returned error can't find the container with id f24a9f98c9dc33675615b3967f045d580600c30b0b92f7b86ad3b45beabd9f5c
Sep 15 12:02:27.123869 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:27.123847    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee282972_ad6d_4a25_a3f5_218dfe607d6a.slice/crio-391b2831b7e104747c036697aa3d3b57ed84de1b37da5a8857cf79c8c98768e9 WatchSource:0}: Error finding container 391b2831b7e104747c036697aa3d3b57ed84de1b37da5a8857cf79c8c98768e9: Status 404 returned error can't find the container with id 391b2831b7e104747c036697aa3d3b57ed84de1b37da5a8857cf79c8c98768e9
Sep 15 12:02:27.125069 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:27.125043    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fa9c45a_2974_45d2_82c2_23a63b9f0662.slice/crio-6f6214499118f076c8d567ff26ba6537300a2a874fdd838782162241f57540f4 WatchSource:0}: Error finding container 6f6214499118f076c8d567ff26ba6537300a2a874fdd838782162241f57540f4: Status 404 returned error can't find the container with id 6f6214499118f076c8d567ff26ba6537300a2a874fdd838782162241f57540f4
Sep 15 12:02:27.127786 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:27.127696    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod365587ef_e477_48b4_aa6d_4e284b0b5951.slice/crio-5b38b2c0e05ee845dba9e6a737346a6dfa01005f9eacb9853033b6529f83995a WatchSource:0}: Error finding container 5b38b2c0e05ee845dba9e6a737346a6dfa01005f9eacb9853033b6529f83995a: Status 404 returned error can't find the container with id 5b38b2c0e05ee845dba9e6a737346a6dfa01005f9eacb9853033b6529f83995a
Sep 15 12:02:27.129630 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:27.129324    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01c26b58_ebc2_4b0f_96dd_14f8c373b0b5.slice/crio-8bf88e952fa00407ded881cb61e1bfab062314cd0a6b7bf527c84fbec65e9af2 WatchSource:0}: Error finding container 8bf88e952fa00407ded881cb61e1bfab062314cd0a6b7bf527c84fbec65e9af2: Status 404 returned error can't find the container with id 8bf88e952fa00407ded881cb61e1bfab062314cd0a6b7bf527c84fbec65e9af2
Sep 15 12:02:27.132904 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:02:27.132860    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5be458c0_7237_4003_b4b9_1218ac635869.slice/crio-e28ad05a73871127d0cc7f5db7aade3980e0fee7087cf703a3439f34fcc54da4 WatchSource:0}: Error finding container e28ad05a73871127d0cc7f5db7aade3980e0fee7087cf703a3439f34fcc54da4: Status 404 returned error can't find the container with id e28ad05a73871127d0cc7f5db7aade3980e0fee7087cf703a3439f34fcc54da4
Sep 15 12:02:27.451402 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:27.451358    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/07f9ddc4-14d7-4b04-9797-67507d4063bc-original-pull-secret\") pod \"global-pull-secret-syncer-g25rd\" (UID: \"07f9ddc4-14d7-4b04-9797-67507d4063bc\") " pod="kube-system/global-pull-secret-syncer-g25rd"
Sep 15 12:02:27.451571 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:27.451462    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 12:02:27.451571 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:27.451487    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9c68d51-9107-4cd4-a57d-6f76c0762106-metrics-certs\") pod \"network-metrics-daemon-bdjwd\" (UID: \"f9c68d51-9107-4cd4-a57d-6f76c0762106\") " pod="openshift-multus/network-metrics-daemon-bdjwd"
Sep 15 12:02:27.451571 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:27.451546    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07f9ddc4-14d7-4b04-9797-67507d4063bc-original-pull-secret podName:07f9ddc4-14d7-4b04-9797-67507d4063bc nodeName:}" failed. No retries permitted until 2026-09-15 12:02:29.451524881 +0000 UTC m=+6.242447397 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/07f9ddc4-14d7-4b04-9797-67507d4063bc-original-pull-secret") pod "global-pull-secret-syncer-g25rd" (UID: "07f9ddc4-14d7-4b04-9797-67507d4063bc") : object "kube-system"/"original-pull-secret" not registered
Sep 15 12:02:27.451571 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:27.451551    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 12:02:27.451728 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:27.451604    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9c68d51-9107-4cd4-a57d-6f76c0762106-metrics-certs podName:f9c68d51-9107-4cd4-a57d-6f76c0762106 nodeName:}" failed. No retries permitted until 2026-09-15 12:02:29.451592935 +0000 UTC m=+6.242515432 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9c68d51-9107-4cd4-a57d-6f76c0762106-metrics-certs") pod "network-metrics-daemon-bdjwd" (UID: "f9c68d51-9107-4cd4-a57d-6f76c0762106") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 12:02:27.552580 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:27.552426    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm9js\" (UniqueName: \"kubernetes.io/projected/090605d8-b2cc-4a2f-9f09-c22538254886-kube-api-access-zm9js\") pod \"network-check-target-wrxv2\" (UID: \"090605d8-b2cc-4a2f-9f09-c22538254886\") " pod="openshift-network-diagnostics/network-check-target-wrxv2"
Sep 15 12:02:27.552754 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:27.552586    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 12:02:27.552754 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:27.552610    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 12:02:27.552754 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:27.552625    2555 projected.go:196] Error preparing data for projected volume kube-api-access-zm9js for pod openshift-network-diagnostics/network-check-target-wrxv2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 12:02:27.552754 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:27.552695    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/090605d8-b2cc-4a2f-9f09-c22538254886-kube-api-access-zm9js podName:090605d8-b2cc-4a2f-9f09-c22538254886 nodeName:}" failed. No retries permitted until 2026-09-15 12:02:29.552676206 +0000 UTC m=+6.343598701 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-zm9js" (UniqueName: "kubernetes.io/projected/090605d8-b2cc-4a2f-9f09-c22538254886-kube-api-access-zm9js") pod "network-check-target-wrxv2" (UID: "090605d8-b2cc-4a2f-9f09-c22538254886") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 12:02:27.789085 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:27.788977    2555 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-14 11:57:26 +0000 UTC" deadline="2028-04-05 11:56:29.820757632 +0000 UTC"
Sep 15 12:02:27.789085 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:27.789016    2555 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13631h54m2.03174632s"
Sep 15 12:02:27.818044 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:27.816806    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wrxv2"
Sep 15 12:02:27.818044 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:27.816926    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wrxv2" podUID="090605d8-b2cc-4a2f-9f09-c22538254886"
Sep 15 12:02:27.818044 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:27.817691    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g25rd"
Sep 15 12:02:27.818044 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:27.817809    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g25rd" podUID="07f9ddc4-14d7-4b04-9797-67507d4063bc"
Sep 15 12:02:27.818044 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:27.817882    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdjwd"
Sep 15 12:02:27.818044 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:27.817962    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdjwd" podUID="f9c68d51-9107-4cd4-a57d-6f76c0762106"
Sep 15 12:02:27.845970 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:27.845929    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-17.ec2.internal" event={"ID":"99788892a4247a0a40d9fb25d8f545f5","Type":"ContainerStarted","Data":"c5e42e056f591f0e2999a8c77303bda16201840af293f513afab397d9d78badd"}
Sep 15 12:02:27.859019 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:27.858953    2555 generic.go:356] "Generic (PLEG): container finished" podID="81882ce609db62ac71292b6338f62ad1" containerID="9bbb2c7e349d98b69673c8fe1236156e407ad3654375b3c672a56289d3afc8a2" exitCode=0
Sep 15 12:02:27.859191 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:27.859102    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-17.ec2.internal" event={"ID":"81882ce609db62ac71292b6338f62ad1","Type":"ContainerDied","Data":"9bbb2c7e349d98b69673c8fe1236156e407ad3654375b3c672a56289d3afc8a2"}
Sep 15 12:02:27.875118 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:27.875011    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-c47md" event={"ID":"5be458c0-7237-4003-b4b9-1218ac635869","Type":"ContainerStarted","Data":"e28ad05a73871127d0cc7f5db7aade3980e0fee7087cf703a3439f34fcc54da4"}
Sep 15 12:02:27.886374 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:27.886226    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-17.ec2.internal" podStartSLOduration=2.8862049279999997 podStartE2EDuration="2.886204928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 12:02:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:02:27.867416656 +0000 UTC m=+4.658339184" watchObservedRunningTime="2026-09-15 12:02:27.886204928 +0000 UTC m=+4.677127436"
Sep 15 12:02:27.889673 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:27.889639    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-kgjkj" event={"ID":"365587ef-e477-48b4-aa6d-4e284b0b5951","Type":"ContainerStarted","Data":"5b38b2c0e05ee845dba9e6a737346a6dfa01005f9eacb9853033b6529f83995a"}
Sep 15 12:02:27.896806 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:27.896769    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qjwgl" event={"ID":"0fa9c45a-2974-45d2-82c2-23a63b9f0662","Type":"ContainerStarted","Data":"6f6214499118f076c8d567ff26ba6537300a2a874fdd838782162241f57540f4"}
Sep 15 12:02:27.914767 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:27.914695    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mkck2" event={"ID":"ee282972-ad6d-4a25-a3f5-218dfe607d6a","Type":"ContainerStarted","Data":"391b2831b7e104747c036697aa3d3b57ed84de1b37da5a8857cf79c8c98768e9"}
Sep 15 12:02:27.932937 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:27.932891    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k62zz" event={"ID":"95528bd1-4100-43e9-bc0a-2e56670bc84f","Type":"ContainerStarted","Data":"0b9c45341afa62738e4487311118f0aeed2c00d01500aa03d02bf238f46ebf8a"}
Sep 15 12:02:27.962082 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:27.962030    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-tnbfp" event={"ID":"01c26b58-ebc2-4b0f-96dd-14f8c373b0b5","Type":"ContainerStarted","Data":"8bf88e952fa00407ded881cb61e1bfab062314cd0a6b7bf527c84fbec65e9af2"}
Sep 15 12:02:27.973696 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:27.973615    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p4rvp" event={"ID":"b9b05b4f-57d5-454e-b739-2778ddcab5fb","Type":"ContainerStarted","Data":"f24a9f98c9dc33675615b3967f045d580600c30b0b92f7b86ad3b45beabd9f5c"}
Sep 15 12:02:27.975191 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:27.975128    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7fbc7" event={"ID":"25dcd72f-7a70-4cd8-ab3c-964f525bd756","Type":"ContainerStarted","Data":"38eabb7f3fb298df5bb5225587a02c88dd93b74d2a4c808d5fb4d183c8be8f10"}
Sep 15 12:02:27.977303 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:27.977262    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2wkff" event={"ID":"e246fd0a-f899-465f-aaeb-661d2a429d06","Type":"ContainerStarted","Data":"4bbf399b48b20516d33ab7e541b2b532910c5c65ba82678fa14068198cecba43"}
Sep 15 12:02:28.991229 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:28.991185    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-17.ec2.internal" event={"ID":"81882ce609db62ac71292b6338f62ad1","Type":"ContainerStarted","Data":"72c58521becc3b37449256450737c564c4f4759db4e06dd72eb1f6a4ff670706"}
Sep 15 12:02:29.010262 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:29.010198    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-17.ec2.internal" podStartSLOduration=4.010174847 podStartE2EDuration="4.010174847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 12:02:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:02:29.00888394 +0000 UTC m=+5.799806457" watchObservedRunningTime="2026-09-15 12:02:29.010174847 +0000 UTC m=+5.801097363"
Sep 15 12:02:29.469243 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:29.468389    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9c68d51-9107-4cd4-a57d-6f76c0762106-metrics-certs\") pod \"network-metrics-daemon-bdjwd\" (UID: \"f9c68d51-9107-4cd4-a57d-6f76c0762106\") " pod="openshift-multus/network-metrics-daemon-bdjwd"
Sep 15 12:02:29.469243 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:29.468450    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/07f9ddc4-14d7-4b04-9797-67507d4063bc-original-pull-secret\") pod \"global-pull-secret-syncer-g25rd\" (UID: \"07f9ddc4-14d7-4b04-9797-67507d4063bc\") " pod="kube-system/global-pull-secret-syncer-g25rd"
Sep 15 12:02:29.469243 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:29.468606    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 12:02:29.469243 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:29.468675    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07f9ddc4-14d7-4b04-9797-67507d4063bc-original-pull-secret podName:07f9ddc4-14d7-4b04-9797-67507d4063bc nodeName:}" failed. No retries permitted until 2026-09-15 12:02:33.468654856 +0000 UTC m=+10.259577350 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/07f9ddc4-14d7-4b04-9797-67507d4063bc-original-pull-secret") pod "global-pull-secret-syncer-g25rd" (UID: "07f9ddc4-14d7-4b04-9797-67507d4063bc") : object "kube-system"/"original-pull-secret" not registered
Sep 15 12:02:29.469243 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:29.469100    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 12:02:29.469243 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:29.469154    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9c68d51-9107-4cd4-a57d-6f76c0762106-metrics-certs podName:f9c68d51-9107-4cd4-a57d-6f76c0762106 nodeName:}" failed. No retries permitted until 2026-09-15 12:02:33.469137854 +0000 UTC m=+10.260060345 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9c68d51-9107-4cd4-a57d-6f76c0762106-metrics-certs") pod "network-metrics-daemon-bdjwd" (UID: "f9c68d51-9107-4cd4-a57d-6f76c0762106") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 12:02:29.569617 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:29.569575    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm9js\" (UniqueName: \"kubernetes.io/projected/090605d8-b2cc-4a2f-9f09-c22538254886-kube-api-access-zm9js\") pod \"network-check-target-wrxv2\" (UID: \"090605d8-b2cc-4a2f-9f09-c22538254886\") " pod="openshift-network-diagnostics/network-check-target-wrxv2"
Sep 15 12:02:29.569818 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:29.569799    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 12:02:29.569891 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:29.569830    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 12:02:29.569891 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:29.569845    2555 projected.go:196] Error preparing data for projected volume kube-api-access-zm9js for pod openshift-network-diagnostics/network-check-target-wrxv2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 12:02:29.570025 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:29.569910    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/090605d8-b2cc-4a2f-9f09-c22538254886-kube-api-access-zm9js podName:090605d8-b2cc-4a2f-9f09-c22538254886 nodeName:}" failed. No retries permitted until 2026-09-15 12:02:33.569890027 +0000 UTC m=+10.360812534 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-zm9js" (UniqueName: "kubernetes.io/projected/090605d8-b2cc-4a2f-9f09-c22538254886-kube-api-access-zm9js") pod "network-check-target-wrxv2" (UID: "090605d8-b2cc-4a2f-9f09-c22538254886") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 12:02:29.814193 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:29.813234    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdjwd"
Sep 15 12:02:29.814193 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:29.813400    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdjwd" podUID="f9c68d51-9107-4cd4-a57d-6f76c0762106"
Sep 15 12:02:29.814193 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:29.813820    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wrxv2"
Sep 15 12:02:29.814193 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:29.813923    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wrxv2" podUID="090605d8-b2cc-4a2f-9f09-c22538254886"
Sep 15 12:02:29.814193 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:29.813987    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g25rd"
Sep 15 12:02:29.814193 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:29.814068    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g25rd" podUID="07f9ddc4-14d7-4b04-9797-67507d4063bc"
Sep 15 12:02:31.811455 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:31.811415    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wrxv2"
Sep 15 12:02:31.811923 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:31.811547    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g25rd"
Sep 15 12:02:31.811923 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:31.811550    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wrxv2" podUID="090605d8-b2cc-4a2f-9f09-c22538254886"
Sep 15 12:02:31.811923 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:31.811621    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g25rd" podUID="07f9ddc4-14d7-4b04-9797-67507d4063bc"
Sep 15 12:02:31.811923 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:31.811651    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdjwd"
Sep 15 12:02:31.811923 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:31.811794    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdjwd" podUID="f9c68d51-9107-4cd4-a57d-6f76c0762106"
Sep 15 12:02:33.502143 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:33.502074    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/07f9ddc4-14d7-4b04-9797-67507d4063bc-original-pull-secret\") pod \"global-pull-secret-syncer-g25rd\" (UID: \"07f9ddc4-14d7-4b04-9797-67507d4063bc\") " pod="kube-system/global-pull-secret-syncer-g25rd"
Sep 15 12:02:33.502630 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:33.502211    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 12:02:33.502630 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:33.502303    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07f9ddc4-14d7-4b04-9797-67507d4063bc-original-pull-secret podName:07f9ddc4-14d7-4b04-9797-67507d4063bc nodeName:}" failed. No retries permitted until 2026-09-15 12:02:41.502282247 +0000 UTC m=+18.293204755 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/07f9ddc4-14d7-4b04-9797-67507d4063bc-original-pull-secret") pod "global-pull-secret-syncer-g25rd" (UID: "07f9ddc4-14d7-4b04-9797-67507d4063bc") : object "kube-system"/"original-pull-secret" not registered
Sep 15 12:02:33.502630 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:33.502336    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9c68d51-9107-4cd4-a57d-6f76c0762106-metrics-certs\") pod \"network-metrics-daemon-bdjwd\" (UID: \"f9c68d51-9107-4cd4-a57d-6f76c0762106\") " pod="openshift-multus/network-metrics-daemon-bdjwd"
Sep 15 12:02:33.502630 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:33.502420    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 12:02:33.502630 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:33.502461    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9c68d51-9107-4cd4-a57d-6f76c0762106-metrics-certs podName:f9c68d51-9107-4cd4-a57d-6f76c0762106 nodeName:}" failed. No retries permitted until 2026-09-15 12:02:41.502449282 +0000 UTC m=+18.293371785 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9c68d51-9107-4cd4-a57d-6f76c0762106-metrics-certs") pod "network-metrics-daemon-bdjwd" (UID: "f9c68d51-9107-4cd4-a57d-6f76c0762106") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 12:02:33.603507 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:33.603460    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm9js\" (UniqueName: \"kubernetes.io/projected/090605d8-b2cc-4a2f-9f09-c22538254886-kube-api-access-zm9js\") pod \"network-check-target-wrxv2\" (UID: \"090605d8-b2cc-4a2f-9f09-c22538254886\") " pod="openshift-network-diagnostics/network-check-target-wrxv2"
Sep 15 12:02:33.603698 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:33.603675    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 12:02:33.603748 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:33.603710    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 12:02:33.603748 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:33.603725    2555 projected.go:196] Error preparing data for projected volume kube-api-access-zm9js for pod openshift-network-diagnostics/network-check-target-wrxv2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 12:02:33.603856 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:33.603790    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/090605d8-b2cc-4a2f-9f09-c22538254886-kube-api-access-zm9js podName:090605d8-b2cc-4a2f-9f09-c22538254886 nodeName:}" failed. No retries permitted until 2026-09-15 12:02:41.603771617 +0000 UTC m=+18.394694109 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-zm9js" (UniqueName: "kubernetes.io/projected/090605d8-b2cc-4a2f-9f09-c22538254886-kube-api-access-zm9js") pod "network-check-target-wrxv2" (UID: "090605d8-b2cc-4a2f-9f09-c22538254886") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 12:02:33.811857 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:33.811782    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wrxv2"
Sep 15 12:02:33.812019 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:33.811872    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wrxv2" podUID="090605d8-b2cc-4a2f-9f09-c22538254886"
Sep 15 12:02:33.812019 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:33.811931    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g25rd"
Sep 15 12:02:33.812019 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:33.812013    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g25rd" podUID="07f9ddc4-14d7-4b04-9797-67507d4063bc"
Sep 15 12:02:33.812188 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:33.812065    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdjwd"
Sep 15 12:02:33.812245 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:33.812199    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdjwd" podUID="f9c68d51-9107-4cd4-a57d-6f76c0762106"
Sep 15 12:02:35.811280 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:35.810784    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g25rd"
Sep 15 12:02:35.811678 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:35.811101    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdjwd"
Sep 15 12:02:35.812171 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:35.812124    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g25rd" podUID="07f9ddc4-14d7-4b04-9797-67507d4063bc"
Sep 15 12:02:35.812307 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:35.812275    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdjwd" podUID="f9c68d51-9107-4cd4-a57d-6f76c0762106"
Sep 15 12:02:35.812365 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:35.811131    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wrxv2"
Sep 15 12:02:35.812634 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:35.812590    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wrxv2" podUID="090605d8-b2cc-4a2f-9f09-c22538254886"
Sep 15 12:02:37.811711 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:37.811671    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdjwd"
Sep 15 12:02:37.812125 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:37.811668    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g25rd"
Sep 15 12:02:37.812125 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:37.811789    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdjwd" podUID="f9c68d51-9107-4cd4-a57d-6f76c0762106"
Sep 15 12:02:37.812125 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:37.811888    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g25rd" podUID="07f9ddc4-14d7-4b04-9797-67507d4063bc"
Sep 15 12:02:37.812125 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:37.811668    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wrxv2"
Sep 15 12:02:37.812125 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:37.811994    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wrxv2" podUID="090605d8-b2cc-4a2f-9f09-c22538254886"
Sep 15 12:02:39.811407 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:39.811371    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wrxv2"
Sep 15 12:02:39.811872 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:39.811371    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g25rd"
Sep 15 12:02:39.811872 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:39.811507    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdjwd"
Sep 15 12:02:39.811872 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:39.811507    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wrxv2" podUID="090605d8-b2cc-4a2f-9f09-c22538254886"
Sep 15 12:02:39.811872 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:39.811597    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g25rd" podUID="07f9ddc4-14d7-4b04-9797-67507d4063bc"
Sep 15 12:02:39.811872 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:39.811677    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdjwd" podUID="f9c68d51-9107-4cd4-a57d-6f76c0762106"
Sep 15 12:02:41.564278 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:41.564238    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9c68d51-9107-4cd4-a57d-6f76c0762106-metrics-certs\") pod \"network-metrics-daemon-bdjwd\" (UID: \"f9c68d51-9107-4cd4-a57d-6f76c0762106\") " pod="openshift-multus/network-metrics-daemon-bdjwd"
Sep 15 12:02:41.564736 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:41.564304    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/07f9ddc4-14d7-4b04-9797-67507d4063bc-original-pull-secret\") pod \"global-pull-secret-syncer-g25rd\" (UID: \"07f9ddc4-14d7-4b04-9797-67507d4063bc\") " pod="kube-system/global-pull-secret-syncer-g25rd"
Sep 15 12:02:41.564736 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:41.564406    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 12:02:41.564736 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:41.564475    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9c68d51-9107-4cd4-a57d-6f76c0762106-metrics-certs podName:f9c68d51-9107-4cd4-a57d-6f76c0762106 nodeName:}" failed. No retries permitted until 2026-09-15 12:02:57.564459088 +0000 UTC m=+34.355381580 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9c68d51-9107-4cd4-a57d-6f76c0762106-metrics-certs") pod "network-metrics-daemon-bdjwd" (UID: "f9c68d51-9107-4cd4-a57d-6f76c0762106") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 12:02:41.564736 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:41.564413    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 12:02:41.564736 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:41.564546    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07f9ddc4-14d7-4b04-9797-67507d4063bc-original-pull-secret podName:07f9ddc4-14d7-4b04-9797-67507d4063bc nodeName:}" failed. No retries permitted until 2026-09-15 12:02:57.564532576 +0000 UTC m=+34.355455082 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/07f9ddc4-14d7-4b04-9797-67507d4063bc-original-pull-secret") pod "global-pull-secret-syncer-g25rd" (UID: "07f9ddc4-14d7-4b04-9797-67507d4063bc") : object "kube-system"/"original-pull-secret" not registered
Sep 15 12:02:41.665381 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:41.665348    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm9js\" (UniqueName: \"kubernetes.io/projected/090605d8-b2cc-4a2f-9f09-c22538254886-kube-api-access-zm9js\") pod \"network-check-target-wrxv2\" (UID: \"090605d8-b2cc-4a2f-9f09-c22538254886\") " pod="openshift-network-diagnostics/network-check-target-wrxv2"
Sep 15 12:02:41.665574 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:41.665552    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 12:02:41.665651 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:41.665585    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 12:02:41.665651 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:41.665599    2555 projected.go:196] Error preparing data for projected volume kube-api-access-zm9js for pod openshift-network-diagnostics/network-check-target-wrxv2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 12:02:41.665718 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:41.665662    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/090605d8-b2cc-4a2f-9f09-c22538254886-kube-api-access-zm9js podName:090605d8-b2cc-4a2f-9f09-c22538254886 nodeName:}" failed. No retries permitted until 2026-09-15 12:02:57.66564333 +0000 UTC m=+34.456565822 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-zm9js" (UniqueName: "kubernetes.io/projected/090605d8-b2cc-4a2f-9f09-c22538254886-kube-api-access-zm9js") pod "network-check-target-wrxv2" (UID: "090605d8-b2cc-4a2f-9f09-c22538254886") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 12:02:41.811011 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:41.810971    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g25rd"
Sep 15 12:02:41.811212 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:41.810971    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdjwd"
Sep 15 12:02:41.811212 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:41.811117    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g25rd" podUID="07f9ddc4-14d7-4b04-9797-67507d4063bc"
Sep 15 12:02:41.811212 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:41.810971    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wrxv2"
Sep 15 12:02:41.811395 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:41.811206    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdjwd" podUID="f9c68d51-9107-4cd4-a57d-6f76c0762106"
Sep 15 12:02:41.811395 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:41.811242    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wrxv2" podUID="090605d8-b2cc-4a2f-9f09-c22538254886"
Sep 15 12:02:43.811595 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:43.811555    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wrxv2"
Sep 15 12:02:43.812048 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:43.811673    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wrxv2" podUID="090605d8-b2cc-4a2f-9f09-c22538254886"
Sep 15 12:02:43.812048 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:43.811752    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g25rd"
Sep 15 12:02:43.812048 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:43.811853    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g25rd" podUID="07f9ddc4-14d7-4b04-9797-67507d4063bc"
Sep 15 12:02:43.812048 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:43.811899    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdjwd"
Sep 15 12:02:43.812485 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:43.812287    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdjwd" podUID="f9c68d51-9107-4cd4-a57d-6f76c0762106"
Sep 15 12:02:45.811144 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:45.810963    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g25rd"
Sep 15 12:02:45.811835 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:45.811009    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdjwd"
Sep 15 12:02:45.811835 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:45.811228    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g25rd" podUID="07f9ddc4-14d7-4b04-9797-67507d4063bc"
Sep 15 12:02:45.811835 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:45.811054    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wrxv2"
Sep 15 12:02:45.811835 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:45.811344    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdjwd" podUID="f9c68d51-9107-4cd4-a57d-6f76c0762106"
Sep 15 12:02:45.811835 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:45.811427    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wrxv2" podUID="090605d8-b2cc-4a2f-9f09-c22538254886"
Sep 15 12:02:46.039890 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:46.039858    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-c47md" event={"ID":"5be458c0-7237-4003-b4b9-1218ac635869","Type":"ContainerStarted","Data":"e70f9fa52207b430680ae7d6be1f6a31fa9456e938b68229020e3905a7c192ac"}
Sep 15 12:02:46.041336 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:46.041308    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-kgjkj" event={"ID":"365587ef-e477-48b4-aa6d-4e284b0b5951","Type":"ContainerStarted","Data":"f4c1baff1e9a5e80448f136619779d1cf956ad0776bea7c750771145d65b4adb"}
Sep 15 12:02:46.042821 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:46.042798    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qjwgl" event={"ID":"0fa9c45a-2974-45d2-82c2-23a63b9f0662","Type":"ContainerStarted","Data":"4e380f54614b81e8cae0c1a18e088179dcd0d2942b9c34670893b3d3bea3680a"}
Sep 15 12:02:46.044122 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:46.044095    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mkck2" event={"ID":"ee282972-ad6d-4a25-a3f5-218dfe607d6a","Type":"ContainerStarted","Data":"daff6ef584b2a7f75b4d8f7988bbf85cfdad3f83aa88435030ad26988d528922"}
Sep 15 12:02:46.046904 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:46.046884    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k62zz" event={"ID":"95528bd1-4100-43e9-bc0a-2e56670bc84f","Type":"ContainerStarted","Data":"65565b554fb57cdd9740320bdbfe613add3b6e7dd43c601689be1a876a627846"}
Sep 15 12:02:46.046993 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:46.046947    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k62zz" event={"ID":"95528bd1-4100-43e9-bc0a-2e56670bc84f","Type":"ContainerStarted","Data":"844f3c9833eda23b18b32257080d9f9c3c1482bd8987b287f9f8ec2330858078"}
Sep 15 12:02:46.046993 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:46.046979    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k62zz" event={"ID":"95528bd1-4100-43e9-bc0a-2e56670bc84f","Type":"ContainerStarted","Data":"326079e8a6a38c2f392ce05a801c84259a3c5a37e3c5fd1e552650bddc0ed06f"}
Sep 15 12:02:46.047066 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:46.047007    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k62zz" event={"ID":"95528bd1-4100-43e9-bc0a-2e56670bc84f","Type":"ContainerStarted","Data":"bc1e77ddedc3c0bc4da29d6b3c1954b7691ff10e7a8541c43806c37fad48421c"}
Sep 15 12:02:46.047066 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:46.047040    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k62zz" event={"ID":"95528bd1-4100-43e9-bc0a-2e56670bc84f","Type":"ContainerStarted","Data":"b93520b30866156ad3777f7e19b08fe34f3d9c6b196e307367661fc1b86b08de"}
Sep 15 12:02:46.047147 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:46.047067    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k62zz" event={"ID":"95528bd1-4100-43e9-bc0a-2e56670bc84f","Type":"ContainerStarted","Data":"26acf6dac96cb5b80984af313a4a09a220b5f3c7bcca1177b3ee9ecbe1308bfd"}
Sep 15 12:02:46.048384 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:46.048362    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-tnbfp" event={"ID":"01c26b58-ebc2-4b0f-96dd-14f8c373b0b5","Type":"ContainerStarted","Data":"62d20a8fbca4dfae5ad3ad019e1380d59145f317d5fac9bdf11cc5b38e3135fb"}
Sep 15 12:02:46.049892 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:46.049869    2555 generic.go:356] "Generic (PLEG): container finished" podID="b9b05b4f-57d5-454e-b739-2778ddcab5fb" containerID="85fb74df536e8cef1c34b009e38e7bd08f62744d343034d9f2e8a88cac464513" exitCode=0
Sep 15 12:02:46.049988 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:46.049934    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p4rvp" event={"ID":"b9b05b4f-57d5-454e-b739-2778ddcab5fb","Type":"ContainerDied","Data":"85fb74df536e8cef1c34b009e38e7bd08f62744d343034d9f2e8a88cac464513"}
Sep 15 12:02:46.051400 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:46.051362    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2wkff" event={"ID":"e246fd0a-f899-465f-aaeb-661d2a429d06","Type":"ContainerStarted","Data":"86f83f96112cb54cc3a6fe08ffb341b07fda48cd7c3b25fdd1b053f88c7cf09b"}
Sep 15 12:02:46.063917 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:46.063869    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-c47md" podStartSLOduration=7.261368966 podStartE2EDuration="22.063856091s" totalImagesPullingTime="14.802487125s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:02:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:02:46.061927535 +0000 UTC m=+22.852850048" watchObservedRunningTime="2026-09-15 12:02:46.063856091 +0000 UTC m=+22.854778601"
Sep 15 12:02:46.094296 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:46.094183    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-mkck2" podStartSLOduration=3.855708136 podStartE2EDuration="22.094165266s" totalImagesPullingTime="18.23845713s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 12:02:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:02:46.094117865 +0000 UTC m=+22.885040397" watchObservedRunningTime="2026-09-15 12:02:46.094165266 +0000 UTC m=+22.885087777"
Sep 15 12:02:46.123576 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:46.123524    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-tnbfp" podStartSLOduration=3.882313667 podStartE2EDuration="22.123508848s" totalImagesPullingTime="18.241195181s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 12:02:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:02:46.122644699 +0000 UTC m=+22.913567245" watchObservedRunningTime="2026-09-15 12:02:46.123508848 +0000 UTC m=+22.914431358"
Sep 15 12:02:46.166305 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:46.166233    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-2wkff" podStartSLOduration=3.870432836 podStartE2EDuration="22.166216069s" totalImagesPullingTime="18.295783233s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 12:02:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:02:46.165046267 +0000 UTC m=+22.955968779" watchObservedRunningTime="2026-09-15 12:02:46.166216069 +0000 UTC m=+22.957138580"
Sep 15 12:02:46.179528 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:46.179461    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-kgjkj" podStartSLOduration=3.94710399 podStartE2EDuration="22.179440547s" totalImagesPullingTime="18.232336557s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 12:02:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:02:46.179326308 +0000 UTC m=+22.970248843" watchObservedRunningTime="2026-09-15 12:02:46.179440547 +0000 UTC m=+22.970363057"
Sep 15 12:02:46.481850 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:46.481825    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-kgjkj_365587ef-e477-48b4-aa6d-4e284b0b5951/dns-node-resolver/0.log"
Sep 15 12:02:46.785916 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:46.785865    2555 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 15 12:02:46.796487 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:46.796412    2555 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-15T12:02:46.786098353Z","UUID":"db624c81-e185-42c1-aac1-1da58629e1b2","Handler":null,"Name":"","Endpoint":""}
Sep 15 12:02:46.798304 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:46.798285    2555 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 15 12:02:46.798409 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:46.798319    2555 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 15 12:02:47.055967 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:47.055884    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7fbc7" event={"ID":"25dcd72f-7a70-4cd8-ab3c-964f525bd756","Type":"ContainerStarted","Data":"1f8b37549c950f4403af0955a5944be7eb44eeed00629add266cc343e67c929d"}
Sep 15 12:02:47.057988 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:47.057954    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qjwgl" event={"ID":"0fa9c45a-2974-45d2-82c2-23a63b9f0662","Type":"ContainerStarted","Data":"c0b3e79d0af80255afda55395b2c953f3ffc85d4e1e80c24bcdb52b66f699b76"}
Sep 15 12:02:47.071800 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:47.071751    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-7fbc7" podStartSLOduration=4.829981979 podStartE2EDuration="23.07173616s" totalImagesPullingTime="18.241754181s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 12:02:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:02:47.071393584 +0000 UTC m=+23.862316136" watchObservedRunningTime="2026-09-15 12:02:47.07173616 +0000 UTC m=+23.862658671"
Sep 15 12:02:47.657999 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:47.657971    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mkck2_ee282972-ad6d-4a25-a3f5-218dfe607d6a/node-ca/0.log"
Sep 15 12:02:47.811833 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:47.811798    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g25rd"
Sep 15 12:02:47.811833 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:47.811822    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wrxv2"
Sep 15 12:02:47.812072 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:47.811888    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdjwd"
Sep 15 12:02:47.812072 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:47.811991    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g25rd" podUID="07f9ddc4-14d7-4b04-9797-67507d4063bc"
Sep 15 12:02:47.812178 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:47.812084    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdjwd" podUID="f9c68d51-9107-4cd4-a57d-6f76c0762106"
Sep 15 12:02:47.812178 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:47.812155    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wrxv2" podUID="090605d8-b2cc-4a2f-9f09-c22538254886"
Sep 15 12:02:48.065483 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:48.065440    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k62zz" event={"ID":"95528bd1-4100-43e9-bc0a-2e56670bc84f","Type":"ContainerStarted","Data":"4f54e69071cf83bff86fe215862da883abc447988745728a1b6809ba18247ef6"}
Sep 15 12:02:48.067875 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:48.067850    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qjwgl" event={"ID":"0fa9c45a-2974-45d2-82c2-23a63b9f0662","Type":"ContainerStarted","Data":"6f168791ba61b6bcc5723815085b6a61ae27c524ca41bd200c2a63c70dc3cbd8"}
Sep 15 12:02:48.086323 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:48.086250    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qjwgl" podStartSLOduration=3.715610593 podStartE2EDuration="24.086227462s" totalImagesPullingTime="20.370616869s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 12:02:24 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:02:48.086189932 +0000 UTC m=+24.877112465" watchObservedRunningTime="2026-09-15 12:02:48.086227462 +0000 UTC m=+24.877149974"
Sep 15 12:02:49.811122 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:49.811086    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g25rd"
Sep 15 12:02:49.811600 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:49.811087    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdjwd"
Sep 15 12:02:49.811600 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:49.811222    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g25rd" podUID="07f9ddc4-14d7-4b04-9797-67507d4063bc"
Sep 15 12:02:49.811600 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:49.811237    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wrxv2"
Sep 15 12:02:49.811600 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:49.811352    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdjwd" podUID="f9c68d51-9107-4cd4-a57d-6f76c0762106"
Sep 15 12:02:49.811600 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:49.811419    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wrxv2" podUID="090605d8-b2cc-4a2f-9f09-c22538254886"
Sep 15 12:02:50.897883 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:50.897842    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-c47md"
Sep 15 12:02:50.898533 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:50.898503    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-c47md"
Sep 15 12:02:51.074718 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:51.074683    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-c47md"
Sep 15 12:02:51.075381 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:51.075359    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-c47md"
Sep 15 12:02:51.810901 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:51.810814    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wrxv2"
Sep 15 12:02:51.811031 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:51.810814    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g25rd"
Sep 15 12:02:51.811031 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:51.810927    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wrxv2" podUID="090605d8-b2cc-4a2f-9f09-c22538254886"
Sep 15 12:02:51.811114 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:51.811031    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g25rd" podUID="07f9ddc4-14d7-4b04-9797-67507d4063bc"
Sep 15 12:02:51.811114 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:51.810831    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdjwd"
Sep 15 12:02:51.811177 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:51.811131    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdjwd" podUID="f9c68d51-9107-4cd4-a57d-6f76c0762106"
Sep 15 12:02:52.080656 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:52.080553    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k62zz" event={"ID":"95528bd1-4100-43e9-bc0a-2e56670bc84f","Type":"ContainerStarted","Data":"7a6049ea409a482f6a05093d8486aaa53f8bbc19e750f32b5275ab3c83732208"}
Sep 15 12:02:52.081214 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:52.080819    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-k62zz"
Sep 15 12:02:52.082457 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:52.082435    2555 generic.go:356] "Generic (PLEG): container finished" podID="b9b05b4f-57d5-454e-b739-2778ddcab5fb" containerID="4a997adbe750a8de6e8bd9d51e45982d87ac8be24832ee321a4321190bfaf463" exitCode=0
Sep 15 12:02:52.082562 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:52.082513    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p4rvp" event={"ID":"b9b05b4f-57d5-454e-b739-2778ddcab5fb","Type":"ContainerDied","Data":"4a997adbe750a8de6e8bd9d51e45982d87ac8be24832ee321a4321190bfaf463"}
Sep 15 12:02:52.096731 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:52.096707    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-k62zz"
Sep 15 12:02:52.129897 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:52.129847    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-k62zz" podStartSLOduration=9.835116437 podStartE2EDuration="28.129832618s" totalImagesPullingTime="18.294716181s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 12:02:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:02:52.10872728 +0000 UTC m=+28.899649812" watchObservedRunningTime="2026-09-15 12:02:52.129832618 +0000 UTC m=+28.920755129"
Sep 15 12:02:53.086910 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:53.086824    2555 generic.go:356] "Generic (PLEG): container finished" podID="b9b05b4f-57d5-454e-b739-2778ddcab5fb" containerID="26c23d71701f38fd3bc351db9c10669dc195d95e81fec6d9300e6082d3e47bf9" exitCode=0
Sep 15 12:02:53.087428 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:53.086909    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p4rvp" event={"ID":"b9b05b4f-57d5-454e-b739-2778ddcab5fb","Type":"ContainerDied","Data":"26c23d71701f38fd3bc351db9c10669dc195d95e81fec6d9300e6082d3e47bf9"}
Sep 15 12:02:53.087530 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:53.087503    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-k62zz"
Sep 15 12:02:53.087614 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:53.087562    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-k62zz"
Sep 15 12:02:53.102100 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:53.102073    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-k62zz"
Sep 15 12:02:53.820467 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:53.820441    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdjwd"
Sep 15 12:02:53.820467 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:53.820465    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wrxv2"
Sep 15 12:02:53.820646 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:53.820441    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g25rd"
Sep 15 12:02:53.820646 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:53.820547    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdjwd" podUID="f9c68d51-9107-4cd4-a57d-6f76c0762106"
Sep 15 12:02:53.820646 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:53.820628    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g25rd" podUID="07f9ddc4-14d7-4b04-9797-67507d4063bc"
Sep 15 12:02:53.820738 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:53.820696    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wrxv2" podUID="090605d8-b2cc-4a2f-9f09-c22538254886"
Sep 15 12:02:54.091331 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:54.091236    2555 generic.go:356] "Generic (PLEG): container finished" podID="b9b05b4f-57d5-454e-b739-2778ddcab5fb" containerID="a50d02232548c0a6092b0e28c871802ebd6d1b835bb2ced41ccd2e32fde005dc" exitCode=0
Sep 15 12:02:54.091331 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:54.091296    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p4rvp" event={"ID":"b9b05b4f-57d5-454e-b739-2778ddcab5fb","Type":"ContainerDied","Data":"a50d02232548c0a6092b0e28c871802ebd6d1b835bb2ced41ccd2e32fde005dc"}
Sep 15 12:02:55.811065 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:55.811030    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g25rd"
Sep 15 12:02:55.811550 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:55.811032    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wrxv2"
Sep 15 12:02:55.811550 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:55.811167    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g25rd" podUID="07f9ddc4-14d7-4b04-9797-67507d4063bc"
Sep 15 12:02:55.811550 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:55.811037    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdjwd"
Sep 15 12:02:55.811550 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:55.811255    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wrxv2" podUID="090605d8-b2cc-4a2f-9f09-c22538254886"
Sep 15 12:02:55.811550 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:55.811326    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdjwd" podUID="f9c68d51-9107-4cd4-a57d-6f76c0762106"
Sep 15 12:02:57.586878 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:57.586833    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9c68d51-9107-4cd4-a57d-6f76c0762106-metrics-certs\") pod \"network-metrics-daemon-bdjwd\" (UID: \"f9c68d51-9107-4cd4-a57d-6f76c0762106\") " pod="openshift-multus/network-metrics-daemon-bdjwd"
Sep 15 12:02:57.586878 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:57.586883    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/07f9ddc4-14d7-4b04-9797-67507d4063bc-original-pull-secret\") pod \"global-pull-secret-syncer-g25rd\" (UID: \"07f9ddc4-14d7-4b04-9797-67507d4063bc\") " pod="kube-system/global-pull-secret-syncer-g25rd"
Sep 15 12:02:57.587487 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:57.586986    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 12:02:57.587487 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:57.587002    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 12:02:57.587487 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:57.587054    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07f9ddc4-14d7-4b04-9797-67507d4063bc-original-pull-secret podName:07f9ddc4-14d7-4b04-9797-67507d4063bc nodeName:}" failed. No retries permitted until 2026-09-15 12:03:29.587035969 +0000 UTC m=+66.377958461 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/07f9ddc4-14d7-4b04-9797-67507d4063bc-original-pull-secret") pod "global-pull-secret-syncer-g25rd" (UID: "07f9ddc4-14d7-4b04-9797-67507d4063bc") : object "kube-system"/"original-pull-secret" not registered
Sep 15 12:02:57.587487 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:57.587070    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9c68d51-9107-4cd4-a57d-6f76c0762106-metrics-certs podName:f9c68d51-9107-4cd4-a57d-6f76c0762106 nodeName:}" failed. No retries permitted until 2026-09-15 12:03:29.587061732 +0000 UTC m=+66.377984221 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9c68d51-9107-4cd4-a57d-6f76c0762106-metrics-certs") pod "network-metrics-daemon-bdjwd" (UID: "f9c68d51-9107-4cd4-a57d-6f76c0762106") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 12:02:57.687485 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:57.687448    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm9js\" (UniqueName: \"kubernetes.io/projected/090605d8-b2cc-4a2f-9f09-c22538254886-kube-api-access-zm9js\") pod \"network-check-target-wrxv2\" (UID: \"090605d8-b2cc-4a2f-9f09-c22538254886\") " pod="openshift-network-diagnostics/network-check-target-wrxv2"
Sep 15 12:02:57.687696 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:57.687665    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 12:02:57.687696 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:57.687699    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 12:02:57.687863 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:57.687713    2555 projected.go:196] Error preparing data for projected volume kube-api-access-zm9js for pod openshift-network-diagnostics/network-check-target-wrxv2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 12:02:57.687863 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:57.687782    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/090605d8-b2cc-4a2f-9f09-c22538254886-kube-api-access-zm9js podName:090605d8-b2cc-4a2f-9f09-c22538254886 nodeName:}" failed. No retries permitted until 2026-09-15 12:03:29.687762267 +0000 UTC m=+66.478684759 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-zm9js" (UniqueName: "kubernetes.io/projected/090605d8-b2cc-4a2f-9f09-c22538254886-kube-api-access-zm9js") pod "network-check-target-wrxv2" (UID: "090605d8-b2cc-4a2f-9f09-c22538254886") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 12:02:57.811935 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:57.811427    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g25rd"
Sep 15 12:02:57.811935 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:57.811461    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdjwd"
Sep 15 12:02:57.811935 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:57.811427    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wrxv2"
Sep 15 12:02:57.811935 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:57.811559    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g25rd" podUID="07f9ddc4-14d7-4b04-9797-67507d4063bc"
Sep 15 12:02:57.811935 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:57.811667    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wrxv2" podUID="090605d8-b2cc-4a2f-9f09-c22538254886"
Sep 15 12:02:57.811935 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:57.811769    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdjwd" podUID="f9c68d51-9107-4cd4-a57d-6f76c0762106"
Sep 15 12:02:59.105965 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:59.105926    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p4rvp" event={"ID":"b9b05b4f-57d5-454e-b739-2778ddcab5fb","Type":"ContainerStarted","Data":"d6f9e8cecf509fdfc195c767f137c15e2a0e423f90a30e3c92e0244907b35883"}
Sep 15 12:02:59.811616 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:59.811578    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wrxv2"
Sep 15 12:02:59.811821 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:59.811578    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdjwd"
Sep 15 12:02:59.811821 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:59.811706    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wrxv2" podUID="090605d8-b2cc-4a2f-9f09-c22538254886"
Sep 15 12:02:59.811821 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:02:59.811714    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g25rd"
Sep 15 12:02:59.811983 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:59.811815    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdjwd" podUID="f9c68d51-9107-4cd4-a57d-6f76c0762106"
Sep 15 12:02:59.811983 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:02:59.811880    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g25rd" podUID="07f9ddc4-14d7-4b04-9797-67507d4063bc"
Sep 15 12:03:00.110775 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:00.110688    2555 generic.go:356] "Generic (PLEG): container finished" podID="b9b05b4f-57d5-454e-b739-2778ddcab5fb" containerID="d6f9e8cecf509fdfc195c767f137c15e2a0e423f90a30e3c92e0244907b35883" exitCode=0
Sep 15 12:03:00.110775 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:00.110728    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p4rvp" event={"ID":"b9b05b4f-57d5-454e-b739-2778ddcab5fb","Type":"ContainerDied","Data":"d6f9e8cecf509fdfc195c767f137c15e2a0e423f90a30e3c92e0244907b35883"}
Sep 15 12:03:01.119821 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:01.119779    2555 generic.go:356] "Generic (PLEG): container finished" podID="b9b05b4f-57d5-454e-b739-2778ddcab5fb" containerID="cb7401db684e36045c2ec9df6daa044fd5a937f89e45d7cc84ee588e4ab249b6" exitCode=0
Sep 15 12:03:01.120215 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:01.119838    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p4rvp" event={"ID":"b9b05b4f-57d5-454e-b739-2778ddcab5fb","Type":"ContainerDied","Data":"cb7401db684e36045c2ec9df6daa044fd5a937f89e45d7cc84ee588e4ab249b6"}
Sep 15 12:03:01.811014 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:01.810970    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wrxv2"
Sep 15 12:03:01.811014 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:01.810990    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdjwd"
Sep 15 12:03:01.811014 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:01.810990    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g25rd"
Sep 15 12:03:01.811334 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:03:01.811080    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wrxv2" podUID="090605d8-b2cc-4a2f-9f09-c22538254886"
Sep 15 12:03:01.811334 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:03:01.811176    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdjwd" podUID="f9c68d51-9107-4cd4-a57d-6f76c0762106"
Sep 15 12:03:01.811334 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:03:01.811237    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g25rd" podUID="07f9ddc4-14d7-4b04-9797-67507d4063bc"
Sep 15 12:03:02.125174 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:02.125087    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p4rvp" event={"ID":"b9b05b4f-57d5-454e-b739-2778ddcab5fb","Type":"ContainerStarted","Data":"4b8fb816609b86d1f9edf7a4de04a021cf85e6c1c0857ef2b377a01b5f94f132"}
Sep 15 12:03:02.146644 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:02.146597    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-p4rvp" podStartSLOduration=8.39392534 podStartE2EDuration="38.146582471s" totalImagesPullingTime="29.752657131s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 12:02:24 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:03:02.146350643 +0000 UTC m=+38.937273175" watchObservedRunningTime="2026-09-15 12:03:02.146582471 +0000 UTC m=+38.937504982"
Sep 15 12:03:03.812368 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:03.812147    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wrxv2"
Sep 15 12:03:03.812774 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:03.812233    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g25rd"
Sep 15 12:03:03.812774 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:03:03.812438    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wrxv2" podUID="090605d8-b2cc-4a2f-9f09-c22538254886"
Sep 15 12:03:03.812774 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:03:03.812507    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g25rd" podUID="07f9ddc4-14d7-4b04-9797-67507d4063bc"
Sep 15 12:03:03.812774 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:03.812258    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdjwd"
Sep 15 12:03:03.812774 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:03:03.812624    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdjwd" podUID="f9c68d51-9107-4cd4-a57d-6f76c0762106"
Sep 15 12:03:05.810797 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:05.810761    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g25rd"
Sep 15 12:03:05.810797 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:05.810791    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdjwd"
Sep 15 12:03:05.811315 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:05.810761    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wrxv2"
Sep 15 12:03:05.811315 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:03:05.810905    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g25rd" podUID="07f9ddc4-14d7-4b04-9797-67507d4063bc"
Sep 15 12:03:05.811315 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:03:05.810961    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdjwd" podUID="f9c68d51-9107-4cd4-a57d-6f76c0762106"
Sep 15 12:03:05.811315 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:03:05.811088    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wrxv2" podUID="090605d8-b2cc-4a2f-9f09-c22538254886"
Sep 15 12:03:07.810985 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:07.810949    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdjwd"
Sep 15 12:03:07.811444 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:07.810949    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wrxv2"
Sep 15 12:03:07.811444 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:07.810948    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g25rd"
Sep 15 12:03:07.811604 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:03:07.811571    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdjwd" podUID="f9c68d51-9107-4cd4-a57d-6f76c0762106"
Sep 15 12:03:07.811961 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:03:07.811923    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wrxv2" podUID="090605d8-b2cc-4a2f-9f09-c22538254886"
Sep 15 12:03:07.812649 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:03:07.812559    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g25rd" podUID="07f9ddc4-14d7-4b04-9797-67507d4063bc"
Sep 15 12:03:09.811409 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:09.811375    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g25rd"
Sep 15 12:03:09.811409 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:09.811412    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdjwd"
Sep 15 12:03:09.811826 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:09.811385    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wrxv2"
Sep 15 12:03:09.811826 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:03:09.811487    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g25rd" podUID="07f9ddc4-14d7-4b04-9797-67507d4063bc"
Sep 15 12:03:09.811826 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:03:09.811570    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wrxv2" podUID="090605d8-b2cc-4a2f-9f09-c22538254886"
Sep 15 12:03:09.811826 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:03:09.811621    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdjwd" podUID="f9c68d51-9107-4cd4-a57d-6f76c0762106"
Sep 15 12:03:10.375683 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:10.375647    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bdjwd"]
Sep 15 12:03:10.375842 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:10.375780    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdjwd"
Sep 15 12:03:10.375932 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:03:10.375908    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdjwd" podUID="f9c68d51-9107-4cd4-a57d-6f76c0762106"
Sep 15 12:03:10.378605 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:10.378572    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wrxv2"]
Sep 15 12:03:10.378759 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:10.378680    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wrxv2"
Sep 15 12:03:10.378834 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:03:10.378784    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wrxv2" podUID="090605d8-b2cc-4a2f-9f09-c22538254886"
Sep 15 12:03:10.379512 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:10.379454    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-g25rd"]
Sep 15 12:03:10.379626 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:10.379540    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g25rd"
Sep 15 12:03:10.379679 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:03:10.379634    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g25rd" podUID="07f9ddc4-14d7-4b04-9797-67507d4063bc"
Sep 15 12:03:11.811693 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:11.811603    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdjwd"
Sep 15 12:03:11.812117 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:03:11.811746    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdjwd" podUID="f9c68d51-9107-4cd4-a57d-6f76c0762106"
Sep 15 12:03:11.812117 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:11.811791    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wrxv2"
Sep 15 12:03:11.812117 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:03:11.811892    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wrxv2" podUID="090605d8-b2cc-4a2f-9f09-c22538254886"
Sep 15 12:03:12.811460 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:12.811423    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g25rd"
Sep 15 12:03:12.811634 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:03:12.811542    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g25rd" podUID="07f9ddc4-14d7-4b04-9797-67507d4063bc"
Sep 15 12:03:13.811490 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:13.811461    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdjwd"
Sep 15 12:03:13.811971 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:13.811536    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wrxv2"
Sep 15 12:03:13.811971 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:03:13.811560    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdjwd" podUID="f9c68d51-9107-4cd4-a57d-6f76c0762106"
Sep 15 12:03:13.811971 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:03:13.811637    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wrxv2" podUID="090605d8-b2cc-4a2f-9f09-c22538254886"
Sep 15 12:03:14.811779 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:14.811592    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g25rd"
Sep 15 12:03:14.812154 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:03:14.811853    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g25rd" podUID="07f9ddc4-14d7-4b04-9797-67507d4063bc"
Sep 15 12:03:14.987934 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:14.987907    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-17.ec2.internal" event="NodeReady"
Sep 15 12:03:14.988070 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:14.988021    2555 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 15 12:03:15.026649 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:15.026613    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-8678457cc7-t287r"]
Sep 15 12:03:15.036013 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:15.035989    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8678457cc7-t287r"
Sep 15 12:03:15.041081 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:15.040342    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 15 12:03:15.041081 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:15.040577    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 15 12:03:15.041081 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:15.040638    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 15 12:03:15.041081 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:15.040880    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-s8xfw\""
Sep 15 12:03:15.044320 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:15.044299    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-tx76t"]
Sep 15 12:03:15.049490 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:15.049463    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 15 12:03:15.052727 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:15.052708    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-8678457cc7-t287r"]
Sep 15 12:03:15.052826 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:15.052786    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-tx76t"
Sep 15 12:03:15.054121 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:15.054102    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-298j7"]
Sep 15 12:03:15.055568 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:15.055551    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-swgf8\""
Sep 15 12:03:15.056106 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:15.056091    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 15 12:03:15.056189 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:15.056106    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 15 12:03:15.056308 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:15.056293    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 15 12:03:15.063320 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:15.063242    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-tx76t"]
Sep 15 12:03:15.063395 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:15.063370    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-298j7"
Sep 15 12:03:15.066798 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:15.066753    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 15 12:03:15.066907 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:15.066779    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 15 12:03:15.066968 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:15.066914    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 15 12:03:15.066968 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:15.066936    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 15 12:03:15.066968 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:15.066850    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-wlcl7\""
Sep 15 12:03:15.069330 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:15.069307    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-298j7"]
Sep 15 12:03:15.143094 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:15.143056    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-2vzvd"]
Sep 15 12:03:15.155299 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:15.155255    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2vzvd"
Sep 15 12:03:15.155826 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:15.155801    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2vzvd"]
Sep 15 12:03:15.158190 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:15.158159    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 15 12:03:15.158474 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:15.158457    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2gwv7\""
Sep 15 12:03:15.158587 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:15.158571    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 15 12:03:15.209601 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:15.209565    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzwd4\" (UniqueName: \"kubernetes.io/projected/e21c5438-c5ca-4375-81e8-db67f537528e-kube-api-access-kzwd4\") pod \"ingress-canary-tx76t\" (UID: \"e21c5438-c5ca-4375-81e8-db67f537528e\") " pod="openshift-ingress-canary/ingress-canary-tx76t"
Sep 15 12:03:15.209601 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:15.209603    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b6d14a02-3f21-44df-9db8-0f057f5eb567-crio-socket\") pod \"insights-runtime-extractor-298j7\" (UID: \"b6d14a02-3f21-44df-9db8-0f057f5eb567\") " pod="openshift-insights/insights-runtime-extractor-298j7"
Sep 15 12:03:15.209837 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:15.209622    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/62c63012-fb07-478e-b95e-01a2d6cbd475-bound-sa-token\") pod \"image-registry-8678457cc7-t287r\" (UID: \"62c63012-fb07-478e-b95e-01a2d6cbd475\") " pod="openshift-image-registry/image-registry-8678457cc7-t287r"
Sep 15 12:03:15.209837 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:15.209650    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/62c63012-fb07-478e-b95e-01a2d6cbd475-registry-tls\") pod \"image-registry-8678457cc7-t287r\" (UID: \"62c63012-fb07-478e-b95e-01a2d6cbd475\") " pod="openshift-image-registry/image-registry-8678457cc7-t287r"
Sep 15 12:03:15.209837 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:15.209669    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/62c63012-fb07-478e-b95e-01a2d6cbd475-registry-certificates\") pod \"image-registry-8678457cc7-t287r\" (UID: \"62c63012-fb07-478e-b95e-01a2d6cbd475\") " pod="openshift-image-registry/image-registry-8678457cc7-t287r"
Sep 15 12:03:15.209837 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:15.209692    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/62c63012-fb07-478e-b95e-01a2d6cbd475-installation-pull-secrets\") pod \"image-registry-8678457cc7-t287r\" (UID: \"62c63012-fb07-478e-b95e-01a2d6cbd475\") " pod="openshift-image-registry/image-registry-8678457cc7-t287r"
Sep 15 12:03:15.209837 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:15.209731    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e21c5438-c5ca-4375-81e8-db67f537528e-cert\") pod \"ingress-canary-tx76t\" (UID: \"e21c5438-c5ca-4375-81e8-db67f537528e\") " pod="openshift-ingress-canary/ingress-canary-tx76t"
Sep 15 12:03:15.209837 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:15.209751    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87q9v\" (UniqueName: \"kubernetes.io/projected/62c63012-fb07-478e-b95e-01a2d6cbd475-kube-api-access-87q9v\") pod \"image-registry-8678457cc7-t287r\" (UID: \"62c63012-fb07-478e-b95e-01a2d6cbd475\") " pod="openshift-image-registry/image-registry-8678457cc7-t287r"
Sep 15 12:03:15.209837 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:15.209809    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b6d14a02-3f21-44df-9db8-0f057f5eb567-data-volume\") pod \"insights-runtime-extractor-298j7\" (UID: \"b6d14a02-3f21-44df-9db8-0f057f5eb567\") " pod="openshift-insights/insights-runtime-extractor-298j7"
Sep 15 12:03:15.210132 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:15.209853    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b6d14a02-3f21-44df-9db8-0f057f5eb567-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-298j7\" (UID: \"b6d14a02-3f21-44df-9db8-0f057f5eb567\") " pod="openshift-insights/insights-runtime-extractor-298j7"
Sep 15 12:03:15.210132 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:15.209881    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b6d14a02-3f21-44df-9db8-0f057f5eb567-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-298j7\" (UID: \"b6d14a02-3f21-44df-9db8-0f057f5eb567\") " pod="openshift-insights/insights-runtime-extractor-298j7"
Sep 15 12:03:15.210132 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:15.209925    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/62c63012-fb07-478e-b95e-01a2d6cbd475-ca-trust-extracted\") pod \"image-registry-8678457cc7-t287r\" (UID: \"62c63012-fb07-478e-b95e-01a2d6cbd475\") " pod="openshift-image-registry/image-registry-8678457cc7-t287r"
Sep 15 12:03:15.210132 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:15.209995    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9tvx\" (UniqueName: \"kubernetes.io/projected/b6d14a02-3f21-44df-9db8-0f057f5eb567-kube-api-access-l9tvx\") pod \"insights-runtime-extractor-298j7\" (UID: \"b6d14a02-3f21-44df-9db8-0f057f5eb567\") " pod="openshift-insights/insights-runtime-extractor-298j7"
Sep 15 12:03:15.210132 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:15.210036    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/62c63012-fb07-478e-b95e-01a2d6cbd475-image-registry-private-configuration\") pod \"image-registry-8678457cc7-t287r\" (UID: \"62c63012-fb07-478e-b95e-01a2d6cbd475\") " pod="openshift-image-registry/image-registry-8678457cc7-t287r"
Sep 15 12:03:15.210132 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:15.210099    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/62c63012-fb07-478e-b95e-01a2d6cbd475-trusted-ca\") pod \"image-registry-8678457cc7-t287r\" (UID: \"62c63012-fb07-478e-b95e-01a2d6cbd475\") " pod="openshift-image-registry/image-registry-8678457cc7-t287r"
Sep 15 12:03:15.310639 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:15.310605    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e21c5438-c5ca-4375-81e8-db67f537528e-cert\") pod \"ingress-canary-tx76t\" (UID: \"e21c5438-c5ca-4375-81e8-db67f537528e\") " pod="openshift-ingress-canary/ingress-canary-tx76t"
Sep 15 12:03:15.310801 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:15.310644    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87q9v\" (UniqueName: \"kubernetes.io/projected/62c63012-fb07-478e-b95e-01a2d6cbd475-kube-api-access-87q9v\") pod \"image-registry-8678457cc7-t287r\" (UID: \"62c63012-fb07-478e-b95e-01a2d6cbd475\") " pod="openshift-image-registry/image-registry-8678457cc7-t287r"
Sep 15 12:03:15.310801 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:15.310669    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6qdn\" (UniqueName: \"kubernetes.io/projected/bc3c99ae-7814-40fe-8d0f-d31159e7ac76-kube-api-access-j6qdn\") pod \"dns-default-2vzvd\" (UID: \"bc3c99ae-7814-40fe-8d0f-d31159e7ac76\") " pod="openshift-dns/dns-default-2vzvd"
Sep 15 12:03:15.310801 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:15.310686    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b6d14a02-3f21-44df-9db8-0f057f5eb567-data-volume\") pod \"insights-runtime-extractor-298j7\" (UID: \"b6d14a02-3f21-44df-9db8-0f057f5eb567\") " pod="openshift-insights/insights-runtime-extractor-298j7"
Sep 15 12:03:15.310801 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:15.310704    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b6d14a02-3f21-44df-9db8-0f057f5eb567-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-298j7\" (UID: \"b6d14a02-3f21-44df-9db8-0f057f5eb567\") " pod="openshift-insights/insights-runtime-extractor-298j7"
Sep 15 12:03:15.310801 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:15.310735    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b6d14a02-3f21-44df-9db8-0f057f5eb567-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-298j7\" (UID: \"b6d14a02-3f21-44df-9db8-0f057f5eb567\") " pod="openshift-insights/insights-runtime-extractor-298j7"
Sep 15 12:03:15.310801 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:15.310763    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/62c63012-fb07-478e-b95e-01a2d6cbd475-ca-trust-extracted\") pod \"image-registry-8678457cc7-t287r\" (UID: \"62c63012-fb07-478e-b95e-01a2d6cbd475\") " pod="openshift-image-registry/image-registry-8678457cc7-t287r"
Sep 15 12:03:15.310801 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:15.310797    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc3c99ae-7814-40fe-8d0f-d31159e7ac76-tmp-dir\") pod \"dns-default-2vzvd\" (UID: \"bc3c99ae-7814-40fe-8d0f-d31159e7ac76\") " pod="openshift-dns/dns-default-2vzvd"
Sep 15 12:03:15.311117 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:15.310826    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bc3c99ae-7814-40fe-8d0f-d31159e7ac76-config-volume\") pod \"dns-default-2vzvd\" (UID: \"bc3c99ae-7814-40fe-8d0f-d31159e7ac76\") " pod="openshift-dns/dns-default-2vzvd"
Sep 15 12:03:15.311117 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:15.310852    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc3c99ae-7814-40fe-8d0f-d31159e7ac76-metrics-tls\") pod \"dns-default-2vzvd\" (UID: \"bc3c99ae-7814-40fe-8d0f-d31159e7ac76\") " pod="openshift-dns/dns-default-2vzvd"
Sep 15 12:03:15.311117 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:15.310878    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9tvx\" (UniqueName: \"kubernetes.io/projected/b6d14a02-3f21-44df-9db8-0f057f5eb567-kube-api-access-l9tvx\") pod \"insights-runtime-extractor-298j7\" (UID: \"b6d14a02-3f21-44df-9db8-0f057f5eb567\") " pod="openshift-insights/insights-runtime-extractor-298j7"
Sep 15 12:03:15.311117 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:15.310915    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/62c63012-fb07-478e-b95e-01a2d6cbd475-image-registry-private-configuration\") pod \"image-registry-8678457cc7-t287r\" (UID: \"62c63012-fb07-478e-b95e-01a2d6cbd475\") " pod="openshift-image-registry/image-registry-8678457cc7-t287r"
Sep 15 12:03:15.311117 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:15.310982    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/62c63012-fb07-478e-b95e-01a2d6cbd475-trusted-ca\") pod \"image-registry-8678457cc7-t287r\" (UID: \"62c63012-fb07-478e-b95e-01a2d6cbd475\") " pod="openshift-image-registry/image-registry-8678457cc7-t287r"
Sep 15 12:03:15.311117 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:15.311037    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzwd4\" (UniqueName: \"kubernetes.io/projected/e21c5438-c5ca-4375-81e8-db67f537528e-kube-api-access-kzwd4\") pod \"ingress-canary-tx76t\" (UID: \"e21c5438-c5ca-4375-81e8-db67f537528e\") " pod="openshift-ingress-canary/ingress-canary-tx76t"
Sep 15 12:03:15.311117 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:15.311061    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b6d14a02-3f21-44df-9db8-0f057f5eb567-crio-socket\") pod \"insights-runtime-extractor-298j7\" (UID: \"b6d14a02-3f21-44df-9db8-0f057f5eb567\") " pod="openshift-insights/insights-runtime-extractor-298j7"
Sep 15 12:03:15.311117 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:15.311083    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/62c63012-fb07-478e-b95e-01a2d6cbd475-bound-sa-token\") pod \"image-registry-8678457cc7-t287r\" (UID: \"62c63012-fb07-478e-b95e-01a2d6cbd475\") " pod="openshift-image-registry/image-registry-8678457cc7-t287r"
Sep 15 12:03:15.311575 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:15.311356    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/62c63012-fb07-478e-b95e-01a2d6cbd475-registry-tls\") pod \"image-registry-8678457cc7-t287r\" (UID: \"62c63012-fb07-478e-b95e-01a2d6cbd475\") " pod="openshift-image-registry/image-registry-8678457cc7-t287r"
Sep 15 12:03:15.311575 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:15.311390    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/62c63012-fb07-478e-b95e-01a2d6cbd475-registry-certificates\") pod \"image-registry-8678457cc7-t287r\" (UID: \"62c63012-fb07-478e-b95e-01a2d6cbd475\") " pod="openshift-image-registry/image-registry-8678457cc7-t287r"
Sep 15 12:03:15.311575 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:15.311417    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/62c63012-fb07-478e-b95e-01a2d6cbd475-installation-pull-secrets\") pod \"image-registry-8678457cc7-t287r\" (UID: \"62c63012-fb07-478e-b95e-01a2d6cbd475\") " pod="openshift-image-registry/image-registry-8678457cc7-t287r"
Sep 15 12:03:15.311575 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:15.311485    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b6d14a02-3f21-44df-9db8-0f057f5eb567-data-volume\") pod \"insights-runtime-extractor-298j7\" (UID: \"b6d14a02-3f21-44df-9db8-0f057f5eb567\") " pod="openshift-insights/insights-runtime-extractor-298j7"
Sep 15 12:03:15.311575 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:15.311489    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/62c63012-fb07-478e-b95e-01a2d6cbd475-ca-trust-extracted\") pod \"image-registry-8678457cc7-t287r\" (UID: \"62c63012-fb07-478e-b95e-01a2d6cbd475\") " pod="openshift-image-registry/image-registry-8678457cc7-t287r"
Sep 15 12:03:15.311884 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:15.311616    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b6d14a02-3f21-44df-9db8-0f057f5eb567-crio-socket\") pod \"insights-runtime-extractor-298j7\" (UID: \"b6d14a02-3f21-44df-9db8-0f057f5eb567\") " pod="openshift-insights/insights-runtime-extractor-298j7"
Sep 15 12:03:15.312533 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:15.312228    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/62c63012-fb07-478e-b95e-01a2d6cbd475-trusted-ca\") pod \"image-registry-8678457cc7-t287r\" (UID: \"62c63012-fb07-478e-b95e-01a2d6cbd475\") " pod="openshift-image-registry/image-registry-8678457cc7-t287r"
Sep 15 12:03:15.312533 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:15.312280    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/62c63012-fb07-478e-b95e-01a2d6cbd475-registry-certificates\") pod \"image-registry-8678457cc7-t287r\" (UID: \"62c63012-fb07-478e-b95e-01a2d6cbd475\") " pod="openshift-image-registry/image-registry-8678457cc7-t287r"
Sep 15 12:03:15.312705 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:15.312655    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b6d14a02-3f21-44df-9db8-0f057f5eb567-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-298j7\" (UID: \"b6d14a02-3f21-44df-9db8-0f057f5eb567\") " pod="openshift-insights/insights-runtime-extractor-298j7"
Sep 15 12:03:15.315277 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:15.315201    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/62c63012-fb07-478e-b95e-01a2d6cbd475-installation-pull-secrets\") pod \"image-registry-8678457cc7-t287r\" (UID: \"62c63012-fb07-478e-b95e-01a2d6cbd475\") " pod="openshift-image-registry/image-registry-8678457cc7-t287r"
Sep 15 12:03:15.315358 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:15.315307    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b6d14a02-3f21-44df-9db8-0f057f5eb567-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-298j7\" (UID: \"b6d14a02-3f21-44df-9db8-0f057f5eb567\") " pod="openshift-insights/insights-runtime-extractor-298j7"
Sep 15 12:03:15.315393 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:15.315362    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/62c63012-fb07-478e-b95e-01a2d6cbd475-registry-tls\") pod \"image-registry-8678457cc7-t287r\" (UID: \"62c63012-fb07-478e-b95e-01a2d6cbd475\") " pod="openshift-image-registry/image-registry-8678457cc7-t287r"
Sep 15 12:03:15.315427 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:15.315405    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e21c5438-c5ca-4375-81e8-db67f537528e-cert\") pod \"ingress-canary-tx76t\" (UID: \"e21c5438-c5ca-4375-81e8-db67f537528e\") " pod="openshift-ingress-canary/ingress-canary-tx76t"
Sep 15 12:03:15.315463 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:15.315412    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/62c63012-fb07-478e-b95e-01a2d6cbd475-image-registry-private-configuration\") pod \"image-registry-8678457cc7-t287r\" (UID: \"62c63012-fb07-478e-b95e-01a2d6cbd475\") " pod="openshift-image-registry/image-registry-8678457cc7-t287r"
Sep 15 12:03:15.320189 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:15.320158    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87q9v\" (UniqueName: \"kubernetes.io/projected/62c63012-fb07-478e-b95e-01a2d6cbd475-kube-api-access-87q9v\") pod \"image-registry-8678457cc7-t287r\" (UID: \"62c63012-fb07-478e-b95e-01a2d6cbd475\") " pod="openshift-image-registry/image-registry-8678457cc7-t287r"
Sep 15 12:03:15.320449 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:15.320428    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9tvx\" (UniqueName: \"kubernetes.io/projected/b6d14a02-3f21-44df-9db8-0f057f5eb567-kube-api-access-l9tvx\") pod \"insights-runtime-extractor-298j7\" (UID: \"b6d14a02-3f21-44df-9db8-0f057f5eb567\") " pod="openshift-insights/insights-runtime-extractor-298j7"
Sep 15 12:03:15.320514 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:15.320451    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzwd4\" (UniqueName: \"kubernetes.io/projected/e21c5438-c5ca-4375-81e8-db67f537528e-kube-api-access-kzwd4\") pod \"ingress-canary-tx76t\" (UID: \"e21c5438-c5ca-4375-81e8-db67f537528e\") " pod="openshift-ingress-canary/ingress-canary-tx76t"
Sep 15 12:03:15.320514 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:15.320459    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/62c63012-fb07-478e-b95e-01a2d6cbd475-bound-sa-token\") pod \"image-registry-8678457cc7-t287r\" (UID: \"62c63012-fb07-478e-b95e-01a2d6cbd475\") " pod="openshift-image-registry/image-registry-8678457cc7-t287r"
Sep 15 12:03:15.349616 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:15.349584    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8678457cc7-t287r"
Sep 15 12:03:15.360716 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:15.360689    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-tx76t"
Sep 15 12:03:15.372887 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:15.372858    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-298j7"
Sep 15 12:03:15.412437 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:15.412402    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bc3c99ae-7814-40fe-8d0f-d31159e7ac76-config-volume\") pod \"dns-default-2vzvd\" (UID: \"bc3c99ae-7814-40fe-8d0f-d31159e7ac76\") " pod="openshift-dns/dns-default-2vzvd"
Sep 15 12:03:15.412437 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:15.412438    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc3c99ae-7814-40fe-8d0f-d31159e7ac76-metrics-tls\") pod \"dns-default-2vzvd\" (UID: \"bc3c99ae-7814-40fe-8d0f-d31159e7ac76\") " pod="openshift-dns/dns-default-2vzvd"
Sep 15 12:03:15.412690 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:15.412664    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6qdn\" (UniqueName: \"kubernetes.io/projected/bc3c99ae-7814-40fe-8d0f-d31159e7ac76-kube-api-access-j6qdn\") pod \"dns-default-2vzvd\" (UID: \"bc3c99ae-7814-40fe-8d0f-d31159e7ac76\") " pod="openshift-dns/dns-default-2vzvd"
Sep 15 12:03:15.414654 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:15.412719    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc3c99ae-7814-40fe-8d0f-d31159e7ac76-tmp-dir\") pod \"dns-default-2vzvd\" (UID: \"bc3c99ae-7814-40fe-8d0f-d31159e7ac76\") " pod="openshift-dns/dns-default-2vzvd"
Sep 15 12:03:15.414654 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:15.413341    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bc3c99ae-7814-40fe-8d0f-d31159e7ac76-config-volume\") pod \"dns-default-2vzvd\" (UID: \"bc3c99ae-7814-40fe-8d0f-d31159e7ac76\") " pod="openshift-dns/dns-default-2vzvd"
Sep 15 12:03:15.414654 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:15.413354    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc3c99ae-7814-40fe-8d0f-d31159e7ac76-tmp-dir\") pod \"dns-default-2vzvd\" (UID: \"bc3c99ae-7814-40fe-8d0f-d31159e7ac76\") " pod="openshift-dns/dns-default-2vzvd"
Sep 15 12:03:15.417098 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:15.415724    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc3c99ae-7814-40fe-8d0f-d31159e7ac76-metrics-tls\") pod \"dns-default-2vzvd\" (UID: \"bc3c99ae-7814-40fe-8d0f-d31159e7ac76\") " pod="openshift-dns/dns-default-2vzvd"
Sep 15 12:03:15.422439 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:15.422412    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6qdn\" (UniqueName: \"kubernetes.io/projected/bc3c99ae-7814-40fe-8d0f-d31159e7ac76-kube-api-access-j6qdn\") pod \"dns-default-2vzvd\" (UID: \"bc3c99ae-7814-40fe-8d0f-d31159e7ac76\") " pod="openshift-dns/dns-default-2vzvd"
Sep 15 12:03:15.465022 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:15.464799    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2vzvd"
Sep 15 12:03:15.681764 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:15.681737    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-tx76t"]
Sep 15 12:03:15.684583 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:15.684558    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-8678457cc7-t287r"]
Sep 15 12:03:15.685324 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:03:15.685229    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode21c5438_c5ca_4375_81e8_db67f537528e.slice/crio-0aa7f2477dd5cae90a7a1b5a83266ea7a4e8283330f56de97dbfe87cb52d4690 WatchSource:0}: Error finding container 0aa7f2477dd5cae90a7a1b5a83266ea7a4e8283330f56de97dbfe87cb52d4690: Status 404 returned error can't find the container with id 0aa7f2477dd5cae90a7a1b5a83266ea7a4e8283330f56de97dbfe87cb52d4690
Sep 15 12:03:15.687381 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:03:15.687356    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62c63012_fb07_478e_b95e_01a2d6cbd475.slice/crio-903ad11adecefc7183f0b8ce97c0c0ef7427a224b4579649965be13707a31971 WatchSource:0}: Error finding container 903ad11adecefc7183f0b8ce97c0c0ef7427a224b4579649965be13707a31971: Status 404 returned error can't find the container with id 903ad11adecefc7183f0b8ce97c0c0ef7427a224b4579649965be13707a31971
Sep 15 12:03:15.703690 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:15.703668    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-298j7"]
Sep 15 12:03:15.705740 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:03:15.705716    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6d14a02_3f21_44df_9db8_0f057f5eb567.slice/crio-01cb9254f3206d730171315196f69f5fba0ed7659ebb84367194a4ce1f3fa9a5 WatchSource:0}: Error finding container 01cb9254f3206d730171315196f69f5fba0ed7659ebb84367194a4ce1f3fa9a5: Status 404 returned error can't find the container with id 01cb9254f3206d730171315196f69f5fba0ed7659ebb84367194a4ce1f3fa9a5
Sep 15 12:03:15.757319 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:15.757157    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2vzvd"]
Sep 15 12:03:15.765319 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:03:15.765287    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc3c99ae_7814_40fe_8d0f_d31159e7ac76.slice/crio-05b082ef187fae4d5d8558918bcd1173400c3a87f0fd8153777862231d512a68 WatchSource:0}: Error finding container 05b082ef187fae4d5d8558918bcd1173400c3a87f0fd8153777862231d512a68: Status 404 returned error can't find the container with id 05b082ef187fae4d5d8558918bcd1173400c3a87f0fd8153777862231d512a68
Sep 15 12:03:15.811456 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:15.811429    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wrxv2"
Sep 15 12:03:15.811675 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:15.811635    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdjwd"
Sep 15 12:03:15.814530 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:15.814499    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-k48ps\""
Sep 15 12:03:15.814530 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:15.814518    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 15 12:03:15.815357 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:15.814607    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 15 12:03:15.815357 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:15.814660    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 15 12:03:15.815357 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:15.815138    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9l82m\""
Sep 15 12:03:16.159686 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:16.159654    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2vzvd" event={"ID":"bc3c99ae-7814-40fe-8d0f-d31159e7ac76","Type":"ContainerStarted","Data":"05b082ef187fae4d5d8558918bcd1173400c3a87f0fd8153777862231d512a68"}
Sep 15 12:03:16.161171 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:16.161145    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8678457cc7-t287r" event={"ID":"62c63012-fb07-478e-b95e-01a2d6cbd475","Type":"ContainerStarted","Data":"2bb6be8b7a6ec5e6de6e10e920efddf87850b41011b0f87f61847ff669c90131"}
Sep 15 12:03:16.161331 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:16.161183    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8678457cc7-t287r" event={"ID":"62c63012-fb07-478e-b95e-01a2d6cbd475","Type":"ContainerStarted","Data":"903ad11adecefc7183f0b8ce97c0c0ef7427a224b4579649965be13707a31971"}
Sep 15 12:03:16.161331 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:16.161236    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-8678457cc7-t287r"
Sep 15 12:03:16.162697 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:16.162679    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-298j7" event={"ID":"b6d14a02-3f21-44df-9db8-0f057f5eb567","Type":"ContainerStarted","Data":"272eb8a23224ff63378cae1fcfa39523f5c3862e46e3dcfcf7744828446cf385"}
Sep 15 12:03:16.162772 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:16.162718    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-298j7" event={"ID":"b6d14a02-3f21-44df-9db8-0f057f5eb567","Type":"ContainerStarted","Data":"01cb9254f3206d730171315196f69f5fba0ed7659ebb84367194a4ce1f3fa9a5"}
Sep 15 12:03:16.163865 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:16.163847    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-tx76t" event={"ID":"e21c5438-c5ca-4375-81e8-db67f537528e","Type":"ContainerStarted","Data":"0aa7f2477dd5cae90a7a1b5a83266ea7a4e8283330f56de97dbfe87cb52d4690"}
Sep 15 12:03:16.187213 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:16.187142    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-8678457cc7-t287r" podStartSLOduration=7.187114317 podStartE2EDuration="7.187114317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:03:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:03:16.186102107 +0000 UTC m=+52.977024618" watchObservedRunningTime="2026-09-15 12:03:16.187114317 +0000 UTC m=+52.978036832"
Sep 15 12:03:16.811573 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:16.811537    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g25rd"
Sep 15 12:03:16.814553 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:16.814524    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 15 12:03:19.175847 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:19.175603    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-298j7" event={"ID":"b6d14a02-3f21-44df-9db8-0f057f5eb567","Type":"ContainerStarted","Data":"ce88ca12c310c01e044abe00744d3099e4ed83058591076b964af4fa5accad88"}
Sep 15 12:03:19.177443 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:19.177410    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-tx76t" event={"ID":"e21c5438-c5ca-4375-81e8-db67f537528e","Type":"ContainerStarted","Data":"fc1cc9b5cceb06fc818f0478b9ad19aa2685cd3d16436c67ccfaf40b70330630"}
Sep 15 12:03:19.179350 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:19.179326    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2vzvd" event={"ID":"bc3c99ae-7814-40fe-8d0f-d31159e7ac76","Type":"ContainerStarted","Data":"2ac4345fb0480368cffad62cca4f4cee11d908f3bb4392d6bb225bb2ad883228"}
Sep 15 12:03:19.179468 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:19.179373    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2vzvd" event={"ID":"bc3c99ae-7814-40fe-8d0f-d31159e7ac76","Type":"ContainerStarted","Data":"c7caea9d32d0c8377ba9bf3a86d32ce8f5bd326140bb393909c689104bd8de4d"}
Sep 15 12:03:19.179468 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:19.179404    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-2vzvd"
Sep 15 12:03:19.194347 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:19.194289    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-tx76t" podStartSLOduration=1.663327287 podStartE2EDuration="4.194247156s" totalImagesPullingTime="2.530919869s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:03:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:03:19.192985034 +0000 UTC m=+55.983907566" watchObservedRunningTime="2026-09-15 12:03:19.194247156 +0000 UTC m=+55.985169663"
Sep 15 12:03:19.209725 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:19.209665    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-2vzvd" podStartSLOduration=1.764618836 podStartE2EDuration="4.209642997s" totalImagesPullingTime="2.445024161s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:03:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:03:19.208980835 +0000 UTC m=+55.999903347" watchObservedRunningTime="2026-09-15 12:03:19.209642997 +0000 UTC m=+56.000565511"
Sep 15 12:03:20.188579 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:20.188487    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-298j7" event={"ID":"b6d14a02-3f21-44df-9db8-0f057f5eb567","Type":"ContainerStarted","Data":"f37cab1de535a848beaca29849664901820446488d431c0558d65b9bd0170e17"}
Sep 15 12:03:20.212448 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:20.212397    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-298j7" podStartSLOduration=1.485442548 podStartE2EDuration="5.212380626s" totalImagesPullingTime="3.726938078s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 12:03:15 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:03:20.211979377 +0000 UTC m=+57.002901887" watchObservedRunningTime="2026-09-15 12:03:20.212380626 +0000 UTC m=+57.003303137"
Sep 15 12:03:24.556251 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:24.556213    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-m7kmx"]
Sep 15 12:03:24.560417 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:24.560396    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-m7kmx"
Sep 15 12:03:24.564186 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:24.564161    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 15 12:03:24.564375 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:24.564199    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 15 12:03:24.564375 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:24.564219    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-bh5pf\""
Sep 15 12:03:24.564375 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:24.564230    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 15 12:03:24.564375 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:24.564161    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 15 12:03:24.564375 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:24.564281    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 15 12:03:24.564686 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:24.564651    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 15 12:03:24.582806 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:24.582774    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-zm5bs"]
Sep 15 12:03:24.586275 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:24.586233    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-zm5bs"
Sep 15 12:03:24.588656 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:24.588637    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 15 12:03:24.588910 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:24.588896    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 15 12:03:24.589198 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:24.589178    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 15 12:03:24.589302 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:24.589222    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-5sflk\""
Sep 15 12:03:24.602711 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:24.602681    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-zm5bs"]
Sep 15 12:03:24.680696 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:24.680657    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/dcbfda12-861a-403f-b2c9-4aa9c1351251-node-exporter-accelerators-collector-config\") pod \"node-exporter-m7kmx\" (UID: \"dcbfda12-861a-403f-b2c9-4aa9c1351251\") " pod="openshift-monitoring/node-exporter-m7kmx"
Sep 15 12:03:24.680885 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:24.680714    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/dcbfda12-861a-403f-b2c9-4aa9c1351251-node-exporter-tls\") pod \"node-exporter-m7kmx\" (UID: \"dcbfda12-861a-403f-b2c9-4aa9c1351251\") " pod="openshift-monitoring/node-exporter-m7kmx"
Sep 15 12:03:24.680885 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:24.680798    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzgpl\" (UniqueName: \"kubernetes.io/projected/dcbfda12-861a-403f-b2c9-4aa9c1351251-kube-api-access-gzgpl\") pod \"node-exporter-m7kmx\" (UID: \"dcbfda12-861a-403f-b2c9-4aa9c1351251\") " pod="openshift-monitoring/node-exporter-m7kmx"
Sep 15 12:03:24.680885 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:24.680823    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/dcbfda12-861a-403f-b2c9-4aa9c1351251-node-exporter-textfile\") pod \"node-exporter-m7kmx\" (UID: \"dcbfda12-861a-403f-b2c9-4aa9c1351251\") " pod="openshift-monitoring/node-exporter-m7kmx"
Sep 15 12:03:24.680885 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:24.680844    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dcbfda12-861a-403f-b2c9-4aa9c1351251-metrics-client-ca\") pod \"node-exporter-m7kmx\" (UID: \"dcbfda12-861a-403f-b2c9-4aa9c1351251\") " pod="openshift-monitoring/node-exporter-m7kmx"
Sep 15 12:03:24.681034 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:24.680911    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/dcbfda12-861a-403f-b2c9-4aa9c1351251-root\") pod \"node-exporter-m7kmx\" (UID: \"dcbfda12-861a-403f-b2c9-4aa9c1351251\") " pod="openshift-monitoring/node-exporter-m7kmx"
Sep 15 12:03:24.681034 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:24.680927    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/dcbfda12-861a-403f-b2c9-4aa9c1351251-node-exporter-wtmp\") pod \"node-exporter-m7kmx\" (UID: \"dcbfda12-861a-403f-b2c9-4aa9c1351251\") " pod="openshift-monitoring/node-exporter-m7kmx"
Sep 15 12:03:24.681034 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:24.680944    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dcbfda12-861a-403f-b2c9-4aa9c1351251-sys\") pod \"node-exporter-m7kmx\" (UID: \"dcbfda12-861a-403f-b2c9-4aa9c1351251\") " pod="openshift-monitoring/node-exporter-m7kmx"
Sep 15 12:03:24.681034 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:24.680966    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dcbfda12-861a-403f-b2c9-4aa9c1351251-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-m7kmx\" (UID: \"dcbfda12-861a-403f-b2c9-4aa9c1351251\") " pod="openshift-monitoring/node-exporter-m7kmx"
Sep 15 12:03:24.782152 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:24.782111    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/dcbfda12-861a-403f-b2c9-4aa9c1351251-node-exporter-accelerators-collector-config\") pod \"node-exporter-m7kmx\" (UID: \"dcbfda12-861a-403f-b2c9-4aa9c1351251\") " pod="openshift-monitoring/node-exporter-m7kmx"
Sep 15 12:03:24.782152 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:24.782157    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/696d839f-64d5-4025-bf41-74dbacde6019-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-zm5bs\" (UID: \"696d839f-64d5-4025-bf41-74dbacde6019\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-zm5bs"
Sep 15 12:03:24.782392 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:24.782186    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jv7qq\" (UniqueName: \"kubernetes.io/projected/696d839f-64d5-4025-bf41-74dbacde6019-kube-api-access-jv7qq\") pod \"kube-state-metrics-59fbdd77d-zm5bs\" (UID: \"696d839f-64d5-4025-bf41-74dbacde6019\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-zm5bs"
Sep 15 12:03:24.782392 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:24.782219    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/dcbfda12-861a-403f-b2c9-4aa9c1351251-node-exporter-textfile\") pod \"node-exporter-m7kmx\" (UID: \"dcbfda12-861a-403f-b2c9-4aa9c1351251\") " pod="openshift-monitoring/node-exporter-m7kmx"
Sep 15 12:03:24.782392 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:24.782312    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dcbfda12-861a-403f-b2c9-4aa9c1351251-sys\") pod \"node-exporter-m7kmx\" (UID: \"dcbfda12-861a-403f-b2c9-4aa9c1351251\") " pod="openshift-monitoring/node-exporter-m7kmx"
Sep 15 12:03:24.782392 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:24.782346    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/dcbfda12-861a-403f-b2c9-4aa9c1351251-root\") pod \"node-exporter-m7kmx\" (UID: \"dcbfda12-861a-403f-b2c9-4aa9c1351251\") " pod="openshift-monitoring/node-exporter-m7kmx"
Sep 15 12:03:24.782392 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:24.782369    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/dcbfda12-861a-403f-b2c9-4aa9c1351251-node-exporter-wtmp\") pod \"node-exporter-m7kmx\" (UID: \"dcbfda12-861a-403f-b2c9-4aa9c1351251\") " pod="openshift-monitoring/node-exporter-m7kmx"
Sep 15 12:03:24.782555 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:24.782414    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dcbfda12-861a-403f-b2c9-4aa9c1351251-sys\") pod \"node-exporter-m7kmx\" (UID: \"dcbfda12-861a-403f-b2c9-4aa9c1351251\") " pod="openshift-monitoring/node-exporter-m7kmx"
Sep 15 12:03:24.782555 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:24.782423    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/696d839f-64d5-4025-bf41-74dbacde6019-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-zm5bs\" (UID: \"696d839f-64d5-4025-bf41-74dbacde6019\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-zm5bs"
Sep 15 12:03:24.782555 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:24.782433    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/dcbfda12-861a-403f-b2c9-4aa9c1351251-root\") pod \"node-exporter-m7kmx\" (UID: \"dcbfda12-861a-403f-b2c9-4aa9c1351251\") " pod="openshift-monitoring/node-exporter-m7kmx"
Sep 15 12:03:24.782555 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:24.782468    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/dcbfda12-861a-403f-b2c9-4aa9c1351251-node-exporter-tls\") pod \"node-exporter-m7kmx\" (UID: \"dcbfda12-861a-403f-b2c9-4aa9c1351251\") " pod="openshift-monitoring/node-exporter-m7kmx"
Sep 15 12:03:24.782555 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:24.782510    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/696d839f-64d5-4025-bf41-74dbacde6019-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-zm5bs\" (UID: \"696d839f-64d5-4025-bf41-74dbacde6019\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-zm5bs"
Sep 15 12:03:24.782555 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:24.782543    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/696d839f-64d5-4025-bf41-74dbacde6019-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-zm5bs\" (UID: \"696d839f-64d5-4025-bf41-74dbacde6019\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-zm5bs"
Sep 15 12:03:24.782555 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:24.782550    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/dcbfda12-861a-403f-b2c9-4aa9c1351251-node-exporter-wtmp\") pod \"node-exporter-m7kmx\" (UID: \"dcbfda12-861a-403f-b2c9-4aa9c1351251\") " pod="openshift-monitoring/node-exporter-m7kmx"
Sep 15 12:03:24.782796 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:24.782580    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzgpl\" (UniqueName: \"kubernetes.io/projected/dcbfda12-861a-403f-b2c9-4aa9c1351251-kube-api-access-gzgpl\") pod \"node-exporter-m7kmx\" (UID: \"dcbfda12-861a-403f-b2c9-4aa9c1351251\") " pod="openshift-monitoring/node-exporter-m7kmx"
Sep 15 12:03:24.782796 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:24.782581    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/dcbfda12-861a-403f-b2c9-4aa9c1351251-node-exporter-textfile\") pod \"node-exporter-m7kmx\" (UID: \"dcbfda12-861a-403f-b2c9-4aa9c1351251\") " pod="openshift-monitoring/node-exporter-m7kmx"
Sep 15 12:03:24.782796 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:24.782612    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dcbfda12-861a-403f-b2c9-4aa9c1351251-metrics-client-ca\") pod \"node-exporter-m7kmx\" (UID: \"dcbfda12-861a-403f-b2c9-4aa9c1351251\") " pod="openshift-monitoring/node-exporter-m7kmx"
Sep 15 12:03:24.782796 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:24.782642    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/696d839f-64d5-4025-bf41-74dbacde6019-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-zm5bs\" (UID: \"696d839f-64d5-4025-bf41-74dbacde6019\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-zm5bs"
Sep 15 12:03:24.782796 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:24.782673    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dcbfda12-861a-403f-b2c9-4aa9c1351251-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-m7kmx\" (UID: \"dcbfda12-861a-403f-b2c9-4aa9c1351251\") " pod="openshift-monitoring/node-exporter-m7kmx"
Sep 15 12:03:24.782960 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:24.782875    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/dcbfda12-861a-403f-b2c9-4aa9c1351251-node-exporter-accelerators-collector-config\") pod \"node-exporter-m7kmx\" (UID: \"dcbfda12-861a-403f-b2c9-4aa9c1351251\") " pod="openshift-monitoring/node-exporter-m7kmx"
Sep 15 12:03:24.783040 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:24.783021    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dcbfda12-861a-403f-b2c9-4aa9c1351251-metrics-client-ca\") pod \"node-exporter-m7kmx\" (UID: \"dcbfda12-861a-403f-b2c9-4aa9c1351251\") " pod="openshift-monitoring/node-exporter-m7kmx"
Sep 15 12:03:24.786371 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:24.786331    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dcbfda12-861a-403f-b2c9-4aa9c1351251-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-m7kmx\" (UID: \"dcbfda12-861a-403f-b2c9-4aa9c1351251\") " pod="openshift-monitoring/node-exporter-m7kmx"
Sep 15 12:03:24.786371 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:24.786361    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/dcbfda12-861a-403f-b2c9-4aa9c1351251-node-exporter-tls\") pod \"node-exporter-m7kmx\" (UID: \"dcbfda12-861a-403f-b2c9-4aa9c1351251\") " pod="openshift-monitoring/node-exporter-m7kmx"
Sep 15 12:03:24.790630 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:24.790604    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzgpl\" (UniqueName: \"kubernetes.io/projected/dcbfda12-861a-403f-b2c9-4aa9c1351251-kube-api-access-gzgpl\") pod \"node-exporter-m7kmx\" (UID: \"dcbfda12-861a-403f-b2c9-4aa9c1351251\") " pod="openshift-monitoring/node-exporter-m7kmx"
Sep 15 12:03:24.869419 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:24.869318    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-m7kmx"
Sep 15 12:03:24.882816 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:03:24.882783    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcbfda12_861a_403f_b2c9_4aa9c1351251.slice/crio-1959b64620c4912cd1848fb6148b164bab10a04b44d7c1b18415bdda494ae03f WatchSource:0}: Error finding container 1959b64620c4912cd1848fb6148b164bab10a04b44d7c1b18415bdda494ae03f: Status 404 returned error can't find the container with id 1959b64620c4912cd1848fb6148b164bab10a04b44d7c1b18415bdda494ae03f
Sep 15 12:03:24.883053 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:24.883029    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/696d839f-64d5-4025-bf41-74dbacde6019-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-zm5bs\" (UID: \"696d839f-64d5-4025-bf41-74dbacde6019\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-zm5bs"
Sep 15 12:03:24.883144 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:24.883084    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/696d839f-64d5-4025-bf41-74dbacde6019-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-zm5bs\" (UID: \"696d839f-64d5-4025-bf41-74dbacde6019\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-zm5bs"
Sep 15 12:03:24.883144 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:24.883112    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/696d839f-64d5-4025-bf41-74dbacde6019-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-zm5bs\" (UID: \"696d839f-64d5-4025-bf41-74dbacde6019\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-zm5bs"
Sep 15 12:03:24.883246 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:24.883152    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/696d839f-64d5-4025-bf41-74dbacde6019-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-zm5bs\" (UID: \"696d839f-64d5-4025-bf41-74dbacde6019\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-zm5bs"
Sep 15 12:03:24.883319 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:24.883276    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/696d839f-64d5-4025-bf41-74dbacde6019-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-zm5bs\" (UID: \"696d839f-64d5-4025-bf41-74dbacde6019\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-zm5bs"
Sep 15 12:03:24.883319 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:24.883314    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jv7qq\" (UniqueName: \"kubernetes.io/projected/696d839f-64d5-4025-bf41-74dbacde6019-kube-api-access-jv7qq\") pod \"kube-state-metrics-59fbdd77d-zm5bs\" (UID: \"696d839f-64d5-4025-bf41-74dbacde6019\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-zm5bs"
Sep 15 12:03:24.883910 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:24.883886    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/696d839f-64d5-4025-bf41-74dbacde6019-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-zm5bs\" (UID: \"696d839f-64d5-4025-bf41-74dbacde6019\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-zm5bs"
Sep 15 12:03:24.883910 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:24.883899    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/696d839f-64d5-4025-bf41-74dbacde6019-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-zm5bs\" (UID: \"696d839f-64d5-4025-bf41-74dbacde6019\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-zm5bs"
Sep 15 12:03:24.884335 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:24.884308    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/696d839f-64d5-4025-bf41-74dbacde6019-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-zm5bs\" (UID: \"696d839f-64d5-4025-bf41-74dbacde6019\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-zm5bs"
Sep 15 12:03:24.886118 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:24.886097    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/696d839f-64d5-4025-bf41-74dbacde6019-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-zm5bs\" (UID: \"696d839f-64d5-4025-bf41-74dbacde6019\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-zm5bs"
Sep 15 12:03:24.886349 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:24.886326    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/696d839f-64d5-4025-bf41-74dbacde6019-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-zm5bs\" (UID: \"696d839f-64d5-4025-bf41-74dbacde6019\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-zm5bs"
Sep 15 12:03:24.891293 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:24.891251    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jv7qq\" (UniqueName: \"kubernetes.io/projected/696d839f-64d5-4025-bf41-74dbacde6019-kube-api-access-jv7qq\") pod \"kube-state-metrics-59fbdd77d-zm5bs\" (UID: \"696d839f-64d5-4025-bf41-74dbacde6019\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-zm5bs"
Sep 15 12:03:24.894885 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:24.894865    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-zm5bs"
Sep 15 12:03:25.104998 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:25.104962    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-k62zz"
Sep 15 12:03:25.168620 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:25.168572    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-zm5bs"]
Sep 15 12:03:25.171779 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:03:25.171745    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod696d839f_64d5_4025_bf41_74dbacde6019.slice/crio-1eb273113a2eb80b040be279b349d09dc70b40c1dd4791258375cb9f046dcfcb WatchSource:0}: Error finding container 1eb273113a2eb80b040be279b349d09dc70b40c1dd4791258375cb9f046dcfcb: Status 404 returned error can't find the container with id 1eb273113a2eb80b040be279b349d09dc70b40c1dd4791258375cb9f046dcfcb
Sep 15 12:03:25.204974 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:25.204938    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-zm5bs" event={"ID":"696d839f-64d5-4025-bf41-74dbacde6019","Type":"ContainerStarted","Data":"1eb273113a2eb80b040be279b349d09dc70b40c1dd4791258375cb9f046dcfcb"}
Sep 15 12:03:25.206075 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:25.206043    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-m7kmx" event={"ID":"dcbfda12-861a-403f-b2c9-4aa9c1351251","Type":"ContainerStarted","Data":"1959b64620c4912cd1848fb6148b164bab10a04b44d7c1b18415bdda494ae03f"}
Sep 15 12:03:26.213185 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:26.213140    2555 generic.go:356] "Generic (PLEG): container finished" podID="dcbfda12-861a-403f-b2c9-4aa9c1351251" containerID="2a24b3538d3c8a1ef4088c8bba6a756a59d2473c90755cafafb92190450d7377" exitCode=0
Sep 15 12:03:26.213625 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:26.213194    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-m7kmx" event={"ID":"dcbfda12-861a-403f-b2c9-4aa9c1351251","Type":"ContainerDied","Data":"2a24b3538d3c8a1ef4088c8bba6a756a59d2473c90755cafafb92190450d7377"}
Sep 15 12:03:27.222967 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:27.222929    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-zm5bs" event={"ID":"696d839f-64d5-4025-bf41-74dbacde6019","Type":"ContainerStarted","Data":"a413e3d936d833860bbd3be151a33a2b88b59cd1f0c7c2e259a38f569bf6c6be"}
Sep 15 12:03:27.223444 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:27.223016    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-zm5bs" event={"ID":"696d839f-64d5-4025-bf41-74dbacde6019","Type":"ContainerStarted","Data":"9319ffb61fb688e55af6c1b6e809c87464bfd18ed6a8d8ad5b69e811e05cfa32"}
Sep 15 12:03:27.223444 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:27.223041    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-zm5bs" event={"ID":"696d839f-64d5-4025-bf41-74dbacde6019","Type":"ContainerStarted","Data":"baab7f7827665e5fdaae4fa26b8ee16deea832ec50b63c3b414f3a948e513c77"}
Sep 15 12:03:27.225140 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:27.225114    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-m7kmx" event={"ID":"dcbfda12-861a-403f-b2c9-4aa9c1351251","Type":"ContainerStarted","Data":"85e9ce58cffd0b9bcc8f2da09b8a68023f79d37c50239aee7d423d228e1f617b"}
Sep 15 12:03:27.225293 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:27.225164    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-m7kmx" event={"ID":"dcbfda12-861a-403f-b2c9-4aa9c1351251","Type":"ContainerStarted","Data":"0c306f308a9b6e7186f1725cb8efb1841003e396f26de51a794478680c2b917a"}
Sep 15 12:03:27.239958 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:27.239857    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-zm5bs" podStartSLOduration=2.014288896 podStartE2EDuration="3.239839145s" totalImagesPullingTime="1.225550249s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:03:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:03:27.23949305 +0000 UTC m=+64.030415580" watchObservedRunningTime="2026-09-15 12:03:27.239839145 +0000 UTC m=+64.030761663"
Sep 15 12:03:27.256603 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:27.256545    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-m7kmx" podStartSLOduration=2.227249645 podStartE2EDuration="3.256532444s" totalImagesPullingTime="967.573851ms" totalInitContainerRuntime="61.708948ms" isStatefulPod=true podCreationTimestamp="2026-09-15 12:03:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:03:27.255597795 +0000 UTC m=+64.046520305" watchObservedRunningTime="2026-09-15 12:03:27.256532444 +0000 UTC m=+64.047454955"
Sep 15 12:03:27.525423 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:27.525345    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-57f7f8b569-5nvq9"]
Sep 15 12:03:27.528870 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:27.528847    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-57f7f8b569-5nvq9"
Sep 15 12:03:27.531412 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:27.531388    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 15 12:03:27.531531 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:27.531390    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 15 12:03:27.531685 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:27.531667    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 15 12:03:27.531805 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:27.531788    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-gh48j\""
Sep 15 12:03:27.531906 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:27.531801    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 15 12:03:27.531906 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:27.531815    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 15 12:03:27.531906 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:27.531836    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-cf5oohttu5km8\""
Sep 15 12:03:27.541955 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:27.541934    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-57f7f8b569-5nvq9"]
Sep 15 12:03:27.602414 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:27.602379    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txmtj\" (UniqueName: \"kubernetes.io/projected/fac096e8-e179-4666-b6db-3fbf219ae265-kube-api-access-txmtj\") pod \"thanos-querier-57f7f8b569-5nvq9\" (UID: \"fac096e8-e179-4666-b6db-3fbf219ae265\") " pod="openshift-monitoring/thanos-querier-57f7f8b569-5nvq9"
Sep 15 12:03:27.602588 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:27.602424    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fac096e8-e179-4666-b6db-3fbf219ae265-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-57f7f8b569-5nvq9\" (UID: \"fac096e8-e179-4666-b6db-3fbf219ae265\") " pod="openshift-monitoring/thanos-querier-57f7f8b569-5nvq9"
Sep 15 12:03:27.602588 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:27.602448    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/fac096e8-e179-4666-b6db-3fbf219ae265-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-57f7f8b569-5nvq9\" (UID: \"fac096e8-e179-4666-b6db-3fbf219ae265\") " pod="openshift-monitoring/thanos-querier-57f7f8b569-5nvq9"
Sep 15 12:03:27.602588 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:27.602498    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/fac096e8-e179-4666-b6db-3fbf219ae265-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-57f7f8b569-5nvq9\" (UID: \"fac096e8-e179-4666-b6db-3fbf219ae265\") " pod="openshift-monitoring/thanos-querier-57f7f8b569-5nvq9"
Sep 15 12:03:27.602588 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:27.602577    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fac096e8-e179-4666-b6db-3fbf219ae265-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-57f7f8b569-5nvq9\" (UID: \"fac096e8-e179-4666-b6db-3fbf219ae265\") " pod="openshift-monitoring/thanos-querier-57f7f8b569-5nvq9"
Sep 15 12:03:27.602728 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:27.602615    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fac096e8-e179-4666-b6db-3fbf219ae265-secret-grpc-tls\") pod \"thanos-querier-57f7f8b569-5nvq9\" (UID: \"fac096e8-e179-4666-b6db-3fbf219ae265\") " pod="openshift-monitoring/thanos-querier-57f7f8b569-5nvq9"
Sep 15 12:03:27.602728 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:27.602656    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/fac096e8-e179-4666-b6db-3fbf219ae265-secret-thanos-querier-tls\") pod \"thanos-querier-57f7f8b569-5nvq9\" (UID: \"fac096e8-e179-4666-b6db-3fbf219ae265\") " pod="openshift-monitoring/thanos-querier-57f7f8b569-5nvq9"
Sep 15 12:03:27.602728 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:27.602678    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fac096e8-e179-4666-b6db-3fbf219ae265-metrics-client-ca\") pod \"thanos-querier-57f7f8b569-5nvq9\" (UID: \"fac096e8-e179-4666-b6db-3fbf219ae265\") " pod="openshift-monitoring/thanos-querier-57f7f8b569-5nvq9"
Sep 15 12:03:27.703162 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:27.703111    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fac096e8-e179-4666-b6db-3fbf219ae265-secret-grpc-tls\") pod \"thanos-querier-57f7f8b569-5nvq9\" (UID: \"fac096e8-e179-4666-b6db-3fbf219ae265\") " pod="openshift-monitoring/thanos-querier-57f7f8b569-5nvq9"
Sep 15 12:03:27.703162 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:27.703169    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/fac096e8-e179-4666-b6db-3fbf219ae265-secret-thanos-querier-tls\") pod \"thanos-querier-57f7f8b569-5nvq9\" (UID: \"fac096e8-e179-4666-b6db-3fbf219ae265\") " pod="openshift-monitoring/thanos-querier-57f7f8b569-5nvq9"
Sep 15 12:03:27.703448 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:27.703194    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fac096e8-e179-4666-b6db-3fbf219ae265-metrics-client-ca\") pod \"thanos-querier-57f7f8b569-5nvq9\" (UID: \"fac096e8-e179-4666-b6db-3fbf219ae265\") " pod="openshift-monitoring/thanos-querier-57f7f8b569-5nvq9"
Sep 15 12:03:27.703448 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:27.703236    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txmtj\" (UniqueName: \"kubernetes.io/projected/fac096e8-e179-4666-b6db-3fbf219ae265-kube-api-access-txmtj\") pod \"thanos-querier-57f7f8b569-5nvq9\" (UID: \"fac096e8-e179-4666-b6db-3fbf219ae265\") " pod="openshift-monitoring/thanos-querier-57f7f8b569-5nvq9"
Sep 15 12:03:27.703448 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:27.703300    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fac096e8-e179-4666-b6db-3fbf219ae265-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-57f7f8b569-5nvq9\" (UID: \"fac096e8-e179-4666-b6db-3fbf219ae265\") " pod="openshift-monitoring/thanos-querier-57f7f8b569-5nvq9"
Sep 15 12:03:27.703448 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:27.703352    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/fac096e8-e179-4666-b6db-3fbf219ae265-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-57f7f8b569-5nvq9\" (UID: \"fac096e8-e179-4666-b6db-3fbf219ae265\") " pod="openshift-monitoring/thanos-querier-57f7f8b569-5nvq9"
Sep 15 12:03:27.703448 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:27.703382    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/fac096e8-e179-4666-b6db-3fbf219ae265-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-57f7f8b569-5nvq9\" (UID: \"fac096e8-e179-4666-b6db-3fbf219ae265\") " pod="openshift-monitoring/thanos-querier-57f7f8b569-5nvq9"
Sep 15 12:03:27.703448 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:27.703442    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fac096e8-e179-4666-b6db-3fbf219ae265-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-57f7f8b569-5nvq9\" (UID: \"fac096e8-e179-4666-b6db-3fbf219ae265\") " pod="openshift-monitoring/thanos-querier-57f7f8b569-5nvq9"
Sep 15 12:03:27.704140 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:27.704106    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fac096e8-e179-4666-b6db-3fbf219ae265-metrics-client-ca\") pod \"thanos-querier-57f7f8b569-5nvq9\" (UID: \"fac096e8-e179-4666-b6db-3fbf219ae265\") " pod="openshift-monitoring/thanos-querier-57f7f8b569-5nvq9"
Sep 15 12:03:27.706320 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:27.705879    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fac096e8-e179-4666-b6db-3fbf219ae265-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-57f7f8b569-5nvq9\" (UID: \"fac096e8-e179-4666-b6db-3fbf219ae265\") " pod="openshift-monitoring/thanos-querier-57f7f8b569-5nvq9"
Sep 15 12:03:27.706320 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:27.705884    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/fac096e8-e179-4666-b6db-3fbf219ae265-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-57f7f8b569-5nvq9\" (UID: \"fac096e8-e179-4666-b6db-3fbf219ae265\") " pod="openshift-monitoring/thanos-querier-57f7f8b569-5nvq9"
Sep 15 12:03:27.706320 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:27.706170    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/fac096e8-e179-4666-b6db-3fbf219ae265-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-57f7f8b569-5nvq9\" (UID: \"fac096e8-e179-4666-b6db-3fbf219ae265\") " pod="openshift-monitoring/thanos-querier-57f7f8b569-5nvq9"
Sep 15 12:03:27.706320 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:27.706246    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/fac096e8-e179-4666-b6db-3fbf219ae265-secret-thanos-querier-tls\") pod \"thanos-querier-57f7f8b569-5nvq9\" (UID: \"fac096e8-e179-4666-b6db-3fbf219ae265\") " pod="openshift-monitoring/thanos-querier-57f7f8b569-5nvq9"
Sep 15 12:03:27.706320 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:27.706284    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fac096e8-e179-4666-b6db-3fbf219ae265-secret-grpc-tls\") pod \"thanos-querier-57f7f8b569-5nvq9\" (UID: \"fac096e8-e179-4666-b6db-3fbf219ae265\") " pod="openshift-monitoring/thanos-querier-57f7f8b569-5nvq9"
Sep 15 12:03:27.706320 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:27.706290    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fac096e8-e179-4666-b6db-3fbf219ae265-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-57f7f8b569-5nvq9\" (UID: \"fac096e8-e179-4666-b6db-3fbf219ae265\") " pod="openshift-monitoring/thanos-querier-57f7f8b569-5nvq9"
Sep 15 12:03:27.711104 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:27.711082    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-txmtj\" (UniqueName: \"kubernetes.io/projected/fac096e8-e179-4666-b6db-3fbf219ae265-kube-api-access-txmtj\") pod \"thanos-querier-57f7f8b569-5nvq9\" (UID: \"fac096e8-e179-4666-b6db-3fbf219ae265\") " pod="openshift-monitoring/thanos-querier-57f7f8b569-5nvq9"
Sep 15 12:03:27.839019 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:27.838925    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-57f7f8b569-5nvq9"
Sep 15 12:03:28.141013 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:28.140987    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-57f7f8b569-5nvq9"]
Sep 15 12:03:28.143100 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:03:28.143072    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfac096e8_e179_4666_b6db_3fbf219ae265.slice/crio-6c374642b4881288ff3e276411dfcc93e9ed94f03eb17b29cd570361c7cb7d73 WatchSource:0}: Error finding container 6c374642b4881288ff3e276411dfcc93e9ed94f03eb17b29cd570361c7cb7d73: Status 404 returned error can't find the container with id 6c374642b4881288ff3e276411dfcc93e9ed94f03eb17b29cd570361c7cb7d73
Sep 15 12:03:28.230577 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:28.230540    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57f7f8b569-5nvq9" event={"ID":"fac096e8-e179-4666-b6db-3fbf219ae265","Type":"ContainerStarted","Data":"6c374642b4881288ff3e276411dfcc93e9ed94f03eb17b29cd570361c7cb7d73"}
Sep 15 12:03:29.191452 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:29.191419    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-2vzvd"
Sep 15 12:03:29.335569 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:29.335536    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-6st6v"]
Sep 15 12:03:29.338784 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:29.338760    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-6st6v"
Sep 15 12:03:29.341503 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:29.341474    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-dswmp\""
Sep 15 12:03:29.341779 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:29.341760    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 15 12:03:29.349400 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:29.349376    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-6st6v"]
Sep 15 12:03:29.421313 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:29.421249    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b1f7a86b-48a2-4ac0-b57d-2b386a559c89-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-6st6v\" (UID: \"b1f7a86b-48a2-4ac0-b57d-2b386a559c89\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-6st6v"
Sep 15 12:03:29.421489 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:29.421361    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blpk2\" (UniqueName: \"kubernetes.io/projected/b1f7a86b-48a2-4ac0-b57d-2b386a559c89-kube-api-access-blpk2\") pod \"monitoring-plugin-5d85778b9d-6st6v\" (UID: \"b1f7a86b-48a2-4ac0-b57d-2b386a559c89\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-6st6v"
Sep 15 12:03:29.524329 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:29.524224    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b1f7a86b-48a2-4ac0-b57d-2b386a559c89-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-6st6v\" (UID: \"b1f7a86b-48a2-4ac0-b57d-2b386a559c89\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-6st6v"
Sep 15 12:03:29.524329 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:29.524325    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blpk2\" (UniqueName: \"kubernetes.io/projected/b1f7a86b-48a2-4ac0-b57d-2b386a559c89-kube-api-access-blpk2\") pod \"monitoring-plugin-5d85778b9d-6st6v\" (UID: \"b1f7a86b-48a2-4ac0-b57d-2b386a559c89\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-6st6v"
Sep 15 12:03:29.524650 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:03:29.524410    2555 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Sep 15 12:03:29.524650 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:03:29.524505    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1f7a86b-48a2-4ac0-b57d-2b386a559c89-monitoring-plugin-cert podName:b1f7a86b-48a2-4ac0-b57d-2b386a559c89 nodeName:}" failed. No retries permitted until 2026-09-15 12:03:30.024481699 +0000 UTC m=+66.815404217 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/b1f7a86b-48a2-4ac0-b57d-2b386a559c89-monitoring-plugin-cert") pod "monitoring-plugin-5d85778b9d-6st6v" (UID: "b1f7a86b-48a2-4ac0-b57d-2b386a559c89") : secret "monitoring-plugin-cert" not found
Sep 15 12:03:29.535502 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:29.535445    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-blpk2\" (UniqueName: \"kubernetes.io/projected/b1f7a86b-48a2-4ac0-b57d-2b386a559c89-kube-api-access-blpk2\") pod \"monitoring-plugin-5d85778b9d-6st6v\" (UID: \"b1f7a86b-48a2-4ac0-b57d-2b386a559c89\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-6st6v"
Sep 15 12:03:29.625504 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:29.625466    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/07f9ddc4-14d7-4b04-9797-67507d4063bc-original-pull-secret\") pod \"global-pull-secret-syncer-g25rd\" (UID: \"07f9ddc4-14d7-4b04-9797-67507d4063bc\") " pod="kube-system/global-pull-secret-syncer-g25rd"
Sep 15 12:03:29.625691 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:29.625604    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9c68d51-9107-4cd4-a57d-6f76c0762106-metrics-certs\") pod \"network-metrics-daemon-bdjwd\" (UID: \"f9c68d51-9107-4cd4-a57d-6f76c0762106\") " pod="openshift-multus/network-metrics-daemon-bdjwd"
Sep 15 12:03:29.628753 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:29.628730    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 15 12:03:29.628753 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:29.628742    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 15 12:03:29.638058 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:29.638025    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9c68d51-9107-4cd4-a57d-6f76c0762106-metrics-certs\") pod \"network-metrics-daemon-bdjwd\" (UID: \"f9c68d51-9107-4cd4-a57d-6f76c0762106\") " pod="openshift-multus/network-metrics-daemon-bdjwd"
Sep 15 12:03:29.638306 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:29.638256    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/07f9ddc4-14d7-4b04-9797-67507d4063bc-original-pull-secret\") pod \"global-pull-secret-syncer-g25rd\" (UID: \"07f9ddc4-14d7-4b04-9797-67507d4063bc\") " pod="kube-system/global-pull-secret-syncer-g25rd"
Sep 15 12:03:29.725418 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:29.725384    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g25rd"
Sep 15 12:03:29.726672 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:29.726639    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm9js\" (UniqueName: \"kubernetes.io/projected/090605d8-b2cc-4a2f-9f09-c22538254886-kube-api-access-zm9js\") pod \"network-check-target-wrxv2\" (UID: \"090605d8-b2cc-4a2f-9f09-c22538254886\") " pod="openshift-network-diagnostics/network-check-target-wrxv2"
Sep 15 12:03:29.729198 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:29.729175    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 15 12:03:29.741441 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:29.741409    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 15 12:03:29.752749 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:29.752718    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm9js\" (UniqueName: \"kubernetes.io/projected/090605d8-b2cc-4a2f-9f09-c22538254886-kube-api-access-zm9js\") pod \"network-check-target-wrxv2\" (UID: \"090605d8-b2cc-4a2f-9f09-c22538254886\") " pod="openshift-network-diagnostics/network-check-target-wrxv2"
Sep 15 12:03:29.924748 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:29.924719    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-k48ps\""
Sep 15 12:03:29.928035 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:29.928003    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9l82m\""
Sep 15 12:03:29.932294 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:29.932255    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wrxv2"
Sep 15 12:03:29.936565 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:29.936533    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdjwd"
Sep 15 12:03:30.029359 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:30.029325    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b1f7a86b-48a2-4ac0-b57d-2b386a559c89-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-6st6v\" (UID: \"b1f7a86b-48a2-4ac0-b57d-2b386a559c89\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-6st6v"
Sep 15 12:03:30.032018 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:30.031990    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b1f7a86b-48a2-4ac0-b57d-2b386a559c89-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-6st6v\" (UID: \"b1f7a86b-48a2-4ac0-b57d-2b386a559c89\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-6st6v"
Sep 15 12:03:30.254953 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:30.254921    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-6st6v"
Sep 15 12:03:30.518647 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:30.518568    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wrxv2"]
Sep 15 12:03:30.521111 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:30.521084    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-g25rd"]
Sep 15 12:03:30.521978 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:30.521959    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bdjwd"]
Sep 15 12:03:30.574523 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:30.574491    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-6st6v"]
Sep 15 12:03:30.622045 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:03:30.622009    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod090605d8_b2cc_4a2f_9f09_c22538254886.slice/crio-e53761f03e59485794dc91bb9ebc1de175ef28aff9909c0dc0b8c7b9c0c1b1e2 WatchSource:0}: Error finding container e53761f03e59485794dc91bb9ebc1de175ef28aff9909c0dc0b8c7b9c0c1b1e2: Status 404 returned error can't find the container with id e53761f03e59485794dc91bb9ebc1de175ef28aff9909c0dc0b8c7b9c0c1b1e2
Sep 15 12:03:30.622696 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:03:30.622662    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9c68d51_9107_4cd4_a57d_6f76c0762106.slice/crio-0735fcffc11b9a750646075ebf070564ca78b3bb59ab32d93fad3de59782c531 WatchSource:0}: Error finding container 0735fcffc11b9a750646075ebf070564ca78b3bb59ab32d93fad3de59782c531: Status 404 returned error can't find the container with id 0735fcffc11b9a750646075ebf070564ca78b3bb59ab32d93fad3de59782c531
Sep 15 12:03:30.623662 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:03:30.623610    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07f9ddc4_14d7_4b04_9797_67507d4063bc.slice/crio-f0e7b9844da8e7ac6ae42f84da62e2a2fdcc89c52c08e0ff59c63b4151ef4ef7 WatchSource:0}: Error finding container f0e7b9844da8e7ac6ae42f84da62e2a2fdcc89c52c08e0ff59c63b4151ef4ef7: Status 404 returned error can't find the container with id f0e7b9844da8e7ac6ae42f84da62e2a2fdcc89c52c08e0ff59c63b4151ef4ef7
Sep 15 12:03:30.624642 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:03:30.624578    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1f7a86b_48a2_4ac0_b57d_2b386a559c89.slice/crio-96de49f6c2062848c8b3b833a838c2f6b125467a41ba05795b4b137c384116eb WatchSource:0}: Error finding container 96de49f6c2062848c8b3b833a838c2f6b125467a41ba05795b4b137c384116eb: Status 404 returned error can't find the container with id 96de49f6c2062848c8b3b833a838c2f6b125467a41ba05795b4b137c384116eb
Sep 15 12:03:31.246520 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:31.246453    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-g25rd" event={"ID":"07f9ddc4-14d7-4b04-9797-67507d4063bc","Type":"ContainerStarted","Data":"f0e7b9844da8e7ac6ae42f84da62e2a2fdcc89c52c08e0ff59c63b4151ef4ef7"}
Sep 15 12:03:31.248332 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:31.248241    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-6st6v" event={"ID":"b1f7a86b-48a2-4ac0-b57d-2b386a559c89","Type":"ContainerStarted","Data":"96de49f6c2062848c8b3b833a838c2f6b125467a41ba05795b4b137c384116eb"}
Sep 15 12:03:31.251251 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:31.251213    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57f7f8b569-5nvq9" event={"ID":"fac096e8-e179-4666-b6db-3fbf219ae265","Type":"ContainerStarted","Data":"2e8f0c4916c6f37f79caaf4feaab5f0cf3e02a3491c122955dc781507f15c29f"}
Sep 15 12:03:31.251414 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:31.251319    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57f7f8b569-5nvq9" event={"ID":"fac096e8-e179-4666-b6db-3fbf219ae265","Type":"ContainerStarted","Data":"1870bf31bfec30f9ebd631dca2654ee5f3c55c4059b620c505cc7c37e60d9e92"}
Sep 15 12:03:31.251414 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:31.251363    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57f7f8b569-5nvq9" event={"ID":"fac096e8-e179-4666-b6db-3fbf219ae265","Type":"ContainerStarted","Data":"1b858692146ce8d2d2261d8b34c400a32dba888d79ee426b6c8e44032f8d7a6a"}
Sep 15 12:03:31.252939 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:31.252898    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bdjwd" event={"ID":"f9c68d51-9107-4cd4-a57d-6f76c0762106","Type":"ContainerStarted","Data":"0735fcffc11b9a750646075ebf070564ca78b3bb59ab32d93fad3de59782c531"}
Sep 15 12:03:31.254418 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:31.254388    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wrxv2" event={"ID":"090605d8-b2cc-4a2f-9f09-c22538254886","Type":"ContainerStarted","Data":"e53761f03e59485794dc91bb9ebc1de175ef28aff9909c0dc0b8c7b9c0c1b1e2"}
Sep 15 12:03:32.260995 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:32.260954    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bdjwd" event={"ID":"f9c68d51-9107-4cd4-a57d-6f76c0762106","Type":"ContainerStarted","Data":"33867c6e29e5d10ed7482dbf74482629a2e465b46310c99da2fbd13ca47fb09b"}
Sep 15 12:03:32.261568 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:32.261059    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bdjwd" event={"ID":"f9c68d51-9107-4cd4-a57d-6f76c0762106","Type":"ContainerStarted","Data":"97051d7e410cfc260421ed8a319f8d1dcd7fc257a9744fc8f6ba99c4c3494883"}
Sep 15 12:03:32.278864 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:32.278797    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-bdjwd" podStartSLOduration=67.167749681 podStartE2EDuration="1m8.27877421s" totalImagesPullingTime="1.111024529s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:02:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:03:32.277434481 +0000 UTC m=+69.068357016" watchObservedRunningTime="2026-09-15 12:03:32.27877421 +0000 UTC m=+69.069696723"
Sep 15 12:03:33.270123 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:33.270082    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57f7f8b569-5nvq9" event={"ID":"fac096e8-e179-4666-b6db-3fbf219ae265","Type":"ContainerStarted","Data":"d0243aea2c4f6fefff828e6ac6df8239666818d31e1aebdc6af456f982fc8567"}
Sep 15 12:03:33.270551 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:33.270173    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57f7f8b569-5nvq9" event={"ID":"fac096e8-e179-4666-b6db-3fbf219ae265","Type":"ContainerStarted","Data":"f573cf9924a9ddcae5b8228a313e1f5f5d85e978db0576b1ca0d2b5e777dd71f"}
Sep 15 12:03:33.270551 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:33.270223    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57f7f8b569-5nvq9" event={"ID":"fac096e8-e179-4666-b6db-3fbf219ae265","Type":"ContainerStarted","Data":"ab0d7e52da2f1ebe4f2ca6da20a0a23227fbf85900c69e043a366dc42f932f2a"}
Sep 15 12:03:33.294358 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:33.294296    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-57f7f8b569-5nvq9" podStartSLOduration=2.475849924 podStartE2EDuration="6.294256504s" totalImagesPullingTime="3.81840658s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:03:27 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:03:33.293061973 +0000 UTC m=+70.083984498" watchObservedRunningTime="2026-09-15 12:03:33.294256504 +0000 UTC m=+70.085179018"
Sep 15 12:03:34.274664 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:34.274629    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-57f7f8b569-5nvq9"
Sep 15 12:03:35.285385 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:35.285357    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-57f7f8b569-5nvq9"
Sep 15 12:03:35.356194 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:35.355832    2555 patch_prober.go:28] interesting pod/image-registry-8678457cc7-t287r container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 15 12:03:35.356194 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:35.355912    2555 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-8678457cc7-t287r" podUID="62c63012-fb07-478e-b95e-01a2d6cbd475" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 12:03:36.283423 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:36.283390    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wrxv2" event={"ID":"090605d8-b2cc-4a2f-9f09-c22538254886","Type":"ContainerStarted","Data":"5b1372a1608319a30bcced64c0677606ab9bcdeae1cfbf6c8b3fd17a43e97343"}
Sep 15 12:03:36.283575 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:36.283464    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-wrxv2"
Sep 15 12:03:36.284745 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:36.284724    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-g25rd" event={"ID":"07f9ddc4-14d7-4b04-9797-67507d4063bc","Type":"ContainerStarted","Data":"f1cf2eda260541fb9381376e7fac73a4c6f3a49155dc47bcb8eacceb559d82a7"}
Sep 15 12:03:36.299187 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:36.299137    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-wrxv2" podStartSLOduration=67.510895626 podStartE2EDuration="1m12.29912154s" totalImagesPullingTime="4.788225914s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:02:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:03:36.298144674 +0000 UTC m=+73.089067207" watchObservedRunningTime="2026-09-15 12:03:36.29912154 +0000 UTC m=+73.090044048"
Sep 15 12:03:36.312442 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:36.312394    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-g25rd" podStartSLOduration=67.518826114 podStartE2EDuration="1m12.312378573s" totalImagesPullingTime="4.793552459s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 12:02:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:03:36.311857039 +0000 UTC m=+73.102779568" watchObservedRunningTime="2026-09-15 12:03:36.312378573 +0000 UTC m=+73.103301084"
Sep 15 12:03:37.171415 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:37.171385    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-8678457cc7-t287r"
Sep 15 12:03:37.289520 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:37.289427    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-6st6v" event={"ID":"b1f7a86b-48a2-4ac0-b57d-2b386a559c89","Type":"ContainerStarted","Data":"6ceb40a7f8647c81de7355c04c86cd42e684ca2e68e5cad3703a897e7f7fe4fc"}
Sep 15 12:03:37.289758 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:37.289735    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-6st6v"
Sep 15 12:03:37.294465 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:37.294441    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-6st6v"
Sep 15 12:03:37.307644 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:37.307471    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-6st6v" podStartSLOduration=2.692895115 podStartE2EDuration="8.307457444s" totalImagesPullingTime="5.614562329s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:03:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:03:37.307344235 +0000 UTC m=+74.098266768" watchObservedRunningTime="2026-09-15 12:03:37.307457444 +0000 UTC m=+74.098379954"
Sep 15 12:03:51.290189 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:03:51.290078    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-tx76t_e21c5438-c5ca-4375-81e8-db67f537528e/serve-healthcheck-canary/0.log"
Sep 15 12:04:07.292098 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:04:07.292061    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-wrxv2"
Sep 15 12:04:48.815144 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:04:48.815106    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-7998877c76-lzgxk"]
Sep 15 12:04:48.818639 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:04:48.818621    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7998877c76-lzgxk"
Sep 15 12:04:48.821215 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:04:48.821192    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 15 12:04:48.821845 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:04:48.821823    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 15 12:04:48.821929 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:04:48.821825    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-wlpb8\""
Sep 15 12:04:48.821929 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:04:48.821825    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 15 12:04:48.822091 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:04:48.822074    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 15 12:04:48.822531 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:04:48.822514    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 15 12:04:48.827484 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:04:48.827461    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44a9b0fc-aeb3-403e-94ab-9d09e3fa44fc-serving-certs-ca-bundle\") pod \"telemeter-client-7998877c76-lzgxk\" (UID: \"44a9b0fc-aeb3-403e-94ab-9d09e3fa44fc\") " pod="openshift-monitoring/telemeter-client-7998877c76-lzgxk"
Sep 15 12:04:48.827561 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:04:48.827496    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/44a9b0fc-aeb3-403e-94ab-9d09e3fa44fc-secret-telemeter-client\") pod \"telemeter-client-7998877c76-lzgxk\" (UID: \"44a9b0fc-aeb3-403e-94ab-9d09e3fa44fc\") " pod="openshift-monitoring/telemeter-client-7998877c76-lzgxk"
Sep 15 12:04:48.827561 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:04:48.827527    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/44a9b0fc-aeb3-403e-94ab-9d09e3fa44fc-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7998877c76-lzgxk\" (UID: \"44a9b0fc-aeb3-403e-94ab-9d09e3fa44fc\") " pod="openshift-monitoring/telemeter-client-7998877c76-lzgxk"
Sep 15 12:04:48.827632 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:04:48.827580    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qm7wb\" (UniqueName: \"kubernetes.io/projected/44a9b0fc-aeb3-403e-94ab-9d09e3fa44fc-kube-api-access-qm7wb\") pod \"telemeter-client-7998877c76-lzgxk\" (UID: \"44a9b0fc-aeb3-403e-94ab-9d09e3fa44fc\") " pod="openshift-monitoring/telemeter-client-7998877c76-lzgxk"
Sep 15 12:04:48.827632 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:04:48.827602    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/44a9b0fc-aeb3-403e-94ab-9d09e3fa44fc-federate-client-tls\") pod \"telemeter-client-7998877c76-lzgxk\" (UID: \"44a9b0fc-aeb3-403e-94ab-9d09e3fa44fc\") " pod="openshift-monitoring/telemeter-client-7998877c76-lzgxk"
Sep 15 12:04:48.827632 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:04:48.827623    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 15 12:04:48.827721 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:04:48.827658    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/44a9b0fc-aeb3-403e-94ab-9d09e3fa44fc-telemeter-client-tls\") pod \"telemeter-client-7998877c76-lzgxk\" (UID: \"44a9b0fc-aeb3-403e-94ab-9d09e3fa44fc\") " pod="openshift-monitoring/telemeter-client-7998877c76-lzgxk"
Sep 15 12:04:48.827721 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:04:48.827682    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/44a9b0fc-aeb3-403e-94ab-9d09e3fa44fc-metrics-client-ca\") pod \"telemeter-client-7998877c76-lzgxk\" (UID: \"44a9b0fc-aeb3-403e-94ab-9d09e3fa44fc\") " pod="openshift-monitoring/telemeter-client-7998877c76-lzgxk"
Sep 15 12:04:48.827721 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:04:48.827705    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44a9b0fc-aeb3-403e-94ab-9d09e3fa44fc-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7998877c76-lzgxk\" (UID: \"44a9b0fc-aeb3-403e-94ab-9d09e3fa44fc\") " pod="openshift-monitoring/telemeter-client-7998877c76-lzgxk"
Sep 15 12:04:48.832672 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:04:48.832649    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7998877c76-lzgxk"]
Sep 15 12:04:48.928276 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:04:48.928235    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/44a9b0fc-aeb3-403e-94ab-9d09e3fa44fc-telemeter-client-tls\") pod \"telemeter-client-7998877c76-lzgxk\" (UID: \"44a9b0fc-aeb3-403e-94ab-9d09e3fa44fc\") " pod="openshift-monitoring/telemeter-client-7998877c76-lzgxk"
Sep 15 12:04:48.928459 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:04:48.928294    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/44a9b0fc-aeb3-403e-94ab-9d09e3fa44fc-metrics-client-ca\") pod \"telemeter-client-7998877c76-lzgxk\" (UID: \"44a9b0fc-aeb3-403e-94ab-9d09e3fa44fc\") " pod="openshift-monitoring/telemeter-client-7998877c76-lzgxk"
Sep 15 12:04:48.928459 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:04:48.928315    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44a9b0fc-aeb3-403e-94ab-9d09e3fa44fc-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7998877c76-lzgxk\" (UID: \"44a9b0fc-aeb3-403e-94ab-9d09e3fa44fc\") " pod="openshift-monitoring/telemeter-client-7998877c76-lzgxk"
Sep 15 12:04:48.928459 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:04:48.928350    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44a9b0fc-aeb3-403e-94ab-9d09e3fa44fc-serving-certs-ca-bundle\") pod \"telemeter-client-7998877c76-lzgxk\" (UID: \"44a9b0fc-aeb3-403e-94ab-9d09e3fa44fc\") " pod="openshift-monitoring/telemeter-client-7998877c76-lzgxk"
Sep 15 12:04:48.928459 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:04:48.928374    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/44a9b0fc-aeb3-403e-94ab-9d09e3fa44fc-secret-telemeter-client\") pod \"telemeter-client-7998877c76-lzgxk\" (UID: \"44a9b0fc-aeb3-403e-94ab-9d09e3fa44fc\") " pod="openshift-monitoring/telemeter-client-7998877c76-lzgxk"
Sep 15 12:04:48.928459 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:04:48.928402    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/44a9b0fc-aeb3-403e-94ab-9d09e3fa44fc-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7998877c76-lzgxk\" (UID: \"44a9b0fc-aeb3-403e-94ab-9d09e3fa44fc\") " pod="openshift-monitoring/telemeter-client-7998877c76-lzgxk"
Sep 15 12:04:48.928459 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:04:48.928445    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qm7wb\" (UniqueName: \"kubernetes.io/projected/44a9b0fc-aeb3-403e-94ab-9d09e3fa44fc-kube-api-access-qm7wb\") pod \"telemeter-client-7998877c76-lzgxk\" (UID: \"44a9b0fc-aeb3-403e-94ab-9d09e3fa44fc\") " pod="openshift-monitoring/telemeter-client-7998877c76-lzgxk"
Sep 15 12:04:48.928720 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:04:48.928483    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/44a9b0fc-aeb3-403e-94ab-9d09e3fa44fc-federate-client-tls\") pod \"telemeter-client-7998877c76-lzgxk\" (UID: \"44a9b0fc-aeb3-403e-94ab-9d09e3fa44fc\") " pod="openshift-monitoring/telemeter-client-7998877c76-lzgxk"
Sep 15 12:04:48.929816 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:04:48.929783    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44a9b0fc-aeb3-403e-94ab-9d09e3fa44fc-serving-certs-ca-bundle\") pod \"telemeter-client-7998877c76-lzgxk\" (UID: \"44a9b0fc-aeb3-403e-94ab-9d09e3fa44fc\") " pod="openshift-monitoring/telemeter-client-7998877c76-lzgxk"
Sep 15 12:04:48.929993 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:04:48.929901    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/44a9b0fc-aeb3-403e-94ab-9d09e3fa44fc-metrics-client-ca\") pod \"telemeter-client-7998877c76-lzgxk\" (UID: \"44a9b0fc-aeb3-403e-94ab-9d09e3fa44fc\") " pod="openshift-monitoring/telemeter-client-7998877c76-lzgxk"
Sep 15 12:04:48.931864 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:04:48.931835    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44a9b0fc-aeb3-403e-94ab-9d09e3fa44fc-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7998877c76-lzgxk\" (UID: \"44a9b0fc-aeb3-403e-94ab-9d09e3fa44fc\") " pod="openshift-monitoring/telemeter-client-7998877c76-lzgxk"
Sep 15 12:04:48.932054 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:04:48.932030    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/44a9b0fc-aeb3-403e-94ab-9d09e3fa44fc-telemeter-client-tls\") pod \"telemeter-client-7998877c76-lzgxk\" (UID: \"44a9b0fc-aeb3-403e-94ab-9d09e3fa44fc\") " pod="openshift-monitoring/telemeter-client-7998877c76-lzgxk"
Sep 15 12:04:48.935465 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:04:48.935436    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/44a9b0fc-aeb3-403e-94ab-9d09e3fa44fc-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7998877c76-lzgxk\" (UID: \"44a9b0fc-aeb3-403e-94ab-9d09e3fa44fc\") " pod="openshift-monitoring/telemeter-client-7998877c76-lzgxk"
Sep 15 12:04:48.935583 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:04:48.935500    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/44a9b0fc-aeb3-403e-94ab-9d09e3fa44fc-secret-telemeter-client\") pod \"telemeter-client-7998877c76-lzgxk\" (UID: \"44a9b0fc-aeb3-403e-94ab-9d09e3fa44fc\") " pod="openshift-monitoring/telemeter-client-7998877c76-lzgxk"
Sep 15 12:04:48.935583 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:04:48.935557    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/44a9b0fc-aeb3-403e-94ab-9d09e3fa44fc-federate-client-tls\") pod \"telemeter-client-7998877c76-lzgxk\" (UID: \"44a9b0fc-aeb3-403e-94ab-9d09e3fa44fc\") " pod="openshift-monitoring/telemeter-client-7998877c76-lzgxk"
Sep 15 12:04:48.938667 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:04:48.938641    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qm7wb\" (UniqueName: \"kubernetes.io/projected/44a9b0fc-aeb3-403e-94ab-9d09e3fa44fc-kube-api-access-qm7wb\") pod \"telemeter-client-7998877c76-lzgxk\" (UID: \"44a9b0fc-aeb3-403e-94ab-9d09e3fa44fc\") " pod="openshift-monitoring/telemeter-client-7998877c76-lzgxk"
Sep 15 12:04:49.128525 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:04:49.128444    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7998877c76-lzgxk"
Sep 15 12:04:49.402743 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:04:49.402695    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7998877c76-lzgxk"]
Sep 15 12:04:49.405083 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:04:49.405050    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44a9b0fc_aeb3_403e_94ab_9d09e3fa44fc.slice/crio-bb9045bb78a41289e19b5079ea752fba1c8977ba4bca3b12268eb17271e8504c WatchSource:0}: Error finding container bb9045bb78a41289e19b5079ea752fba1c8977ba4bca3b12268eb17271e8504c: Status 404 returned error can't find the container with id bb9045bb78a41289e19b5079ea752fba1c8977ba4bca3b12268eb17271e8504c
Sep 15 12:04:49.535450 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:04:49.535411    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7998877c76-lzgxk" event={"ID":"44a9b0fc-aeb3-403e-94ab-9d09e3fa44fc","Type":"ContainerStarted","Data":"bb9045bb78a41289e19b5079ea752fba1c8977ba4bca3b12268eb17271e8504c"}
Sep 15 12:04:51.543729 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:04:51.543642    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7998877c76-lzgxk" event={"ID":"44a9b0fc-aeb3-403e-94ab-9d09e3fa44fc","Type":"ContainerStarted","Data":"b67c3abd18fc6a4f96de6212879f625d81e77425abb6f8655fdb865f9d0e90b9"}
Sep 15 12:04:52.549607 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:04:52.549511    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7998877c76-lzgxk" event={"ID":"44a9b0fc-aeb3-403e-94ab-9d09e3fa44fc","Type":"ContainerStarted","Data":"c9735742f39baae23c552b5a393084de72e83a7b55e740d5e31ba92ede034568"}
Sep 15 12:04:52.549607 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:04:52.549577    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7998877c76-lzgxk" event={"ID":"44a9b0fc-aeb3-403e-94ab-9d09e3fa44fc","Type":"ContainerStarted","Data":"a6b6822a498eb5dcf49a167ac2143b884aace560794088801f5ce300ed27e4fa"}
Sep 15 12:04:52.573093 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:04:52.573037    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-7998877c76-lzgxk" podStartSLOduration=1.792980459 podStartE2EDuration="4.573019227s" totalImagesPullingTime="2.780038768s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:04:48 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:04:52.572402839 +0000 UTC m=+149.363325362" watchObservedRunningTime="2026-09-15 12:04:52.573019227 +0000 UTC m=+149.363941738"
Sep 15 12:07:05.213385 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:05.213294    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hp4js"]
Sep 15 12:07:05.216566 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:05.216549    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hp4js"
Sep 15 12:07:05.219248 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:05.219231    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-9dvrd\""
Sep 15 12:07:05.219459 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:05.219441    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 15 12:07:05.220253 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:05.220226    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 15 12:07:05.229286 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:05.229247    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hp4js"]
Sep 15 12:07:05.338763 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:05.338722    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp4dh\" (UniqueName: \"kubernetes.io/projected/3e47f0cd-9c14-4d5d-90f5-dfc571bccd79-kube-api-access-rp4dh\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hp4js\" (UID: \"3e47f0cd-9c14-4d5d-90f5-dfc571bccd79\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hp4js"
Sep 15 12:07:05.338931 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:05.338784    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3e47f0cd-9c14-4d5d-90f5-dfc571bccd79-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hp4js\" (UID: \"3e47f0cd-9c14-4d5d-90f5-dfc571bccd79\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hp4js"
Sep 15 12:07:05.338931 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:05.338814    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3e47f0cd-9c14-4d5d-90f5-dfc571bccd79-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hp4js\" (UID: \"3e47f0cd-9c14-4d5d-90f5-dfc571bccd79\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hp4js"
Sep 15 12:07:05.440048 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:05.440014    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3e47f0cd-9c14-4d5d-90f5-dfc571bccd79-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hp4js\" (UID: \"3e47f0cd-9c14-4d5d-90f5-dfc571bccd79\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hp4js"
Sep 15 12:07:05.440219 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:05.440058    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp4dh\" (UniqueName: \"kubernetes.io/projected/3e47f0cd-9c14-4d5d-90f5-dfc571bccd79-kube-api-access-rp4dh\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hp4js\" (UID: \"3e47f0cd-9c14-4d5d-90f5-dfc571bccd79\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hp4js"
Sep 15 12:07:05.440219 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:05.440106    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3e47f0cd-9c14-4d5d-90f5-dfc571bccd79-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hp4js\" (UID: \"3e47f0cd-9c14-4d5d-90f5-dfc571bccd79\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hp4js"
Sep 15 12:07:05.440414 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:05.440396    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3e47f0cd-9c14-4d5d-90f5-dfc571bccd79-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hp4js\" (UID: \"3e47f0cd-9c14-4d5d-90f5-dfc571bccd79\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hp4js"
Sep 15 12:07:05.440478 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:05.440463    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3e47f0cd-9c14-4d5d-90f5-dfc571bccd79-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hp4js\" (UID: \"3e47f0cd-9c14-4d5d-90f5-dfc571bccd79\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hp4js"
Sep 15 12:07:05.449529 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:05.449500    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp4dh\" (UniqueName: \"kubernetes.io/projected/3e47f0cd-9c14-4d5d-90f5-dfc571bccd79-kube-api-access-rp4dh\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hp4js\" (UID: \"3e47f0cd-9c14-4d5d-90f5-dfc571bccd79\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hp4js"
Sep 15 12:07:05.525599 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:05.525510    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hp4js"
Sep 15 12:07:05.796365 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:05.796321    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hp4js"]
Sep 15 12:07:05.799113 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:07:05.799086    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e47f0cd_9c14_4d5d_90f5_dfc571bccd79.slice/crio-cecaca8a6e1d260e4356e5cdac8e7201382fccf6bac3c3a010297dd41c051060 WatchSource:0}: Error finding container cecaca8a6e1d260e4356e5cdac8e7201382fccf6bac3c3a010297dd41c051060: Status 404 returned error can't find the container with id cecaca8a6e1d260e4356e5cdac8e7201382fccf6bac3c3a010297dd41c051060
Sep 15 12:07:06.009092 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:06.009059    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hp4js" event={"ID":"3e47f0cd-9c14-4d5d-90f5-dfc571bccd79","Type":"ContainerStarted","Data":"cecaca8a6e1d260e4356e5cdac8e7201382fccf6bac3c3a010297dd41c051060"}
Sep 15 12:07:17.053844 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:17.053804    2555 generic.go:356] "Generic (PLEG): container finished" podID="3e47f0cd-9c14-4d5d-90f5-dfc571bccd79" containerID="43f92d6ad2eff0ac66baffe2510592a9de420ad0e4c87c7ed02d152d97f4a932" exitCode=0
Sep 15 12:07:17.054313 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:17.053902    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hp4js" event={"ID":"3e47f0cd-9c14-4d5d-90f5-dfc571bccd79","Type":"ContainerDied","Data":"43f92d6ad2eff0ac66baffe2510592a9de420ad0e4c87c7ed02d152d97f4a932"}
Sep 15 12:07:23.076180 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:23.076151    2555 generic.go:356] "Generic (PLEG): container finished" podID="3e47f0cd-9c14-4d5d-90f5-dfc571bccd79" containerID="7b946fd2d2337664713c2939085c51e6269ef9c7388a57cba7082c1f049c614a" exitCode=0
Sep 15 12:07:23.076605 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:23.076229    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hp4js" event={"ID":"3e47f0cd-9c14-4d5d-90f5-dfc571bccd79","Type":"ContainerDied","Data":"7b946fd2d2337664713c2939085c51e6269ef9c7388a57cba7082c1f049c614a"}
Sep 15 12:07:23.739633 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:23.739603    2555 kubelet.go:1648] "Image garbage collection succeeded"
Sep 15 12:07:30.106981 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:30.106946    2555 generic.go:356] "Generic (PLEG): container finished" podID="3e47f0cd-9c14-4d5d-90f5-dfc571bccd79" containerID="a0c7cb2b080ddecccdb1c0bb34818abd3cd48939573615af3306e8a38d526ad5" exitCode=0
Sep 15 12:07:30.107408 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:30.107037    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hp4js" event={"ID":"3e47f0cd-9c14-4d5d-90f5-dfc571bccd79","Type":"ContainerDied","Data":"a0c7cb2b080ddecccdb1c0bb34818abd3cd48939573615af3306e8a38d526ad5"}
Sep 15 12:07:31.231633 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:31.231602    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hp4js"
Sep 15 12:07:31.239289 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:31.239248    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3e47f0cd-9c14-4d5d-90f5-dfc571bccd79-bundle\") pod \"3e47f0cd-9c14-4d5d-90f5-dfc571bccd79\" (UID: \"3e47f0cd-9c14-4d5d-90f5-dfc571bccd79\") "
Sep 15 12:07:31.239389 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:31.239338    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3e47f0cd-9c14-4d5d-90f5-dfc571bccd79-util\") pod \"3e47f0cd-9c14-4d5d-90f5-dfc571bccd79\" (UID: \"3e47f0cd-9c14-4d5d-90f5-dfc571bccd79\") "
Sep 15 12:07:31.239426 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:31.239390    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rp4dh\" (UniqueName: \"kubernetes.io/projected/3e47f0cd-9c14-4d5d-90f5-dfc571bccd79-kube-api-access-rp4dh\") pod \"3e47f0cd-9c14-4d5d-90f5-dfc571bccd79\" (UID: \"3e47f0cd-9c14-4d5d-90f5-dfc571bccd79\") "
Sep 15 12:07:31.239915 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:31.239887    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e47f0cd-9c14-4d5d-90f5-dfc571bccd79-bundle" (OuterVolumeSpecName: "bundle") pod "3e47f0cd-9c14-4d5d-90f5-dfc571bccd79" (UID: "3e47f0cd-9c14-4d5d-90f5-dfc571bccd79"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:07:31.241605 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:31.241577    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e47f0cd-9c14-4d5d-90f5-dfc571bccd79-kube-api-access-rp4dh" (OuterVolumeSpecName: "kube-api-access-rp4dh") pod "3e47f0cd-9c14-4d5d-90f5-dfc571bccd79" (UID: "3e47f0cd-9c14-4d5d-90f5-dfc571bccd79"). InnerVolumeSpecName "kube-api-access-rp4dh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:07:31.243662 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:31.243640    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e47f0cd-9c14-4d5d-90f5-dfc571bccd79-util" (OuterVolumeSpecName: "util") pod "3e47f0cd-9c14-4d5d-90f5-dfc571bccd79" (UID: "3e47f0cd-9c14-4d5d-90f5-dfc571bccd79"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:07:31.340085 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:31.340042    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rp4dh\" (UniqueName: \"kubernetes.io/projected/3e47f0cd-9c14-4d5d-90f5-dfc571bccd79-kube-api-access-rp4dh\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:07:31.340085 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:31.340078    2555 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3e47f0cd-9c14-4d5d-90f5-dfc571bccd79-bundle\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:07:31.340085 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:31.340090    2555 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3e47f0cd-9c14-4d5d-90f5-dfc571bccd79-util\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:07:32.116024 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:32.115955    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hp4js"
Sep 15 12:07:32.116024 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:32.115964    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hp4js" event={"ID":"3e47f0cd-9c14-4d5d-90f5-dfc571bccd79","Type":"ContainerDied","Data":"cecaca8a6e1d260e4356e5cdac8e7201382fccf6bac3c3a010297dd41c051060"}
Sep 15 12:07:32.116024 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:32.116006    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cecaca8a6e1d260e4356e5cdac8e7201382fccf6bac3c3a010297dd41c051060"
Sep 15 12:07:38.481208 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:38.481169    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-v9l8n"]
Sep 15 12:07:38.481837 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:38.481754    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e47f0cd-9c14-4d5d-90f5-dfc571bccd79" containerName="extract"
Sep 15 12:07:38.481837 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:38.481777    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e47f0cd-9c14-4d5d-90f5-dfc571bccd79" containerName="extract"
Sep 15 12:07:38.481837 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:38.481800    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e47f0cd-9c14-4d5d-90f5-dfc571bccd79" containerName="util"
Sep 15 12:07:38.481837 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:38.481809    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e47f0cd-9c14-4d5d-90f5-dfc571bccd79" containerName="util"
Sep 15 12:07:38.481837 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:38.481832    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e47f0cd-9c14-4d5d-90f5-dfc571bccd79" containerName="pull"
Sep 15 12:07:38.481837 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:38.481840    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e47f0cd-9c14-4d5d-90f5-dfc571bccd79" containerName="pull"
Sep 15 12:07:38.482125 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:38.481903    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e47f0cd-9c14-4d5d-90f5-dfc571bccd79" containerName="extract"
Sep 15 12:07:38.487871 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:38.487844    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-v9l8n"
Sep 15 12:07:38.490890 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:38.490861    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Sep 15 12:07:38.491102 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:38.491086    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Sep 15 12:07:38.491242 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:38.491173    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-kln76\""
Sep 15 12:07:38.503043 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:38.503010    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-v9l8n"]
Sep 15 12:07:38.587390 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:38.587352    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6119930e-15f2-4c23-b494-beded7f9e2ca-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-v9l8n\" (UID: \"6119930e-15f2-4c23-b494-beded7f9e2ca\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-v9l8n"
Sep 15 12:07:38.587574 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:38.587434    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2f8sq\" (UniqueName: \"kubernetes.io/projected/6119930e-15f2-4c23-b494-beded7f9e2ca-kube-api-access-2f8sq\") pod \"cert-manager-operator-controller-manager-55c444bf86-v9l8n\" (UID: \"6119930e-15f2-4c23-b494-beded7f9e2ca\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-v9l8n"
Sep 15 12:07:38.688102 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:38.688067    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6119930e-15f2-4c23-b494-beded7f9e2ca-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-v9l8n\" (UID: \"6119930e-15f2-4c23-b494-beded7f9e2ca\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-v9l8n"
Sep 15 12:07:38.688315 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:38.688121    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f8sq\" (UniqueName: \"kubernetes.io/projected/6119930e-15f2-4c23-b494-beded7f9e2ca-kube-api-access-2f8sq\") pod \"cert-manager-operator-controller-manager-55c444bf86-v9l8n\" (UID: \"6119930e-15f2-4c23-b494-beded7f9e2ca\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-v9l8n"
Sep 15 12:07:38.688493 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:38.688473    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6119930e-15f2-4c23-b494-beded7f9e2ca-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-v9l8n\" (UID: \"6119930e-15f2-4c23-b494-beded7f9e2ca\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-v9l8n"
Sep 15 12:07:38.698178 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:38.698152    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2f8sq\" (UniqueName: \"kubernetes.io/projected/6119930e-15f2-4c23-b494-beded7f9e2ca-kube-api-access-2f8sq\") pod \"cert-manager-operator-controller-manager-55c444bf86-v9l8n\" (UID: \"6119930e-15f2-4c23-b494-beded7f9e2ca\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-v9l8n"
Sep 15 12:07:38.797181 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:38.797084    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-v9l8n"
Sep 15 12:07:39.070584 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:39.070562    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-v9l8n"]
Sep 15 12:07:39.073568 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:07:39.073537    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6119930e_15f2_4c23_b494_beded7f9e2ca.slice/crio-dc0bc51a290122fcf4de453cd8594d81247d002c00e8d91982a5d35d194c32ba WatchSource:0}: Error finding container dc0bc51a290122fcf4de453cd8594d81247d002c00e8d91982a5d35d194c32ba: Status 404 returned error can't find the container with id dc0bc51a290122fcf4de453cd8594d81247d002c00e8d91982a5d35d194c32ba
Sep 15 12:07:39.076335 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:39.076317    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 12:07:39.141709 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:39.141679    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-v9l8n" event={"ID":"6119930e-15f2-4c23-b494-beded7f9e2ca","Type":"ContainerStarted","Data":"dc0bc51a290122fcf4de453cd8594d81247d002c00e8d91982a5d35d194c32ba"}
Sep 15 12:07:41.151477 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:41.151437    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-v9l8n" event={"ID":"6119930e-15f2-4c23-b494-beded7f9e2ca","Type":"ContainerStarted","Data":"faf8e2e8b51f64ca34dda1efc07c28912c237c4987671f430a1e7200713d8fbc"}
Sep 15 12:07:41.173460 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:41.173397    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-v9l8n" podStartSLOduration=1.458963918 podStartE2EDuration="3.173377883s" totalImagesPullingTime="1.714413965s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:07:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:07:41.171669036 +0000 UTC m=+317.962591590" watchObservedRunningTime="2026-09-15 12:07:41.173377883 +0000 UTC m=+317.964300393"
Sep 15 12:07:42.348215 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:42.348179    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbmdn4"]
Sep 15 12:07:42.351617 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:42.351599    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbmdn4"
Sep 15 12:07:42.354274 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:42.354238    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 15 12:07:42.354415 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:42.354330    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 15 12:07:42.355347 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:42.355328    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-9dvrd\""
Sep 15 12:07:42.362728 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:42.362699    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbmdn4"]
Sep 15 12:07:42.421870 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:42.421831    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f996483d-cc76-4463-b985-94083ac1de7d-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbmdn4\" (UID: \"f996483d-cc76-4463-b985-94083ac1de7d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbmdn4"
Sep 15 12:07:42.421870 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:42.421867    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54q6c\" (UniqueName: \"kubernetes.io/projected/f996483d-cc76-4463-b985-94083ac1de7d-kube-api-access-54q6c\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbmdn4\" (UID: \"f996483d-cc76-4463-b985-94083ac1de7d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbmdn4"
Sep 15 12:07:42.422088 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:42.421928    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f996483d-cc76-4463-b985-94083ac1de7d-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbmdn4\" (UID: \"f996483d-cc76-4463-b985-94083ac1de7d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbmdn4"
Sep 15 12:07:42.522631 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:42.522599    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f996483d-cc76-4463-b985-94083ac1de7d-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbmdn4\" (UID: \"f996483d-cc76-4463-b985-94083ac1de7d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbmdn4"
Sep 15 12:07:42.522631 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:42.522636    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54q6c\" (UniqueName: \"kubernetes.io/projected/f996483d-cc76-4463-b985-94083ac1de7d-kube-api-access-54q6c\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbmdn4\" (UID: \"f996483d-cc76-4463-b985-94083ac1de7d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbmdn4"
Sep 15 12:07:42.522854 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:42.522694    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f996483d-cc76-4463-b985-94083ac1de7d-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbmdn4\" (UID: \"f996483d-cc76-4463-b985-94083ac1de7d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbmdn4"
Sep 15 12:07:42.523013 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:42.522994    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f996483d-cc76-4463-b985-94083ac1de7d-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbmdn4\" (UID: \"f996483d-cc76-4463-b985-94083ac1de7d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbmdn4"
Sep 15 12:07:42.523111 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:42.523093    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f996483d-cc76-4463-b985-94083ac1de7d-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbmdn4\" (UID: \"f996483d-cc76-4463-b985-94083ac1de7d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbmdn4"
Sep 15 12:07:42.531834 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:42.531797    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54q6c\" (UniqueName: \"kubernetes.io/projected/f996483d-cc76-4463-b985-94083ac1de7d-kube-api-access-54q6c\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbmdn4\" (UID: \"f996483d-cc76-4463-b985-94083ac1de7d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbmdn4"
Sep 15 12:07:42.660149 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:42.660099    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbmdn4"
Sep 15 12:07:42.768034 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:42.767997    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-wsb6s"]
Sep 15 12:07:42.771342 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:42.771324    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-wsb6s"
Sep 15 12:07:42.774133 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:42.774110    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 15 12:07:42.774295 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:42.774120    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 15 12:07:42.774295 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:42.774192    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-74fjv\""
Sep 15 12:07:42.780776 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:42.780751    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-wsb6s"]
Sep 15 12:07:42.826351 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:42.826309    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/81743a44-d17c-4b29-b1da-77efc4d8a79f-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-wsb6s\" (UID: \"81743a44-d17c-4b29-b1da-77efc4d8a79f\") " pod="cert-manager/cert-manager-webhook-6486ffd997-wsb6s"
Sep 15 12:07:42.826532 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:42.826373    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7rds\" (UniqueName: \"kubernetes.io/projected/81743a44-d17c-4b29-b1da-77efc4d8a79f-kube-api-access-j7rds\") pod \"cert-manager-webhook-6486ffd997-wsb6s\" (UID: \"81743a44-d17c-4b29-b1da-77efc4d8a79f\") " pod="cert-manager/cert-manager-webhook-6486ffd997-wsb6s"
Sep 15 12:07:42.927535 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:42.927418    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7rds\" (UniqueName: \"kubernetes.io/projected/81743a44-d17c-4b29-b1da-77efc4d8a79f-kube-api-access-j7rds\") pod \"cert-manager-webhook-6486ffd997-wsb6s\" (UID: \"81743a44-d17c-4b29-b1da-77efc4d8a79f\") " pod="cert-manager/cert-manager-webhook-6486ffd997-wsb6s"
Sep 15 12:07:42.927535 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:42.927508    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/81743a44-d17c-4b29-b1da-77efc4d8a79f-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-wsb6s\" (UID: \"81743a44-d17c-4b29-b1da-77efc4d8a79f\") " pod="cert-manager/cert-manager-webhook-6486ffd997-wsb6s"
Sep 15 12:07:42.929521 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:42.929494    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbmdn4"]
Sep 15 12:07:42.932179 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:07:42.932155    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf996483d_cc76_4463_b985_94083ac1de7d.slice/crio-f7241ec72f33d5f7e5daff1af406dff8d21fb394015e978bb8457472aa73c1c3 WatchSource:0}: Error finding container f7241ec72f33d5f7e5daff1af406dff8d21fb394015e978bb8457472aa73c1c3: Status 404 returned error can't find the container with id f7241ec72f33d5f7e5daff1af406dff8d21fb394015e978bb8457472aa73c1c3
Sep 15 12:07:42.936092 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:42.936070    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7rds\" (UniqueName: \"kubernetes.io/projected/81743a44-d17c-4b29-b1da-77efc4d8a79f-kube-api-access-j7rds\") pod \"cert-manager-webhook-6486ffd997-wsb6s\" (UID: \"81743a44-d17c-4b29-b1da-77efc4d8a79f\") " pod="cert-manager/cert-manager-webhook-6486ffd997-wsb6s"
Sep 15 12:07:42.936380 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:42.936353    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/81743a44-d17c-4b29-b1da-77efc4d8a79f-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-wsb6s\" (UID: \"81743a44-d17c-4b29-b1da-77efc4d8a79f\") " pod="cert-manager/cert-manager-webhook-6486ffd997-wsb6s"
Sep 15 12:07:43.093860 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:43.093824    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-wsb6s"
Sep 15 12:07:43.161071 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:43.161043    2555 generic.go:356] "Generic (PLEG): container finished" podID="f996483d-cc76-4463-b985-94083ac1de7d" containerID="5c98005bd11dcb7483a6e7dc3288acf70b72573d8b3395d85e2982e31ec5bebe" exitCode=0
Sep 15 12:07:43.161224 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:43.161127    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbmdn4" event={"ID":"f996483d-cc76-4463-b985-94083ac1de7d","Type":"ContainerDied","Data":"5c98005bd11dcb7483a6e7dc3288acf70b72573d8b3395d85e2982e31ec5bebe"}
Sep 15 12:07:43.161224 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:43.161189    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbmdn4" event={"ID":"f996483d-cc76-4463-b985-94083ac1de7d","Type":"ContainerStarted","Data":"f7241ec72f33d5f7e5daff1af406dff8d21fb394015e978bb8457472aa73c1c3"}
Sep 15 12:07:43.384072 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:43.384041    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-wsb6s"]
Sep 15 12:07:43.386792 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:07:43.386754    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod81743a44_d17c_4b29_b1da_77efc4d8a79f.slice/crio-253bfac20af6ac7f6eff4701f1db631c15868a6a137ac0c64f26d5c0d89cd2d9 WatchSource:0}: Error finding container 253bfac20af6ac7f6eff4701f1db631c15868a6a137ac0c64f26d5c0d89cd2d9: Status 404 returned error can't find the container with id 253bfac20af6ac7f6eff4701f1db631c15868a6a137ac0c64f26d5c0d89cd2d9
Sep 15 12:07:44.166778 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:44.166740    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-wsb6s" event={"ID":"81743a44-d17c-4b29-b1da-77efc4d8a79f","Type":"ContainerStarted","Data":"253bfac20af6ac7f6eff4701f1db631c15868a6a137ac0c64f26d5c0d89cd2d9"}
Sep 15 12:07:44.424801 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:44.424713    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-2sdfz"]
Sep 15 12:07:44.428087 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:44.428055    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-2sdfz"
Sep 15 12:07:44.430837 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:44.430812    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-std6h\""
Sep 15 12:07:44.438326 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:44.438292    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-2sdfz"]
Sep 15 12:07:44.539854 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:44.539799    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fad8c7f9-5e19-41d3-a6b4-29e9f1dd7fee-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-2sdfz\" (UID: \"fad8c7f9-5e19-41d3-a6b4-29e9f1dd7fee\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2sdfz"
Sep 15 12:07:44.540046 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:44.539926    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fwrt\" (UniqueName: \"kubernetes.io/projected/fad8c7f9-5e19-41d3-a6b4-29e9f1dd7fee-kube-api-access-9fwrt\") pod \"cert-manager-cainjector-559cdd8c87-2sdfz\" (UID: \"fad8c7f9-5e19-41d3-a6b4-29e9f1dd7fee\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2sdfz"
Sep 15 12:07:44.640748 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:44.640708    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fad8c7f9-5e19-41d3-a6b4-29e9f1dd7fee-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-2sdfz\" (UID: \"fad8c7f9-5e19-41d3-a6b4-29e9f1dd7fee\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2sdfz"
Sep 15 12:07:44.640926 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:44.640779    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fwrt\" (UniqueName: \"kubernetes.io/projected/fad8c7f9-5e19-41d3-a6b4-29e9f1dd7fee-kube-api-access-9fwrt\") pod \"cert-manager-cainjector-559cdd8c87-2sdfz\" (UID: \"fad8c7f9-5e19-41d3-a6b4-29e9f1dd7fee\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2sdfz"
Sep 15 12:07:44.649817 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:44.649782    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fad8c7f9-5e19-41d3-a6b4-29e9f1dd7fee-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-2sdfz\" (UID: \"fad8c7f9-5e19-41d3-a6b4-29e9f1dd7fee\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2sdfz"
Sep 15 12:07:44.649955 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:44.649886    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fwrt\" (UniqueName: \"kubernetes.io/projected/fad8c7f9-5e19-41d3-a6b4-29e9f1dd7fee-kube-api-access-9fwrt\") pod \"cert-manager-cainjector-559cdd8c87-2sdfz\" (UID: \"fad8c7f9-5e19-41d3-a6b4-29e9f1dd7fee\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2sdfz"
Sep 15 12:07:44.743651 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:44.743568    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-2sdfz"
Sep 15 12:07:45.494121 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:45.493948    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-2sdfz"]
Sep 15 12:07:45.496620 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:07:45.496587    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfad8c7f9_5e19_41d3_a6b4_29e9f1dd7fee.slice/crio-3f01c45a4bed53df60eed2cd91b480201147e9c2ce6bebb75d81fdb873b2e018 WatchSource:0}: Error finding container 3f01c45a4bed53df60eed2cd91b480201147e9c2ce6bebb75d81fdb873b2e018: Status 404 returned error can't find the container with id 3f01c45a4bed53df60eed2cd91b480201147e9c2ce6bebb75d81fdb873b2e018
Sep 15 12:07:46.176579 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:46.176541    2555 generic.go:356] "Generic (PLEG): container finished" podID="f996483d-cc76-4463-b985-94083ac1de7d" containerID="2ea2d5e13e7313b9fa044d8df0fcf61c62d765e6dccd1a8e766f7aca47d4b23b" exitCode=0
Sep 15 12:07:46.176784 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:46.176632    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbmdn4" event={"ID":"f996483d-cc76-4463-b985-94083ac1de7d","Type":"ContainerDied","Data":"2ea2d5e13e7313b9fa044d8df0fcf61c62d765e6dccd1a8e766f7aca47d4b23b"}
Sep 15 12:07:46.177827 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:46.177802    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-2sdfz" event={"ID":"fad8c7f9-5e19-41d3-a6b4-29e9f1dd7fee","Type":"ContainerStarted","Data":"3f01c45a4bed53df60eed2cd91b480201147e9c2ce6bebb75d81fdb873b2e018"}
Sep 15 12:07:47.185980 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:47.185941    2555 generic.go:356] "Generic (PLEG): container finished" podID="f996483d-cc76-4463-b985-94083ac1de7d" containerID="bcd7267de90dedd5151cf477363c097636b417aa3305617275cb356c73cfb004" exitCode=0
Sep 15 12:07:47.186472 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:47.186016    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbmdn4" event={"ID":"f996483d-cc76-4463-b985-94083ac1de7d","Type":"ContainerDied","Data":"bcd7267de90dedd5151cf477363c097636b417aa3305617275cb356c73cfb004"}
Sep 15 12:07:48.503888 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:48.503860    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbmdn4"
Sep 15 12:07:48.573409 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:48.573380    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-54q6c\" (UniqueName: \"kubernetes.io/projected/f996483d-cc76-4463-b985-94083ac1de7d-kube-api-access-54q6c\") pod \"f996483d-cc76-4463-b985-94083ac1de7d\" (UID: \"f996483d-cc76-4463-b985-94083ac1de7d\") "
Sep 15 12:07:48.573550 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:48.573463    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f996483d-cc76-4463-b985-94083ac1de7d-util\") pod \"f996483d-cc76-4463-b985-94083ac1de7d\" (UID: \"f996483d-cc76-4463-b985-94083ac1de7d\") "
Sep 15 12:07:48.573550 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:48.573505    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f996483d-cc76-4463-b985-94083ac1de7d-bundle\") pod \"f996483d-cc76-4463-b985-94083ac1de7d\" (UID: \"f996483d-cc76-4463-b985-94083ac1de7d\") "
Sep 15 12:07:48.573933 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:48.573905    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f996483d-cc76-4463-b985-94083ac1de7d-bundle" (OuterVolumeSpecName: "bundle") pod "f996483d-cc76-4463-b985-94083ac1de7d" (UID: "f996483d-cc76-4463-b985-94083ac1de7d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:07:48.575542 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:48.575511    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f996483d-cc76-4463-b985-94083ac1de7d-kube-api-access-54q6c" (OuterVolumeSpecName: "kube-api-access-54q6c") pod "f996483d-cc76-4463-b985-94083ac1de7d" (UID: "f996483d-cc76-4463-b985-94083ac1de7d"). InnerVolumeSpecName "kube-api-access-54q6c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:07:48.578326 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:48.578280    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f996483d-cc76-4463-b985-94083ac1de7d-util" (OuterVolumeSpecName: "util") pod "f996483d-cc76-4463-b985-94083ac1de7d" (UID: "f996483d-cc76-4463-b985-94083ac1de7d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:07:48.674559 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:48.674525    2555 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f996483d-cc76-4463-b985-94083ac1de7d-util\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:07:48.674559 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:48.674554    2555 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f996483d-cc76-4463-b985-94083ac1de7d-bundle\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:07:48.674559 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:48.674564    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54q6c\" (UniqueName: \"kubernetes.io/projected/f996483d-cc76-4463-b985-94083ac1de7d-kube-api-access-54q6c\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:07:49.195679 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:49.195641    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-2sdfz" event={"ID":"fad8c7f9-5e19-41d3-a6b4-29e9f1dd7fee","Type":"ContainerStarted","Data":"14be93eebbe53fbef082b68733a806cfee08e42d853cf9445b92e037b1e1d345"}
Sep 15 12:07:49.197456 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:49.197432    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbmdn4"
Sep 15 12:07:49.197576 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:49.197436    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbmdn4" event={"ID":"f996483d-cc76-4463-b985-94083ac1de7d","Type":"ContainerDied","Data":"f7241ec72f33d5f7e5daff1af406dff8d21fb394015e978bb8457472aa73c1c3"}
Sep 15 12:07:49.197576 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:49.197549    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f7241ec72f33d5f7e5daff1af406dff8d21fb394015e978bb8457472aa73c1c3"
Sep 15 12:07:49.198911 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:49.198890    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-wsb6s" event={"ID":"81743a44-d17c-4b29-b1da-77efc4d8a79f","Type":"ContainerStarted","Data":"52a1ddb593a62584ec8338167d3c15abbaa7c0c1f427ec1fb592fc37a1f32fa8"}
Sep 15 12:07:49.199012 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:49.198997    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-wsb6s"
Sep 15 12:07:49.219348 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:49.219302    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-2sdfz" podStartSLOduration=2.145008738 podStartE2EDuration="5.219286894s" totalImagesPullingTime="3.074278156s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:07:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:07:49.217952018 +0000 UTC m=+326.008874541" watchObservedRunningTime="2026-09-15 12:07:49.219286894 +0000 UTC m=+326.010209405"
Sep 15 12:07:49.235906 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:49.235851    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-wsb6s" podStartSLOduration=2.045238023 podStartE2EDuration="7.23583291s" totalImagesPullingTime="5.190594887s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:07:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:07:49.235238993 +0000 UTC m=+326.026161506" watchObservedRunningTime="2026-09-15 12:07:49.23583291 +0000 UTC m=+326.026755421"
Sep 15 12:07:55.205029 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:07:55.204984    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-wsb6s"
Sep 15 12:08:02.731206 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:02.731171    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg9xkp"]
Sep 15 12:08:02.731690 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:02.731501    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f996483d-cc76-4463-b985-94083ac1de7d" containerName="pull"
Sep 15 12:08:02.731690 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:02.731512    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="f996483d-cc76-4463-b985-94083ac1de7d" containerName="pull"
Sep 15 12:08:02.731690 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:02.731522    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f996483d-cc76-4463-b985-94083ac1de7d" containerName="util"
Sep 15 12:08:02.731690 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:02.731528    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="f996483d-cc76-4463-b985-94083ac1de7d" containerName="util"
Sep 15 12:08:02.731690 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:02.731545    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f996483d-cc76-4463-b985-94083ac1de7d" containerName="extract"
Sep 15 12:08:02.731690 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:02.731550    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="f996483d-cc76-4463-b985-94083ac1de7d" containerName="extract"
Sep 15 12:08:02.731690 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:02.731587    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="f996483d-cc76-4463-b985-94083ac1de7d" containerName="extract"
Sep 15 12:08:02.737403 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:02.737374    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg9xkp"
Sep 15 12:08:02.741231 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:02.741207    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 15 12:08:02.741376 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:02.741296    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-9dvrd\""
Sep 15 12:08:02.742387 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:02.742368    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 15 12:08:02.751675 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:02.751650    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg9xkp"]
Sep 15 12:08:02.787745 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:02.787707    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/455e2e0b-9901-408e-985b-dfa8c62acd28-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg9xkp\" (UID: \"455e2e0b-9901-408e-985b-dfa8c62acd28\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg9xkp"
Sep 15 12:08:02.787948 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:02.787772    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/455e2e0b-9901-408e-985b-dfa8c62acd28-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg9xkp\" (UID: \"455e2e0b-9901-408e-985b-dfa8c62acd28\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg9xkp"
Sep 15 12:08:02.787948 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:02.787819    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2d5fp\" (UniqueName: \"kubernetes.io/projected/455e2e0b-9901-408e-985b-dfa8c62acd28-kube-api-access-2d5fp\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg9xkp\" (UID: \"455e2e0b-9901-408e-985b-dfa8c62acd28\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg9xkp"
Sep 15 12:08:02.888957 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:02.888921    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d5fp\" (UniqueName: \"kubernetes.io/projected/455e2e0b-9901-408e-985b-dfa8c62acd28-kube-api-access-2d5fp\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg9xkp\" (UID: \"455e2e0b-9901-408e-985b-dfa8c62acd28\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg9xkp"
Sep 15 12:08:02.889119 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:02.888982    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/455e2e0b-9901-408e-985b-dfa8c62acd28-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg9xkp\" (UID: \"455e2e0b-9901-408e-985b-dfa8c62acd28\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg9xkp"
Sep 15 12:08:02.889119 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:02.889014    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/455e2e0b-9901-408e-985b-dfa8c62acd28-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg9xkp\" (UID: \"455e2e0b-9901-408e-985b-dfa8c62acd28\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg9xkp"
Sep 15 12:08:02.889368 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:02.889349    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/455e2e0b-9901-408e-985b-dfa8c62acd28-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg9xkp\" (UID: \"455e2e0b-9901-408e-985b-dfa8c62acd28\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg9xkp"
Sep 15 12:08:02.889422 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:02.889408    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/455e2e0b-9901-408e-985b-dfa8c62acd28-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg9xkp\" (UID: \"455e2e0b-9901-408e-985b-dfa8c62acd28\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg9xkp"
Sep 15 12:08:02.897682 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:02.897658    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2d5fp\" (UniqueName: \"kubernetes.io/projected/455e2e0b-9901-408e-985b-dfa8c62acd28-kube-api-access-2d5fp\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg9xkp\" (UID: \"455e2e0b-9901-408e-985b-dfa8c62acd28\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg9xkp"
Sep 15 12:08:03.046326 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:03.046222    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg9xkp"
Sep 15 12:08:03.327119 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:03.327091    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg9xkp"]
Sep 15 12:08:03.329883 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:08:03.329855    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod455e2e0b_9901_408e_985b_dfa8c62acd28.slice/crio-1366acbabc28fcae67df30ed62ce7aae9794b5f690088bdbfcfd2b610d6485d9 WatchSource:0}: Error finding container 1366acbabc28fcae67df30ed62ce7aae9794b5f690088bdbfcfd2b610d6485d9: Status 404 returned error can't find the container with id 1366acbabc28fcae67df30ed62ce7aae9794b5f690088bdbfcfd2b610d6485d9
Sep 15 12:08:04.261811 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:04.261715    2555 generic.go:356] "Generic (PLEG): container finished" podID="455e2e0b-9901-408e-985b-dfa8c62acd28" containerID="9ec7868881454b9703cf2957c6e206fa6bddfcf3f732876e4fa1818216c7eff1" exitCode=0
Sep 15 12:08:04.261811 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:04.261779    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg9xkp" event={"ID":"455e2e0b-9901-408e-985b-dfa8c62acd28","Type":"ContainerDied","Data":"9ec7868881454b9703cf2957c6e206fa6bddfcf3f732876e4fa1818216c7eff1"}
Sep 15 12:08:04.262247 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:04.261828    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg9xkp" event={"ID":"455e2e0b-9901-408e-985b-dfa8c62acd28","Type":"ContainerStarted","Data":"1366acbabc28fcae67df30ed62ce7aae9794b5f690088bdbfcfd2b610d6485d9"}
Sep 15 12:08:05.267782 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:05.267690    2555 generic.go:356] "Generic (PLEG): container finished" podID="455e2e0b-9901-408e-985b-dfa8c62acd28" containerID="b598c8ed56f5da7ab112aba166d4c50f2c809bcbad5d1bb02d06750c1cc716c5" exitCode=0
Sep 15 12:08:05.267782 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:05.267744    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg9xkp" event={"ID":"455e2e0b-9901-408e-985b-dfa8c62acd28","Type":"ContainerDied","Data":"b598c8ed56f5da7ab112aba166d4c50f2c809bcbad5d1bb02d06750c1cc716c5"}
Sep 15 12:08:06.274031 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:06.273990    2555 generic.go:356] "Generic (PLEG): container finished" podID="455e2e0b-9901-408e-985b-dfa8c62acd28" containerID="76a324b91b1106ac624b326e4b6b0c17a4d34ecf0259beb5a7b6b3f4e1f41c25" exitCode=0
Sep 15 12:08:06.274439 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:06.274071    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg9xkp" event={"ID":"455e2e0b-9901-408e-985b-dfa8c62acd28","Type":"ContainerDied","Data":"76a324b91b1106ac624b326e4b6b0c17a4d34ecf0259beb5a7b6b3f4e1f41c25"}
Sep 15 12:08:07.396748 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:07.396720    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg9xkp"
Sep 15 12:08:07.527152 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:07.527106    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/455e2e0b-9901-408e-985b-dfa8c62acd28-bundle\") pod \"455e2e0b-9901-408e-985b-dfa8c62acd28\" (UID: \"455e2e0b-9901-408e-985b-dfa8c62acd28\") "
Sep 15 12:08:07.527394 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:07.527192    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2d5fp\" (UniqueName: \"kubernetes.io/projected/455e2e0b-9901-408e-985b-dfa8c62acd28-kube-api-access-2d5fp\") pod \"455e2e0b-9901-408e-985b-dfa8c62acd28\" (UID: \"455e2e0b-9901-408e-985b-dfa8c62acd28\") "
Sep 15 12:08:07.527394 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:07.527214    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/455e2e0b-9901-408e-985b-dfa8c62acd28-util\") pod \"455e2e0b-9901-408e-985b-dfa8c62acd28\" (UID: \"455e2e0b-9901-408e-985b-dfa8c62acd28\") "
Sep 15 12:08:07.527868 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:07.527845    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/455e2e0b-9901-408e-985b-dfa8c62acd28-bundle" (OuterVolumeSpecName: "bundle") pod "455e2e0b-9901-408e-985b-dfa8c62acd28" (UID: "455e2e0b-9901-408e-985b-dfa8c62acd28"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:08:07.529616 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:07.529564    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/455e2e0b-9901-408e-985b-dfa8c62acd28-kube-api-access-2d5fp" (OuterVolumeSpecName: "kube-api-access-2d5fp") pod "455e2e0b-9901-408e-985b-dfa8c62acd28" (UID: "455e2e0b-9901-408e-985b-dfa8c62acd28"). InnerVolumeSpecName "kube-api-access-2d5fp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:08:07.532355 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:07.532331    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/455e2e0b-9901-408e-985b-dfa8c62acd28-util" (OuterVolumeSpecName: "util") pod "455e2e0b-9901-408e-985b-dfa8c62acd28" (UID: "455e2e0b-9901-408e-985b-dfa8c62acd28"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:08:07.628354 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:07.628307    2555 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/455e2e0b-9901-408e-985b-dfa8c62acd28-bundle\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:08:07.628354 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:07.628348    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2d5fp\" (UniqueName: \"kubernetes.io/projected/455e2e0b-9901-408e-985b-dfa8c62acd28-kube-api-access-2d5fp\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:08:07.628354 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:07.628358    2555 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/455e2e0b-9901-408e-985b-dfa8c62acd28-util\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:08:08.285213 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:08.285180    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg9xkp" event={"ID":"455e2e0b-9901-408e-985b-dfa8c62acd28","Type":"ContainerDied","Data":"1366acbabc28fcae67df30ed62ce7aae9794b5f690088bdbfcfd2b610d6485d9"}
Sep 15 12:08:08.285213 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:08.285204    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg9xkp"
Sep 15 12:08:08.285438 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:08.285226    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1366acbabc28fcae67df30ed62ce7aae9794b5f690088bdbfcfd2b610d6485d9"
Sep 15 12:08:14.166923 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:14.166886    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b087zgk"]
Sep 15 12:08:14.167360 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:14.167144    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="455e2e0b-9901-408e-985b-dfa8c62acd28" containerName="util"
Sep 15 12:08:14.167360 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:14.167154    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="455e2e0b-9901-408e-985b-dfa8c62acd28" containerName="util"
Sep 15 12:08:14.167360 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:14.167173    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="455e2e0b-9901-408e-985b-dfa8c62acd28" containerName="extract"
Sep 15 12:08:14.167360 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:14.167178    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="455e2e0b-9901-408e-985b-dfa8c62acd28" containerName="extract"
Sep 15 12:08:14.167360 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:14.167186    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="455e2e0b-9901-408e-985b-dfa8c62acd28" containerName="pull"
Sep 15 12:08:14.167360 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:14.167191    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="455e2e0b-9901-408e-985b-dfa8c62acd28" containerName="pull"
Sep 15 12:08:14.167360 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:14.167235    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="455e2e0b-9901-408e-985b-dfa8c62acd28" containerName="extract"
Sep 15 12:08:14.170515 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:14.170496    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b087zgk"
Sep 15 12:08:14.174564 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:14.174544    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 15 12:08:14.174681 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:14.174582    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-9dvrd\""
Sep 15 12:08:14.175583 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:14.175558    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 15 12:08:14.194585 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:14.194558    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b087zgk"]
Sep 15 12:08:14.283599 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:14.283563    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpjpm\" (UniqueName: \"kubernetes.io/projected/77496261-219a-4117-ad02-cefc40b535af-kube-api-access-xpjpm\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b087zgk\" (UID: \"77496261-219a-4117-ad02-cefc40b535af\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b087zgk"
Sep 15 12:08:14.283781 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:14.283624    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/77496261-219a-4117-ad02-cefc40b535af-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b087zgk\" (UID: \"77496261-219a-4117-ad02-cefc40b535af\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b087zgk"
Sep 15 12:08:14.283781 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:14.283680    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/77496261-219a-4117-ad02-cefc40b535af-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b087zgk\" (UID: \"77496261-219a-4117-ad02-cefc40b535af\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b087zgk"
Sep 15 12:08:14.384716 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:14.384676    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpjpm\" (UniqueName: \"kubernetes.io/projected/77496261-219a-4117-ad02-cefc40b535af-kube-api-access-xpjpm\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b087zgk\" (UID: \"77496261-219a-4117-ad02-cefc40b535af\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b087zgk"
Sep 15 12:08:14.384912 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:14.384729    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/77496261-219a-4117-ad02-cefc40b535af-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b087zgk\" (UID: \"77496261-219a-4117-ad02-cefc40b535af\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b087zgk"
Sep 15 12:08:14.384912 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:14.384752    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/77496261-219a-4117-ad02-cefc40b535af-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b087zgk\" (UID: \"77496261-219a-4117-ad02-cefc40b535af\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b087zgk"
Sep 15 12:08:14.385104 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:14.385081    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/77496261-219a-4117-ad02-cefc40b535af-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b087zgk\" (UID: \"77496261-219a-4117-ad02-cefc40b535af\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b087zgk"
Sep 15 12:08:14.385104 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:14.385096    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/77496261-219a-4117-ad02-cefc40b535af-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b087zgk\" (UID: \"77496261-219a-4117-ad02-cefc40b535af\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b087zgk"
Sep 15 12:08:14.393058 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:14.393033    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpjpm\" (UniqueName: \"kubernetes.io/projected/77496261-219a-4117-ad02-cefc40b535af-kube-api-access-xpjpm\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b087zgk\" (UID: \"77496261-219a-4117-ad02-cefc40b535af\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b087zgk"
Sep 15 12:08:14.479061 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:14.478983    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b087zgk"
Sep 15 12:08:14.772361 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:14.772333    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b087zgk"]
Sep 15 12:08:14.774638 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:08:14.774587    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77496261_219a_4117_ad02_cefc40b535af.slice/crio-fdf83ee76e4cdcc22f2c512df3fb0c48165eedfb311accab45a17d239eab8c2a WatchSource:0}: Error finding container fdf83ee76e4cdcc22f2c512df3fb0c48165eedfb311accab45a17d239eab8c2a: Status 404 returned error can't find the container with id fdf83ee76e4cdcc22f2c512df3fb0c48165eedfb311accab45a17d239eab8c2a
Sep 15 12:08:15.316420 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:15.316385    2555 generic.go:356] "Generic (PLEG): container finished" podID="77496261-219a-4117-ad02-cefc40b535af" containerID="a901f66e7c9b7b1f062b3b199f7c7e462de081cab9dc74b3d689814433512850" exitCode=0
Sep 15 12:08:15.316810 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:15.316476    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b087zgk" event={"ID":"77496261-219a-4117-ad02-cefc40b535af","Type":"ContainerDied","Data":"a901f66e7c9b7b1f062b3b199f7c7e462de081cab9dc74b3d689814433512850"}
Sep 15 12:08:15.316810 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:15.316528    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b087zgk" event={"ID":"77496261-219a-4117-ad02-cefc40b535af","Type":"ContainerStarted","Data":"fdf83ee76e4cdcc22f2c512df3fb0c48165eedfb311accab45a17d239eab8c2a"}
Sep 15 12:08:16.195708 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:16.195669    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7b747b9585-zwmdf"]
Sep 15 12:08:16.198791 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:16.198753    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7b747b9585-zwmdf"
Sep 15 12:08:16.201461 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:16.201433    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 15 12:08:16.201582 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:16.201466    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 15 12:08:16.201582 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:16.201518    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 15 12:08:16.201692 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:16.201599    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-khcjr\""
Sep 15 12:08:16.201692 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:16.201613    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 15 12:08:16.216225 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:16.216150    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7b747b9585-zwmdf"]
Sep 15 12:08:16.299160 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:16.299111    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b04cd056-f3ff-4332-bc4c-bd80688da3d2-tmp\") pod \"opendatahub-operator-controller-manager-7b747b9585-zwmdf\" (UID: \"b04cd056-f3ff-4332-bc4c-bd80688da3d2\") " pod="opendatahub/opendatahub-operator-controller-manager-7b747b9585-zwmdf"
Sep 15 12:08:16.299381 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:16.299170    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b04cd056-f3ff-4332-bc4c-bd80688da3d2-webhook-cert\") pod \"opendatahub-operator-controller-manager-7b747b9585-zwmdf\" (UID: \"b04cd056-f3ff-4332-bc4c-bd80688da3d2\") " pod="opendatahub/opendatahub-operator-controller-manager-7b747b9585-zwmdf"
Sep 15 12:08:16.299381 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:16.299201    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59cdb\" (UniqueName: \"kubernetes.io/projected/b04cd056-f3ff-4332-bc4c-bd80688da3d2-kube-api-access-59cdb\") pod \"opendatahub-operator-controller-manager-7b747b9585-zwmdf\" (UID: \"b04cd056-f3ff-4332-bc4c-bd80688da3d2\") " pod="opendatahub/opendatahub-operator-controller-manager-7b747b9585-zwmdf"
Sep 15 12:08:16.299381 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:16.299288    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b04cd056-f3ff-4332-bc4c-bd80688da3d2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7b747b9585-zwmdf\" (UID: \"b04cd056-f3ff-4332-bc4c-bd80688da3d2\") " pod="opendatahub/opendatahub-operator-controller-manager-7b747b9585-zwmdf"
Sep 15 12:08:16.299381 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:16.299347    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b04cd056-f3ff-4332-bc4c-bd80688da3d2-manifests\") pod \"opendatahub-operator-controller-manager-7b747b9585-zwmdf\" (UID: \"b04cd056-f3ff-4332-bc4c-bd80688da3d2\") " pod="opendatahub/opendatahub-operator-controller-manager-7b747b9585-zwmdf"
Sep 15 12:08:16.322885 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:16.322848    2555 generic.go:356] "Generic (PLEG): container finished" podID="77496261-219a-4117-ad02-cefc40b535af" containerID="b4e2683f66441a2856b81c75db03ca069721f3952927041232025d0224db4fe2" exitCode=0
Sep 15 12:08:16.323332 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:16.322893    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b087zgk" event={"ID":"77496261-219a-4117-ad02-cefc40b535af","Type":"ContainerDied","Data":"b4e2683f66441a2856b81c75db03ca069721f3952927041232025d0224db4fe2"}
Sep 15 12:08:16.399721 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:16.399689    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b04cd056-f3ff-4332-bc4c-bd80688da3d2-tmp\") pod \"opendatahub-operator-controller-manager-7b747b9585-zwmdf\" (UID: \"b04cd056-f3ff-4332-bc4c-bd80688da3d2\") " pod="opendatahub/opendatahub-operator-controller-manager-7b747b9585-zwmdf"
Sep 15 12:08:16.399907 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:16.399726    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b04cd056-f3ff-4332-bc4c-bd80688da3d2-webhook-cert\") pod \"opendatahub-operator-controller-manager-7b747b9585-zwmdf\" (UID: \"b04cd056-f3ff-4332-bc4c-bd80688da3d2\") " pod="opendatahub/opendatahub-operator-controller-manager-7b747b9585-zwmdf"
Sep 15 12:08:16.399907 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:16.399744    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59cdb\" (UniqueName: \"kubernetes.io/projected/b04cd056-f3ff-4332-bc4c-bd80688da3d2-kube-api-access-59cdb\") pod \"opendatahub-operator-controller-manager-7b747b9585-zwmdf\" (UID: \"b04cd056-f3ff-4332-bc4c-bd80688da3d2\") " pod="opendatahub/opendatahub-operator-controller-manager-7b747b9585-zwmdf"
Sep 15 12:08:16.399907 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:16.399861    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b04cd056-f3ff-4332-bc4c-bd80688da3d2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7b747b9585-zwmdf\" (UID: \"b04cd056-f3ff-4332-bc4c-bd80688da3d2\") " pod="opendatahub/opendatahub-operator-controller-manager-7b747b9585-zwmdf"
Sep 15 12:08:16.400084 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:16.399926    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b04cd056-f3ff-4332-bc4c-bd80688da3d2-manifests\") pod \"opendatahub-operator-controller-manager-7b747b9585-zwmdf\" (UID: \"b04cd056-f3ff-4332-bc4c-bd80688da3d2\") " pod="opendatahub/opendatahub-operator-controller-manager-7b747b9585-zwmdf"
Sep 15 12:08:16.400297 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:16.400277    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b04cd056-f3ff-4332-bc4c-bd80688da3d2-manifests\") pod \"opendatahub-operator-controller-manager-7b747b9585-zwmdf\" (UID: \"b04cd056-f3ff-4332-bc4c-bd80688da3d2\") " pod="opendatahub/opendatahub-operator-controller-manager-7b747b9585-zwmdf"
Sep 15 12:08:16.402458 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:16.402434    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b04cd056-f3ff-4332-bc4c-bd80688da3d2-tmp\") pod \"opendatahub-operator-controller-manager-7b747b9585-zwmdf\" (UID: \"b04cd056-f3ff-4332-bc4c-bd80688da3d2\") " pod="opendatahub/opendatahub-operator-controller-manager-7b747b9585-zwmdf"
Sep 15 12:08:16.402566 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:16.402442    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b04cd056-f3ff-4332-bc4c-bd80688da3d2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7b747b9585-zwmdf\" (UID: \"b04cd056-f3ff-4332-bc4c-bd80688da3d2\") " pod="opendatahub/opendatahub-operator-controller-manager-7b747b9585-zwmdf"
Sep 15 12:08:16.402757 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:16.402738    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b04cd056-f3ff-4332-bc4c-bd80688da3d2-webhook-cert\") pod \"opendatahub-operator-controller-manager-7b747b9585-zwmdf\" (UID: \"b04cd056-f3ff-4332-bc4c-bd80688da3d2\") " pod="opendatahub/opendatahub-operator-controller-manager-7b747b9585-zwmdf"
Sep 15 12:08:16.423501 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:16.423476    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59cdb\" (UniqueName: \"kubernetes.io/projected/b04cd056-f3ff-4332-bc4c-bd80688da3d2-kube-api-access-59cdb\") pod \"opendatahub-operator-controller-manager-7b747b9585-zwmdf\" (UID: \"b04cd056-f3ff-4332-bc4c-bd80688da3d2\") " pod="opendatahub/opendatahub-operator-controller-manager-7b747b9585-zwmdf"
Sep 15 12:08:16.509459 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:16.509431    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7b747b9585-zwmdf"
Sep 15 12:08:16.786165 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:16.786140    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7b747b9585-zwmdf"]
Sep 15 12:08:16.788686 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:08:16.788661    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb04cd056_f3ff_4332_bc4c_bd80688da3d2.slice/crio-d6ff96fbd9dbcb1f181bfc7ac2da462105259662b695d618e1d67c3116632b01 WatchSource:0}: Error finding container d6ff96fbd9dbcb1f181bfc7ac2da462105259662b695d618e1d67c3116632b01: Status 404 returned error can't find the container with id d6ff96fbd9dbcb1f181bfc7ac2da462105259662b695d618e1d67c3116632b01
Sep 15 12:08:17.328818 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:17.328783    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7b747b9585-zwmdf" event={"ID":"b04cd056-f3ff-4332-bc4c-bd80688da3d2","Type":"ContainerStarted","Data":"d6ff96fbd9dbcb1f181bfc7ac2da462105259662b695d618e1d67c3116632b01"}
Sep 15 12:08:17.330749 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:17.330731    2555 generic.go:356] "Generic (PLEG): container finished" podID="77496261-219a-4117-ad02-cefc40b535af" containerID="c3e1b345388e20d3d62717fd9dcd61cc9de5d9cf0d14c48705c190bb35e06c42" exitCode=0
Sep 15 12:08:17.330837 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:17.330805    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b087zgk" event={"ID":"77496261-219a-4117-ad02-cefc40b535af","Type":"ContainerDied","Data":"c3e1b345388e20d3d62717fd9dcd61cc9de5d9cf0d14c48705c190bb35e06c42"}
Sep 15 12:08:19.751652 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:19.751625    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b087zgk"
Sep 15 12:08:19.830591 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:19.830568    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/77496261-219a-4117-ad02-cefc40b535af-util\") pod \"77496261-219a-4117-ad02-cefc40b535af\" (UID: \"77496261-219a-4117-ad02-cefc40b535af\") "
Sep 15 12:08:19.830691 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:19.830657    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xpjpm\" (UniqueName: \"kubernetes.io/projected/77496261-219a-4117-ad02-cefc40b535af-kube-api-access-xpjpm\") pod \"77496261-219a-4117-ad02-cefc40b535af\" (UID: \"77496261-219a-4117-ad02-cefc40b535af\") "
Sep 15 12:08:19.830691 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:19.830677    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/77496261-219a-4117-ad02-cefc40b535af-bundle\") pod \"77496261-219a-4117-ad02-cefc40b535af\" (UID: \"77496261-219a-4117-ad02-cefc40b535af\") "
Sep 15 12:08:19.831377 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:19.831351    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77496261-219a-4117-ad02-cefc40b535af-bundle" (OuterVolumeSpecName: "bundle") pod "77496261-219a-4117-ad02-cefc40b535af" (UID: "77496261-219a-4117-ad02-cefc40b535af"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:08:19.832658 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:19.832635    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77496261-219a-4117-ad02-cefc40b535af-kube-api-access-xpjpm" (OuterVolumeSpecName: "kube-api-access-xpjpm") pod "77496261-219a-4117-ad02-cefc40b535af" (UID: "77496261-219a-4117-ad02-cefc40b535af"). InnerVolumeSpecName "kube-api-access-xpjpm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:08:19.836371 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:19.836225    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77496261-219a-4117-ad02-cefc40b535af-util" (OuterVolumeSpecName: "util") pod "77496261-219a-4117-ad02-cefc40b535af" (UID: "77496261-219a-4117-ad02-cefc40b535af"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:08:19.931894 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:19.931861    2555 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/77496261-219a-4117-ad02-cefc40b535af-util\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:08:19.932035 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:19.931907    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpjpm\" (UniqueName: \"kubernetes.io/projected/77496261-219a-4117-ad02-cefc40b535af-kube-api-access-xpjpm\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:08:19.932035 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:19.931916    2555 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/77496261-219a-4117-ad02-cefc40b535af-bundle\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:08:20.348802 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:20.348772    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b087zgk"
Sep 15 12:08:20.348998 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:20.348809    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b087zgk" event={"ID":"77496261-219a-4117-ad02-cefc40b535af","Type":"ContainerDied","Data":"fdf83ee76e4cdcc22f2c512df3fb0c48165eedfb311accab45a17d239eab8c2a"}
Sep 15 12:08:20.348998 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:20.348859    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fdf83ee76e4cdcc22f2c512df3fb0c48165eedfb311accab45a17d239eab8c2a"
Sep 15 12:08:20.350584 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:20.350554    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7b747b9585-zwmdf" event={"ID":"b04cd056-f3ff-4332-bc4c-bd80688da3d2","Type":"ContainerStarted","Data":"433de9acc401e694568ffacaa03aa6876fb0c2982aa96c050538aa83378a66f6"}
Sep 15 12:08:21.356828 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:21.356797    2555 generic.go:356] "Generic (PLEG): container finished" podID="b04cd056-f3ff-4332-bc4c-bd80688da3d2" containerID="433de9acc401e694568ffacaa03aa6876fb0c2982aa96c050538aa83378a66f6" exitCode=0
Sep 15 12:08:21.357231 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:21.356892    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7b747b9585-zwmdf" event={"ID":"b04cd056-f3ff-4332-bc4c-bd80688da3d2","Type":"ContainerDied","Data":"433de9acc401e694568ffacaa03aa6876fb0c2982aa96c050538aa83378a66f6"}
Sep 15 12:08:24.371492 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:24.371454    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7b747b9585-zwmdf" event={"ID":"b04cd056-f3ff-4332-bc4c-bd80688da3d2","Type":"ContainerStarted","Data":"ff10016e0c9465e1bf84fddb60e376b49e2fbd9cd28d9cb8a0c0b07ed17afd12"}
Sep 15 12:08:24.371916 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:24.371540    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7b747b9585-zwmdf"
Sep 15 12:08:24.400384 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:24.400323    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7b747b9585-zwmdf" podStartSLOduration=2.004955935 podStartE2EDuration="8.40030268s" totalImagesPullingTime="5.388511609s" totalInitContainerRuntime="1.006835136s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:08:16 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:08:24.398830255 +0000 UTC m=+361.189752787" watchObservedRunningTime="2026-09-15 12:08:24.40030268 +0000 UTC m=+361.191225192"
Sep 15 12:08:35.379479 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:35.379448    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7b747b9585-zwmdf"
Sep 15 12:08:35.980989 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:35.980956    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-86c794cf5f-8k59f"]
Sep 15 12:08:35.981244 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:35.981233    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77496261-219a-4117-ad02-cefc40b535af" containerName="extract"
Sep 15 12:08:35.981320 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:35.981246    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="77496261-219a-4117-ad02-cefc40b535af" containerName="extract"
Sep 15 12:08:35.981320 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:35.981256    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77496261-219a-4117-ad02-cefc40b535af" containerName="pull"
Sep 15 12:08:35.981320 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:35.981276    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="77496261-219a-4117-ad02-cefc40b535af" containerName="pull"
Sep 15 12:08:35.981320 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:35.981292    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77496261-219a-4117-ad02-cefc40b535af" containerName="util"
Sep 15 12:08:35.981320 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:35.981309    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="77496261-219a-4117-ad02-cefc40b535af" containerName="util"
Sep 15 12:08:35.981487 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:35.981359    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="77496261-219a-4117-ad02-cefc40b535af" containerName="extract"
Sep 15 12:08:35.994967 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:35.994943    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-86c794cf5f-8k59f"]
Sep 15 12:08:35.995118 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:35.995040    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-86c794cf5f-8k59f"
Sep 15 12:08:35.999170 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:35.999138    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 15 12:08:35.999651 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:35.999630    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 15 12:08:36.000017 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:36.000001    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 15 12:08:36.000394 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:36.000376    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-4snpp\""
Sep 15 12:08:36.000901 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:36.000718    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 15 12:08:36.000901 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:36.000733    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 15 12:08:36.168716 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:36.168677    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/baad4933-bf2c-4c6e-a373-40ae40aa415c-cert\") pod \"lws-controller-manager-86c794cf5f-8k59f\" (UID: \"baad4933-bf2c-4c6e-a373-40ae40aa415c\") " pod="openshift-lws-operator/lws-controller-manager-86c794cf5f-8k59f"
Sep 15 12:08:36.168905 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:36.168733    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/baad4933-bf2c-4c6e-a373-40ae40aa415c-metrics-cert\") pod \"lws-controller-manager-86c794cf5f-8k59f\" (UID: \"baad4933-bf2c-4c6e-a373-40ae40aa415c\") " pod="openshift-lws-operator/lws-controller-manager-86c794cf5f-8k59f"
Sep 15 12:08:36.168905 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:36.168793    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r592m\" (UniqueName: \"kubernetes.io/projected/baad4933-bf2c-4c6e-a373-40ae40aa415c-kube-api-access-r592m\") pod \"lws-controller-manager-86c794cf5f-8k59f\" (UID: \"baad4933-bf2c-4c6e-a373-40ae40aa415c\") " pod="openshift-lws-operator/lws-controller-manager-86c794cf5f-8k59f"
Sep 15 12:08:36.168905 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:36.168842    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/baad4933-bf2c-4c6e-a373-40ae40aa415c-manager-config\") pod \"lws-controller-manager-86c794cf5f-8k59f\" (UID: \"baad4933-bf2c-4c6e-a373-40ae40aa415c\") " pod="openshift-lws-operator/lws-controller-manager-86c794cf5f-8k59f"
Sep 15 12:08:36.270219 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:36.270130    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/baad4933-bf2c-4c6e-a373-40ae40aa415c-manager-config\") pod \"lws-controller-manager-86c794cf5f-8k59f\" (UID: \"baad4933-bf2c-4c6e-a373-40ae40aa415c\") " pod="openshift-lws-operator/lws-controller-manager-86c794cf5f-8k59f"
Sep 15 12:08:36.270219 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:36.270203    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/baad4933-bf2c-4c6e-a373-40ae40aa415c-cert\") pod \"lws-controller-manager-86c794cf5f-8k59f\" (UID: \"baad4933-bf2c-4c6e-a373-40ae40aa415c\") " pod="openshift-lws-operator/lws-controller-manager-86c794cf5f-8k59f"
Sep 15 12:08:36.270373 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:36.270234    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/baad4933-bf2c-4c6e-a373-40ae40aa415c-metrics-cert\") pod \"lws-controller-manager-86c794cf5f-8k59f\" (UID: \"baad4933-bf2c-4c6e-a373-40ae40aa415c\") " pod="openshift-lws-operator/lws-controller-manager-86c794cf5f-8k59f"
Sep 15 12:08:36.270373 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:36.270255    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r592m\" (UniqueName: \"kubernetes.io/projected/baad4933-bf2c-4c6e-a373-40ae40aa415c-kube-api-access-r592m\") pod \"lws-controller-manager-86c794cf5f-8k59f\" (UID: \"baad4933-bf2c-4c6e-a373-40ae40aa415c\") " pod="openshift-lws-operator/lws-controller-manager-86c794cf5f-8k59f"
Sep 15 12:08:36.270943 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:36.270913    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/baad4933-bf2c-4c6e-a373-40ae40aa415c-manager-config\") pod \"lws-controller-manager-86c794cf5f-8k59f\" (UID: \"baad4933-bf2c-4c6e-a373-40ae40aa415c\") " pod="openshift-lws-operator/lws-controller-manager-86c794cf5f-8k59f"
Sep 15 12:08:36.272704 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:36.272681    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/baad4933-bf2c-4c6e-a373-40ae40aa415c-cert\") pod \"lws-controller-manager-86c794cf5f-8k59f\" (UID: \"baad4933-bf2c-4c6e-a373-40ae40aa415c\") " pod="openshift-lws-operator/lws-controller-manager-86c794cf5f-8k59f"
Sep 15 12:08:36.272852 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:36.272829    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/baad4933-bf2c-4c6e-a373-40ae40aa415c-metrics-cert\") pod \"lws-controller-manager-86c794cf5f-8k59f\" (UID: \"baad4933-bf2c-4c6e-a373-40ae40aa415c\") " pod="openshift-lws-operator/lws-controller-manager-86c794cf5f-8k59f"
Sep 15 12:08:36.281486 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:36.281441    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r592m\" (UniqueName: \"kubernetes.io/projected/baad4933-bf2c-4c6e-a373-40ae40aa415c-kube-api-access-r592m\") pod \"lws-controller-manager-86c794cf5f-8k59f\" (UID: \"baad4933-bf2c-4c6e-a373-40ae40aa415c\") " pod="openshift-lws-operator/lws-controller-manager-86c794cf5f-8k59f"
Sep 15 12:08:36.308377 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:36.308346    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-86c794cf5f-8k59f"
Sep 15 12:08:36.588496 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:36.588469    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-86c794cf5f-8k59f"]
Sep 15 12:08:36.591041 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:08:36.591015    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaad4933_bf2c_4c6e_a373_40ae40aa415c.slice/crio-ce8c9ad7ebcccd667a4b14ae26375355fd9931ac8390d041cc1a79bdf3cd58f9 WatchSource:0}: Error finding container ce8c9ad7ebcccd667a4b14ae26375355fd9931ac8390d041cc1a79bdf3cd58f9: Status 404 returned error can't find the container with id ce8c9ad7ebcccd667a4b14ae26375355fd9931ac8390d041cc1a79bdf3cd58f9
Sep 15 12:08:37.431240 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:37.431195    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-86c794cf5f-8k59f" event={"ID":"baad4933-bf2c-4c6e-a373-40ae40aa415c","Type":"ContainerStarted","Data":"ce8c9ad7ebcccd667a4b14ae26375355fd9931ac8390d041cc1a79bdf3cd58f9"}
Sep 15 12:08:37.442108 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:37.442080    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7cf78458b7-j5nq4"]
Sep 15 12:08:37.445786 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:37.445751    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7cf78458b7-j5nq4"
Sep 15 12:08:37.449537 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:37.449510    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 15 12:08:37.449692 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:37.449655    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 15 12:08:37.450844 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:37.450763    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-wzldz\""
Sep 15 12:08:37.450844 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:37.450775    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 15 12:08:37.450844 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:37.450841    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 15 12:08:37.470131 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:37.470104    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7cf78458b7-j5nq4"]
Sep 15 12:08:37.579745 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:37.579708    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48f1beca-892d-4af7-a297-f5c524d5344f-tls-certs\") pod \"kube-auth-proxy-7cf78458b7-j5nq4\" (UID: \"48f1beca-892d-4af7-a297-f5c524d5344f\") " pod="openshift-ingress/kube-auth-proxy-7cf78458b7-j5nq4"
Sep 15 12:08:37.579939 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:37.579767    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/48f1beca-892d-4af7-a297-f5c524d5344f-tmp\") pod \"kube-auth-proxy-7cf78458b7-j5nq4\" (UID: \"48f1beca-892d-4af7-a297-f5c524d5344f\") " pod="openshift-ingress/kube-auth-proxy-7cf78458b7-j5nq4"
Sep 15 12:08:37.579939 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:37.579844    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxhkk\" (UniqueName: \"kubernetes.io/projected/48f1beca-892d-4af7-a297-f5c524d5344f-kube-api-access-sxhkk\") pod \"kube-auth-proxy-7cf78458b7-j5nq4\" (UID: \"48f1beca-892d-4af7-a297-f5c524d5344f\") " pod="openshift-ingress/kube-auth-proxy-7cf78458b7-j5nq4"
Sep 15 12:08:37.680685 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:37.680646    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48f1beca-892d-4af7-a297-f5c524d5344f-tls-certs\") pod \"kube-auth-proxy-7cf78458b7-j5nq4\" (UID: \"48f1beca-892d-4af7-a297-f5c524d5344f\") " pod="openshift-ingress/kube-auth-proxy-7cf78458b7-j5nq4"
Sep 15 12:08:37.681156 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:37.680694    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/48f1beca-892d-4af7-a297-f5c524d5344f-tmp\") pod \"kube-auth-proxy-7cf78458b7-j5nq4\" (UID: \"48f1beca-892d-4af7-a297-f5c524d5344f\") " pod="openshift-ingress/kube-auth-proxy-7cf78458b7-j5nq4"
Sep 15 12:08:37.681156 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:37.680717    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxhkk\" (UniqueName: \"kubernetes.io/projected/48f1beca-892d-4af7-a297-f5c524d5344f-kube-api-access-sxhkk\") pod \"kube-auth-proxy-7cf78458b7-j5nq4\" (UID: \"48f1beca-892d-4af7-a297-f5c524d5344f\") " pod="openshift-ingress/kube-auth-proxy-7cf78458b7-j5nq4"
Sep 15 12:08:37.683437 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:37.683366    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/48f1beca-892d-4af7-a297-f5c524d5344f-tmp\") pod \"kube-auth-proxy-7cf78458b7-j5nq4\" (UID: \"48f1beca-892d-4af7-a297-f5c524d5344f\") " pod="openshift-ingress/kube-auth-proxy-7cf78458b7-j5nq4"
Sep 15 12:08:37.683605 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:37.683582    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48f1beca-892d-4af7-a297-f5c524d5344f-tls-certs\") pod \"kube-auth-proxy-7cf78458b7-j5nq4\" (UID: \"48f1beca-892d-4af7-a297-f5c524d5344f\") " pod="openshift-ingress/kube-auth-proxy-7cf78458b7-j5nq4"
Sep 15 12:08:37.695828 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:37.695798    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxhkk\" (UniqueName: \"kubernetes.io/projected/48f1beca-892d-4af7-a297-f5c524d5344f-kube-api-access-sxhkk\") pod \"kube-auth-proxy-7cf78458b7-j5nq4\" (UID: \"48f1beca-892d-4af7-a297-f5c524d5344f\") " pod="openshift-ingress/kube-auth-proxy-7cf78458b7-j5nq4"
Sep 15 12:08:37.756964 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:37.756930    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7cf78458b7-j5nq4"
Sep 15 12:08:38.390718 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:08:38.390686    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48f1beca_892d_4af7_a297_f5c524d5344f.slice/crio-84452a15325efcec383bcf55cbcfe3794d4db9d6f51679d4d9dbcba1eaa0af0d WatchSource:0}: Error finding container 84452a15325efcec383bcf55cbcfe3794d4db9d6f51679d4d9dbcba1eaa0af0d: Status 404 returned error can't find the container with id 84452a15325efcec383bcf55cbcfe3794d4db9d6f51679d4d9dbcba1eaa0af0d
Sep 15 12:08:38.392207 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:38.392176    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7cf78458b7-j5nq4"]
Sep 15 12:08:38.436905 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:38.436865    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7cf78458b7-j5nq4" event={"ID":"48f1beca-892d-4af7-a297-f5c524d5344f","Type":"ContainerStarted","Data":"84452a15325efcec383bcf55cbcfe3794d4db9d6f51679d4d9dbcba1eaa0af0d"}
Sep 15 12:08:38.438472 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:38.438444    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-86c794cf5f-8k59f" event={"ID":"baad4933-bf2c-4c6e-a373-40ae40aa415c","Type":"ContainerStarted","Data":"d272fdef82a6f940b6959f57a5c946a1e8f766255f927033dcf8e91545ae105a"}
Sep 15 12:08:38.438663 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:38.438645    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-86c794cf5f-8k59f"
Sep 15 12:08:38.477151 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:38.477088    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-86c794cf5f-8k59f" podStartSLOduration=1.9126422870000002 podStartE2EDuration="3.477072032s" totalImagesPullingTime="1.564429745s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:08:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:08:38.475601992 +0000 UTC m=+375.266524521" watchObservedRunningTime="2026-09-15 12:08:38.477072032 +0000 UTC m=+375.267994543"
Sep 15 12:08:41.456326 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:41.456227    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7cf78458b7-j5nq4" event={"ID":"48f1beca-892d-4af7-a297-f5c524d5344f","Type":"ContainerStarted","Data":"27a2e9b8571c25987490279a49dc144b4f178d9d93b465b329547f678fd07552"}
Sep 15 12:08:41.476927 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:41.476872    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7cf78458b7-j5nq4" podStartSLOduration=1.706425509 podStartE2EDuration="4.476855451s" totalImagesPullingTime="2.770429942s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:08:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:08:41.474709014 +0000 UTC m=+378.265631526" watchObservedRunningTime="2026-09-15 12:08:41.476855451 +0000 UTC m=+378.267778107"
Sep 15 12:08:49.447765 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:49.447728    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-86c794cf5f-8k59f"
Sep 15 12:08:59.072608 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:59.072565    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-5x5dz"]
Sep 15 12:08:59.075786 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:59.075766    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5x5dz"
Sep 15 12:08:59.080233 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:59.080206    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Sep 15 12:08:59.080354 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:59.080330    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-8sphb\""
Sep 15 12:08:59.117817 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:59.117782    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-5x5dz"]
Sep 15 12:08:59.170338 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:59.170298    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fc231159-17e6-4216-93c4-ffb080ec55d8-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-5x5dz\" (UID: \"fc231159-17e6-4216-93c4-ffb080ec55d8\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5x5dz"
Sep 15 12:08:59.170545 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:59.170377    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvzjl\" (UniqueName: \"kubernetes.io/projected/fc231159-17e6-4216-93c4-ffb080ec55d8-kube-api-access-pvzjl\") pod \"llmisvc-controller-manager-777c6bd7d5-5x5dz\" (UID: \"fc231159-17e6-4216-93c4-ffb080ec55d8\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5x5dz"
Sep 15 12:08:59.271489 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:59.271446    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fc231159-17e6-4216-93c4-ffb080ec55d8-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-5x5dz\" (UID: \"fc231159-17e6-4216-93c4-ffb080ec55d8\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5x5dz"
Sep 15 12:08:59.271632 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:59.271540    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvzjl\" (UniqueName: \"kubernetes.io/projected/fc231159-17e6-4216-93c4-ffb080ec55d8-kube-api-access-pvzjl\") pod \"llmisvc-controller-manager-777c6bd7d5-5x5dz\" (UID: \"fc231159-17e6-4216-93c4-ffb080ec55d8\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5x5dz"
Sep 15 12:08:59.274029 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:59.274003    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fc231159-17e6-4216-93c4-ffb080ec55d8-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-5x5dz\" (UID: \"fc231159-17e6-4216-93c4-ffb080ec55d8\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5x5dz"
Sep 15 12:08:59.279900 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:59.279879    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvzjl\" (UniqueName: \"kubernetes.io/projected/fc231159-17e6-4216-93c4-ffb080ec55d8-kube-api-access-pvzjl\") pod \"llmisvc-controller-manager-777c6bd7d5-5x5dz\" (UID: \"fc231159-17e6-4216-93c4-ffb080ec55d8\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5x5dz"
Sep 15 12:08:59.385879 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:59.385848    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5x5dz"
Sep 15 12:08:59.657850 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:08:59.657826    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-5x5dz"]
Sep 15 12:08:59.660459 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:08:59.660426    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podfc231159_17e6_4216_93c4_ffb080ec55d8.slice/crio-d13ab7b9f77b53b1567e129a972204c076adaebf66d8274829d7cf2467afe01a WatchSource:0}: Error finding container d13ab7b9f77b53b1567e129a972204c076adaebf66d8274829d7cf2467afe01a: Status 404 returned error can't find the container with id d13ab7b9f77b53b1567e129a972204c076adaebf66d8274829d7cf2467afe01a
Sep 15 12:09:00.547469 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:09:00.547424    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5x5dz" event={"ID":"fc231159-17e6-4216-93c4-ffb080ec55d8","Type":"ContainerStarted","Data":"d13ab7b9f77b53b1567e129a972204c076adaebf66d8274829d7cf2467afe01a"}
Sep 15 12:09:03.564323 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:09:03.564284    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5x5dz" event={"ID":"fc231159-17e6-4216-93c4-ffb080ec55d8","Type":"ContainerStarted","Data":"fcf7106c0c0bb8de07b52dc9a747f992e161187e671c66363a50a20deaacb3a3"}
Sep 15 12:09:03.564753 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:09:03.564354    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5x5dz"
Sep 15 12:09:03.581310 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:09:03.581239    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5x5dz" podStartSLOduration=1.4713560079999999 podStartE2EDuration="4.581223544s" totalImagesPullingTime="3.109867536s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:08:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:09:03.579956275 +0000 UTC m=+400.370878780" watchObservedRunningTime="2026-09-15 12:09:03.581223544 +0000 UTC m=+400.372146054"
Sep 15 12:09:34.570342 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:09:34.570303    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5x5dz"
Sep 15 12:09:55.692784 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:09:55.692701    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xcpgl"]
Sep 15 12:09:55.696153 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:09:55.696135    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xcpgl"
Sep 15 12:09:55.699322 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:09:55.699302    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 15 12:09:55.700439 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:09:55.700419    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 15 12:09:55.700506 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:09:55.700461    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-9dvrd\""
Sep 15 12:09:55.708287 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:09:55.708241    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xcpgl"]
Sep 15 12:09:55.738689 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:09:55.738653    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e888b4d0-d4ad-4251-a456-223a36581af4-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xcpgl\" (UID: \"e888b4d0-d4ad-4251-a456-223a36581af4\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xcpgl"
Sep 15 12:09:55.738867 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:09:55.738696    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfmwd\" (UniqueName: \"kubernetes.io/projected/e888b4d0-d4ad-4251-a456-223a36581af4-kube-api-access-vfmwd\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xcpgl\" (UID: \"e888b4d0-d4ad-4251-a456-223a36581af4\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xcpgl"
Sep 15 12:09:55.738867 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:09:55.738754    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e888b4d0-d4ad-4251-a456-223a36581af4-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xcpgl\" (UID: \"e888b4d0-d4ad-4251-a456-223a36581af4\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xcpgl"
Sep 15 12:09:55.820892 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:09:55.820860    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccv29f"]
Sep 15 12:09:55.824278 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:09:55.824250    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccv29f"
Sep 15 12:09:55.835685 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:09:55.835653    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccv29f"]
Sep 15 12:09:55.839953 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:09:55.839927    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e888b4d0-d4ad-4251-a456-223a36581af4-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xcpgl\" (UID: \"e888b4d0-d4ad-4251-a456-223a36581af4\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xcpgl"
Sep 15 12:09:55.840096 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:09:55.839971    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfmwd\" (UniqueName: \"kubernetes.io/projected/e888b4d0-d4ad-4251-a456-223a36581af4-kube-api-access-vfmwd\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xcpgl\" (UID: \"e888b4d0-d4ad-4251-a456-223a36581af4\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xcpgl"
Sep 15 12:09:55.840096 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:09:55.840012    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e888b4d0-d4ad-4251-a456-223a36581af4-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xcpgl\" (UID: \"e888b4d0-d4ad-4251-a456-223a36581af4\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xcpgl"
Sep 15 12:09:55.840388 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:09:55.840364    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e888b4d0-d4ad-4251-a456-223a36581af4-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xcpgl\" (UID: \"e888b4d0-d4ad-4251-a456-223a36581af4\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xcpgl"
Sep 15 12:09:55.840388 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:09:55.840381    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e888b4d0-d4ad-4251-a456-223a36581af4-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xcpgl\" (UID: \"e888b4d0-d4ad-4251-a456-223a36581af4\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xcpgl"
Sep 15 12:09:55.854249 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:09:55.854219    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfmwd\" (UniqueName: \"kubernetes.io/projected/e888b4d0-d4ad-4251-a456-223a36581af4-kube-api-access-vfmwd\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xcpgl\" (UID: \"e888b4d0-d4ad-4251-a456-223a36581af4\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xcpgl"
Sep 15 12:09:55.898932 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:09:55.898899    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7g9km9"]
Sep 15 12:09:55.902488 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:09:55.902466    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7g9km9"
Sep 15 12:09:55.914309 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:09:55.914279    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7g9km9"]
Sep 15 12:09:55.940697 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:09:55.940657    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d902b49c-0cfa-41b0-94a7-7c667e90303e-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccv29f\" (UID: \"d902b49c-0cfa-41b0-94a7-7c667e90303e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccv29f"
Sep 15 12:09:55.940864 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:09:55.940707    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d902b49c-0cfa-41b0-94a7-7c667e90303e-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccv29f\" (UID: \"d902b49c-0cfa-41b0-94a7-7c667e90303e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccv29f"
Sep 15 12:09:55.940864 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:09:55.940739    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xfwg\" (UniqueName: \"kubernetes.io/projected/d902b49c-0cfa-41b0-94a7-7c667e90303e-kube-api-access-8xfwg\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccv29f\" (UID: \"d902b49c-0cfa-41b0-94a7-7c667e90303e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccv29f"
Sep 15 12:09:55.984347 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:09:55.984250    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atj89n"]
Sep 15 12:09:55.987784 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:09:55.987766    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atj89n"
Sep 15 12:09:55.998860 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:09:55.998830    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atj89n"]
Sep 15 12:09:56.005026 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:09:56.005000    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xcpgl"
Sep 15 12:09:56.042400 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:09:56.042004    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6bdc\" (UniqueName: \"kubernetes.io/projected/4cfe2b8b-0cb2-4ec9-90d7-db82647c7798-kube-api-access-t6bdc\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atj89n\" (UID: \"4cfe2b8b-0cb2-4ec9-90d7-db82647c7798\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atj89n"
Sep 15 12:09:56.042400 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:09:56.042070    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4cfe2b8b-0cb2-4ec9-90d7-db82647c7798-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atj89n\" (UID: \"4cfe2b8b-0cb2-4ec9-90d7-db82647c7798\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atj89n"
Sep 15 12:09:56.042400 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:09:56.042111    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d902b49c-0cfa-41b0-94a7-7c667e90303e-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccv29f\" (UID: \"d902b49c-0cfa-41b0-94a7-7c667e90303e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccv29f"
Sep 15 12:09:56.042400 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:09:56.042140    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnw4g\" (UniqueName: \"kubernetes.io/projected/97b591c2-3999-44b9-a081-1374cb45a663-kube-api-access-pnw4g\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7g9km9\" (UID: \"97b591c2-3999-44b9-a081-1374cb45a663\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7g9km9"
Sep 15 12:09:56.042400 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:09:56.042185    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d902b49c-0cfa-41b0-94a7-7c667e90303e-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccv29f\" (UID: \"d902b49c-0cfa-41b0-94a7-7c667e90303e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccv29f"
Sep 15 12:09:56.042400 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:09:56.042210    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4cfe2b8b-0cb2-4ec9-90d7-db82647c7798-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atj89n\" (UID: \"4cfe2b8b-0cb2-4ec9-90d7-db82647c7798\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atj89n"
Sep 15 12:09:56.042400 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:09:56.042241    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xfwg\" (UniqueName: \"kubernetes.io/projected/d902b49c-0cfa-41b0-94a7-7c667e90303e-kube-api-access-8xfwg\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccv29f\" (UID: \"d902b49c-0cfa-41b0-94a7-7c667e90303e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccv29f"
Sep 15 12:09:56.042400 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:09:56.042287    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/97b591c2-3999-44b9-a081-1374cb45a663-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7g9km9\" (UID: \"97b591c2-3999-44b9-a081-1374cb45a663\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7g9km9"
Sep 15 12:09:56.042400 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:09:56.042324    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/97b591c2-3999-44b9-a081-1374cb45a663-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7g9km9\" (UID: \"97b591c2-3999-44b9-a081-1374cb45a663\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7g9km9"
Sep 15 12:09:56.042988 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:09:56.042789    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d902b49c-0cfa-41b0-94a7-7c667e90303e-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccv29f\" (UID: \"d902b49c-0cfa-41b0-94a7-7c667e90303e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccv29f"
Sep 15 12:09:56.042988 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:09:56.042829    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d902b49c-0cfa-41b0-94a7-7c667e90303e-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccv29f\" (UID: \"d902b49c-0cfa-41b0-94a7-7c667e90303e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccv29f"
Sep 15 12:09:56.052943 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:09:56.052144    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xfwg\" (UniqueName: \"kubernetes.io/projected/d902b49c-0cfa-41b0-94a7-7c667e90303e-kube-api-access-8xfwg\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccv29f\" (UID: \"d902b49c-0cfa-41b0-94a7-7c667e90303e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccv29f"
Sep 15 12:09:56.133060 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:09:56.133022    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccv29f"
Sep 15 12:09:56.143595 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:09:56.143572    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4cfe2b8b-0cb2-4ec9-90d7-db82647c7798-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atj89n\" (UID: \"4cfe2b8b-0cb2-4ec9-90d7-db82647c7798\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atj89n"
Sep 15 12:09:56.143677 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:09:56.143612    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnw4g\" (UniqueName: \"kubernetes.io/projected/97b591c2-3999-44b9-a081-1374cb45a663-kube-api-access-pnw4g\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7g9km9\" (UID: \"97b591c2-3999-44b9-a081-1374cb45a663\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7g9km9"
Sep 15 12:09:56.143677 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:09:56.143645    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4cfe2b8b-0cb2-4ec9-90d7-db82647c7798-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atj89n\" (UID: \"4cfe2b8b-0cb2-4ec9-90d7-db82647c7798\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atj89n"
Sep 15 12:09:56.143761 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:09:56.143673    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/97b591c2-3999-44b9-a081-1374cb45a663-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7g9km9\" (UID: \"97b591c2-3999-44b9-a081-1374cb45a663\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7g9km9"
Sep 15 12:09:56.143761 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:09:56.143709    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/97b591c2-3999-44b9-a081-1374cb45a663-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7g9km9\" (UID: \"97b591c2-3999-44b9-a081-1374cb45a663\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7g9km9"
Sep 15 12:09:56.143934 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:09:56.143771    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6bdc\" (UniqueName: \"kubernetes.io/projected/4cfe2b8b-0cb2-4ec9-90d7-db82647c7798-kube-api-access-t6bdc\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atj89n\" (UID: \"4cfe2b8b-0cb2-4ec9-90d7-db82647c7798\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atj89n"
Sep 15 12:09:56.144028 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:09:56.144004    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4cfe2b8b-0cb2-4ec9-90d7-db82647c7798-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atj89n\" (UID: \"4cfe2b8b-0cb2-4ec9-90d7-db82647c7798\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atj89n"
Sep 15 12:09:56.144113 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:09:56.144037    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4cfe2b8b-0cb2-4ec9-90d7-db82647c7798-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atj89n\" (UID: \"4cfe2b8b-0cb2-4ec9-90d7-db82647c7798\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atj89n"
Sep 15 12:09:56.144170 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:09:56.144125    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/97b591c2-3999-44b9-a081-1374cb45a663-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7g9km9\" (UID: \"97b591c2-3999-44b9-a081-1374cb45a663\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7g9km9"
Sep 15 12:09:56.144235 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:09:56.144213    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/97b591c2-3999-44b9-a081-1374cb45a663-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7g9km9\" (UID: \"97b591c2-3999-44b9-a081-1374cb45a663\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7g9km9"
Sep 15 12:09:56.152985 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:09:56.152959    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnw4g\" (UniqueName: \"kubernetes.io/projected/97b591c2-3999-44b9-a081-1374cb45a663-kube-api-access-pnw4g\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7g9km9\" (UID: \"97b591c2-3999-44b9-a081-1374cb45a663\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7g9km9"
Sep 15 12:09:56.153229 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:09:56.153211    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6bdc\" (UniqueName: \"kubernetes.io/projected/4cfe2b8b-0cb2-4ec9-90d7-db82647c7798-kube-api-access-t6bdc\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atj89n\" (UID: \"4cfe2b8b-0cb2-4ec9-90d7-db82647c7798\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atj89n"
Sep 15 12:09:56.212429 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:09:56.212399    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7g9km9"
Sep 15 12:09:56.278102 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:09:56.278077    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xcpgl"]
Sep 15 12:09:56.279430 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:09:56.279404    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode888b4d0_d4ad_4251_a456_223a36581af4.slice/crio-95f5e2c3b5effcece66dbfa45fbdb2f655d9fe0997b4a77389fbbb55ebaf00f2 WatchSource:0}: Error finding container 95f5e2c3b5effcece66dbfa45fbdb2f655d9fe0997b4a77389fbbb55ebaf00f2: Status 404 returned error can't find the container with id 95f5e2c3b5effcece66dbfa45fbdb2f655d9fe0997b4a77389fbbb55ebaf00f2
Sep 15 12:09:56.297059 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:09:56.297026    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atj89n"
Sep 15 12:09:56.408468 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:09:56.408435    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccv29f"]
Sep 15 12:09:56.461127 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:09:56.461096    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd902b49c_0cfa_41b0_94a7_7c667e90303e.slice/crio-a7b4d76e1cbd860c1ec74fd32b7e77727e6bf4e88e2d0394fdcacd04214bf34e WatchSource:0}: Error finding container a7b4d76e1cbd860c1ec74fd32b7e77727e6bf4e88e2d0394fdcacd04214bf34e: Status 404 returned error can't find the container with id a7b4d76e1cbd860c1ec74fd32b7e77727e6bf4e88e2d0394fdcacd04214bf34e
Sep 15 12:09:56.486697 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:09:56.486676    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7g9km9"]
Sep 15 12:09:56.500423 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:09:56.500393    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97b591c2_3999_44b9_a081_1374cb45a663.slice/crio-d1b552ffc169bbc7f1774ddacd112a7699ec06bd6e8762131b683d7ba744d909 WatchSource:0}: Error finding container d1b552ffc169bbc7f1774ddacd112a7699ec06bd6e8762131b683d7ba744d909: Status 404 returned error can't find the container with id d1b552ffc169bbc7f1774ddacd112a7699ec06bd6e8762131b683d7ba744d909
Sep 15 12:09:56.574868 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:09:56.574841    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atj89n"]
Sep 15 12:09:56.578615 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:09:56.578586    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cfe2b8b_0cb2_4ec9_90d7_db82647c7798.slice/crio-a022de86374d6f6252e6dedee1d5de94bec493a95bf9b15d5daac435944d2cf5 WatchSource:0}: Error finding container a022de86374d6f6252e6dedee1d5de94bec493a95bf9b15d5daac435944d2cf5: Status 404 returned error can't find the container with id a022de86374d6f6252e6dedee1d5de94bec493a95bf9b15d5daac435944d2cf5
Sep 15 12:09:56.797407 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:09:56.797318    2555 generic.go:356] "Generic (PLEG): container finished" podID="d902b49c-0cfa-41b0-94a7-7c667e90303e" containerID="c86914ad87b96dd0d512d51e798989247a872c384f25ebf46d82e23347b51c36" exitCode=0
Sep 15 12:09:56.797407 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:09:56.797377    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccv29f" event={"ID":"d902b49c-0cfa-41b0-94a7-7c667e90303e","Type":"ContainerDied","Data":"c86914ad87b96dd0d512d51e798989247a872c384f25ebf46d82e23347b51c36"}
Sep 15 12:09:56.797985 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:09:56.797429    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccv29f" event={"ID":"d902b49c-0cfa-41b0-94a7-7c667e90303e","Type":"ContainerStarted","Data":"a7b4d76e1cbd860c1ec74fd32b7e77727e6bf4e88e2d0394fdcacd04214bf34e"}
Sep 15 12:09:56.799107 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:09:56.799084    2555 generic.go:356] "Generic (PLEG): container finished" podID="4cfe2b8b-0cb2-4ec9-90d7-db82647c7798" containerID="8815470177d585869fabb469a9522734e0f4b3a55b490607de9af1bf30d16520" exitCode=0
Sep 15 12:09:56.799180 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:09:56.799146    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atj89n" event={"ID":"4cfe2b8b-0cb2-4ec9-90d7-db82647c7798","Type":"ContainerDied","Data":"8815470177d585869fabb469a9522734e0f4b3a55b490607de9af1bf30d16520"}
Sep 15 12:09:56.799236 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:09:56.799193    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atj89n" event={"ID":"4cfe2b8b-0cb2-4ec9-90d7-db82647c7798","Type":"ContainerStarted","Data":"a022de86374d6f6252e6dedee1d5de94bec493a95bf9b15d5daac435944d2cf5"}
Sep 15 12:09:56.800795 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:09:56.800775    2555 generic.go:356] "Generic (PLEG): container finished" podID="97b591c2-3999-44b9-a081-1374cb45a663" containerID="11556c0e3eef6f92a837005170edf74f88818002f8260229d2b9d2b59a709b98" exitCode=0
Sep 15 12:09:56.800863 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:09:56.800830    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7g9km9" event={"ID":"97b591c2-3999-44b9-a081-1374cb45a663","Type":"ContainerDied","Data":"11556c0e3eef6f92a837005170edf74f88818002f8260229d2b9d2b59a709b98"}
Sep 15 12:09:56.800923 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:09:56.800872    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7g9km9" event={"ID":"97b591c2-3999-44b9-a081-1374cb45a663","Type":"ContainerStarted","Data":"d1b552ffc169bbc7f1774ddacd112a7699ec06bd6e8762131b683d7ba744d909"}
Sep 15 12:09:56.802684 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:09:56.802561    2555 generic.go:356] "Generic (PLEG): container finished" podID="e888b4d0-d4ad-4251-a456-223a36581af4" containerID="2859aed1041223438e95ec94e193c599837939a80e74e3263db76d0261d59f53" exitCode=0
Sep 15 12:09:56.802684 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:09:56.802605    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xcpgl" event={"ID":"e888b4d0-d4ad-4251-a456-223a36581af4","Type":"ContainerDied","Data":"2859aed1041223438e95ec94e193c599837939a80e74e3263db76d0261d59f53"}
Sep 15 12:09:56.802684 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:09:56.802634    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xcpgl" event={"ID":"e888b4d0-d4ad-4251-a456-223a36581af4","Type":"ContainerStarted","Data":"95f5e2c3b5effcece66dbfa45fbdb2f655d9fe0997b4a77389fbbb55ebaf00f2"}
Sep 15 12:09:57.810520 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:09:57.810478    2555 generic.go:356] "Generic (PLEG): container finished" podID="4cfe2b8b-0cb2-4ec9-90d7-db82647c7798" containerID="52e7ecb27442f031f7b7341f307f87d9542318d47124fc69ca2672eb755746b3" exitCode=0
Sep 15 12:09:57.810970 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:09:57.810541    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atj89n" event={"ID":"4cfe2b8b-0cb2-4ec9-90d7-db82647c7798","Type":"ContainerDied","Data":"52e7ecb27442f031f7b7341f307f87d9542318d47124fc69ca2672eb755746b3"}
Sep 15 12:09:57.815828 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:09:57.815801    2555 generic.go:356] "Generic (PLEG): container finished" podID="97b591c2-3999-44b9-a081-1374cb45a663" containerID="d18ad4273542d1b2311f89eeb94c0bec15e9b31d6d181114dae076a52247267a" exitCode=0
Sep 15 12:09:57.818549 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:09:57.818522    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7g9km9" event={"ID":"97b591c2-3999-44b9-a081-1374cb45a663","Type":"ContainerDied","Data":"d18ad4273542d1b2311f89eeb94c0bec15e9b31d6d181114dae076a52247267a"}
Sep 15 12:09:58.822907 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:09:58.822871    2555 generic.go:356] "Generic (PLEG): container finished" podID="97b591c2-3999-44b9-a081-1374cb45a663" containerID="15a7a4dde46994464c381a2154ad46cdb2b51094b38f21adbc693ae833d8e3a8" exitCode=0
Sep 15 12:09:58.823352 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:09:58.822963    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7g9km9" event={"ID":"97b591c2-3999-44b9-a081-1374cb45a663","Type":"ContainerDied","Data":"15a7a4dde46994464c381a2154ad46cdb2b51094b38f21adbc693ae833d8e3a8"}
Sep 15 12:09:58.824717 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:09:58.824695    2555 generic.go:356] "Generic (PLEG): container finished" podID="e888b4d0-d4ad-4251-a456-223a36581af4" containerID="3df53afbd112e7f361174c287c6c711783b1c191d7030b37d54f86e3fc797ccf" exitCode=0
Sep 15 12:09:58.824821 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:09:58.824772    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xcpgl" event={"ID":"e888b4d0-d4ad-4251-a456-223a36581af4","Type":"ContainerDied","Data":"3df53afbd112e7f361174c287c6c711783b1c191d7030b37d54f86e3fc797ccf"}
Sep 15 12:09:58.826665 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:09:58.826647    2555 generic.go:356] "Generic (PLEG): container finished" podID="d902b49c-0cfa-41b0-94a7-7c667e90303e" containerID="cb7e84a9ebf9a7b4dddff5cf29c136a285e50e262d8bbc5c20a67a02f00b2ebe" exitCode=0
Sep 15 12:09:58.826755 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:09:58.826732    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccv29f" event={"ID":"d902b49c-0cfa-41b0-94a7-7c667e90303e","Type":"ContainerDied","Data":"cb7e84a9ebf9a7b4dddff5cf29c136a285e50e262d8bbc5c20a67a02f00b2ebe"}
Sep 15 12:09:58.828945 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:09:58.828928    2555 generic.go:356] "Generic (PLEG): container finished" podID="4cfe2b8b-0cb2-4ec9-90d7-db82647c7798" containerID="0530afab0b3feb68e91a9d0bab69c0654d88deebc45fd6705aad829d1d291cd2" exitCode=0
Sep 15 12:09:58.829034 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:09:58.828965    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atj89n" event={"ID":"4cfe2b8b-0cb2-4ec9-90d7-db82647c7798","Type":"ContainerDied","Data":"0530afab0b3feb68e91a9d0bab69c0654d88deebc45fd6705aad829d1d291cd2"}
Sep 15 12:09:59.835864 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:09:59.835833    2555 generic.go:356] "Generic (PLEG): container finished" podID="e888b4d0-d4ad-4251-a456-223a36581af4" containerID="4502025d975590b3ff5b592e11f8d0440270bcc591bca64b1a9798138fee33c6" exitCode=0
Sep 15 12:09:59.836337 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:09:59.835898    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xcpgl" event={"ID":"e888b4d0-d4ad-4251-a456-223a36581af4","Type":"ContainerDied","Data":"4502025d975590b3ff5b592e11f8d0440270bcc591bca64b1a9798138fee33c6"}
Sep 15 12:09:59.838077 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:09:59.838056    2555 generic.go:356] "Generic (PLEG): container finished" podID="d902b49c-0cfa-41b0-94a7-7c667e90303e" containerID="17b728a643644e8feccdf1ffae654bba0070d045bffc928dd44df509438d4276" exitCode=0
Sep 15 12:09:59.838167 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:09:59.838137    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccv29f" event={"ID":"d902b49c-0cfa-41b0-94a7-7c667e90303e","Type":"ContainerDied","Data":"17b728a643644e8feccdf1ffae654bba0070d045bffc928dd44df509438d4276"}
Sep 15 12:09:59.970064 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:09:59.970035    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7g9km9"
Sep 15 12:10:00.001669 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:10:00.001640    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atj89n"
Sep 15 12:10:00.076938 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:10:00.076905    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t6bdc\" (UniqueName: \"kubernetes.io/projected/4cfe2b8b-0cb2-4ec9-90d7-db82647c7798-kube-api-access-t6bdc\") pod \"4cfe2b8b-0cb2-4ec9-90d7-db82647c7798\" (UID: \"4cfe2b8b-0cb2-4ec9-90d7-db82647c7798\") "
Sep 15 12:10:00.076938 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:10:00.076943    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pnw4g\" (UniqueName: \"kubernetes.io/projected/97b591c2-3999-44b9-a081-1374cb45a663-kube-api-access-pnw4g\") pod \"97b591c2-3999-44b9-a081-1374cb45a663\" (UID: \"97b591c2-3999-44b9-a081-1374cb45a663\") "
Sep 15 12:10:00.077191 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:10:00.077004    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/97b591c2-3999-44b9-a081-1374cb45a663-bundle\") pod \"97b591c2-3999-44b9-a081-1374cb45a663\" (UID: \"97b591c2-3999-44b9-a081-1374cb45a663\") "
Sep 15 12:10:00.077191 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:10:00.077044    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/97b591c2-3999-44b9-a081-1374cb45a663-util\") pod \"97b591c2-3999-44b9-a081-1374cb45a663\" (UID: \"97b591c2-3999-44b9-a081-1374cb45a663\") "
Sep 15 12:10:00.077191 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:10:00.077067    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4cfe2b8b-0cb2-4ec9-90d7-db82647c7798-bundle\") pod \"4cfe2b8b-0cb2-4ec9-90d7-db82647c7798\" (UID: \"4cfe2b8b-0cb2-4ec9-90d7-db82647c7798\") "
Sep 15 12:10:00.077191 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:10:00.077085    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4cfe2b8b-0cb2-4ec9-90d7-db82647c7798-util\") pod \"4cfe2b8b-0cb2-4ec9-90d7-db82647c7798\" (UID: \"4cfe2b8b-0cb2-4ec9-90d7-db82647c7798\") "
Sep 15 12:10:00.077772 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:10:00.077740    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97b591c2-3999-44b9-a081-1374cb45a663-bundle" (OuterVolumeSpecName: "bundle") pod "97b591c2-3999-44b9-a081-1374cb45a663" (UID: "97b591c2-3999-44b9-a081-1374cb45a663"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:10:00.077987 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:10:00.077965    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cfe2b8b-0cb2-4ec9-90d7-db82647c7798-bundle" (OuterVolumeSpecName: "bundle") pod "4cfe2b8b-0cb2-4ec9-90d7-db82647c7798" (UID: "4cfe2b8b-0cb2-4ec9-90d7-db82647c7798"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:10:00.079139 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:10:00.079120    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97b591c2-3999-44b9-a081-1374cb45a663-kube-api-access-pnw4g" (OuterVolumeSpecName: "kube-api-access-pnw4g") pod "97b591c2-3999-44b9-a081-1374cb45a663" (UID: "97b591c2-3999-44b9-a081-1374cb45a663"). InnerVolumeSpecName "kube-api-access-pnw4g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:10:00.079493 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:10:00.079469    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cfe2b8b-0cb2-4ec9-90d7-db82647c7798-kube-api-access-t6bdc" (OuterVolumeSpecName: "kube-api-access-t6bdc") pod "4cfe2b8b-0cb2-4ec9-90d7-db82647c7798" (UID: "4cfe2b8b-0cb2-4ec9-90d7-db82647c7798"). InnerVolumeSpecName "kube-api-access-t6bdc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:10:00.083217 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:10:00.083177    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cfe2b8b-0cb2-4ec9-90d7-db82647c7798-util" (OuterVolumeSpecName: "util") pod "4cfe2b8b-0cb2-4ec9-90d7-db82647c7798" (UID: "4cfe2b8b-0cb2-4ec9-90d7-db82647c7798"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:10:00.084700 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:10:00.084673    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97b591c2-3999-44b9-a081-1374cb45a663-util" (OuterVolumeSpecName: "util") pod "97b591c2-3999-44b9-a081-1374cb45a663" (UID: "97b591c2-3999-44b9-a081-1374cb45a663"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:10:00.178217 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:10:00.178181    2555 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4cfe2b8b-0cb2-4ec9-90d7-db82647c7798-util\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:10:00.178217 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:10:00.178214    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6bdc\" (UniqueName: \"kubernetes.io/projected/4cfe2b8b-0cb2-4ec9-90d7-db82647c7798-kube-api-access-t6bdc\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:10:00.178217 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:10:00.178225    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnw4g\" (UniqueName: \"kubernetes.io/projected/97b591c2-3999-44b9-a081-1374cb45a663-kube-api-access-pnw4g\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:10:00.178473 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:10:00.178235    2555 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/97b591c2-3999-44b9-a081-1374cb45a663-bundle\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:10:00.178473 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:10:00.178243    2555 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/97b591c2-3999-44b9-a081-1374cb45a663-util\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:10:00.178473 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:10:00.178251    2555 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4cfe2b8b-0cb2-4ec9-90d7-db82647c7798-bundle\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:10:00.849622 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:10:00.849590    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atj89n"
Sep 15 12:10:00.849622 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:10:00.849609    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atj89n" event={"ID":"4cfe2b8b-0cb2-4ec9-90d7-db82647c7798","Type":"ContainerDied","Data":"a022de86374d6f6252e6dedee1d5de94bec493a95bf9b15d5daac435944d2cf5"}
Sep 15 12:10:00.850082 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:10:00.849662    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a022de86374d6f6252e6dedee1d5de94bec493a95bf9b15d5daac435944d2cf5"
Sep 15 12:10:00.851555 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:10:00.851531    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7g9km9"
Sep 15 12:10:00.851705 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:10:00.851561    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7g9km9" event={"ID":"97b591c2-3999-44b9-a081-1374cb45a663","Type":"ContainerDied","Data":"d1b552ffc169bbc7f1774ddacd112a7699ec06bd6e8762131b683d7ba744d909"}
Sep 15 12:10:00.851705 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:10:00.851597    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d1b552ffc169bbc7f1774ddacd112a7699ec06bd6e8762131b683d7ba744d909"
Sep 15 12:10:01.006931 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:10:01.006903    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xcpgl"
Sep 15 12:10:01.008930 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:10:01.008912    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccv29f"
Sep 15 12:10:01.085240 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:10:01.085201    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vfmwd\" (UniqueName: \"kubernetes.io/projected/e888b4d0-d4ad-4251-a456-223a36581af4-kube-api-access-vfmwd\") pod \"e888b4d0-d4ad-4251-a456-223a36581af4\" (UID: \"e888b4d0-d4ad-4251-a456-223a36581af4\") "
Sep 15 12:10:01.085477 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:10:01.085247    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d902b49c-0cfa-41b0-94a7-7c667e90303e-util\") pod \"d902b49c-0cfa-41b0-94a7-7c667e90303e\" (UID: \"d902b49c-0cfa-41b0-94a7-7c667e90303e\") "
Sep 15 12:10:01.085477 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:10:01.085317    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d902b49c-0cfa-41b0-94a7-7c667e90303e-bundle\") pod \"d902b49c-0cfa-41b0-94a7-7c667e90303e\" (UID: \"d902b49c-0cfa-41b0-94a7-7c667e90303e\") "
Sep 15 12:10:01.085477 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:10:01.085375    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e888b4d0-d4ad-4251-a456-223a36581af4-util\") pod \"e888b4d0-d4ad-4251-a456-223a36581af4\" (UID: \"e888b4d0-d4ad-4251-a456-223a36581af4\") "
Sep 15 12:10:01.085477 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:10:01.085414    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8xfwg\" (UniqueName: \"kubernetes.io/projected/d902b49c-0cfa-41b0-94a7-7c667e90303e-kube-api-access-8xfwg\") pod \"d902b49c-0cfa-41b0-94a7-7c667e90303e\" (UID: \"d902b49c-0cfa-41b0-94a7-7c667e90303e\") "
Sep 15 12:10:01.085477 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:10:01.085449    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e888b4d0-d4ad-4251-a456-223a36581af4-bundle\") pod \"e888b4d0-d4ad-4251-a456-223a36581af4\" (UID: \"e888b4d0-d4ad-4251-a456-223a36581af4\") "
Sep 15 12:10:01.085990 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:10:01.085959    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d902b49c-0cfa-41b0-94a7-7c667e90303e-bundle" (OuterVolumeSpecName: "bundle") pod "d902b49c-0cfa-41b0-94a7-7c667e90303e" (UID: "d902b49c-0cfa-41b0-94a7-7c667e90303e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:10:01.085990 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:10:01.085983    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e888b4d0-d4ad-4251-a456-223a36581af4-bundle" (OuterVolumeSpecName: "bundle") pod "e888b4d0-d4ad-4251-a456-223a36581af4" (UID: "e888b4d0-d4ad-4251-a456-223a36581af4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:10:01.087599 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:10:01.087571    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e888b4d0-d4ad-4251-a456-223a36581af4-kube-api-access-vfmwd" (OuterVolumeSpecName: "kube-api-access-vfmwd") pod "e888b4d0-d4ad-4251-a456-223a36581af4" (UID: "e888b4d0-d4ad-4251-a456-223a36581af4"). InnerVolumeSpecName "kube-api-access-vfmwd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:10:01.087697 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:10:01.087601    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d902b49c-0cfa-41b0-94a7-7c667e90303e-kube-api-access-8xfwg" (OuterVolumeSpecName: "kube-api-access-8xfwg") pod "d902b49c-0cfa-41b0-94a7-7c667e90303e" (UID: "d902b49c-0cfa-41b0-94a7-7c667e90303e"). InnerVolumeSpecName "kube-api-access-8xfwg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:10:01.091204 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:10:01.091176    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d902b49c-0cfa-41b0-94a7-7c667e90303e-util" (OuterVolumeSpecName: "util") pod "d902b49c-0cfa-41b0-94a7-7c667e90303e" (UID: "d902b49c-0cfa-41b0-94a7-7c667e90303e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:10:01.093665 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:10:01.093636    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e888b4d0-d4ad-4251-a456-223a36581af4-util" (OuterVolumeSpecName: "util") pod "e888b4d0-d4ad-4251-a456-223a36581af4" (UID: "e888b4d0-d4ad-4251-a456-223a36581af4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:10:01.186385 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:10:01.186337    2555 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e888b4d0-d4ad-4251-a456-223a36581af4-bundle\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:10:01.186385 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:10:01.186378    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfmwd\" (UniqueName: \"kubernetes.io/projected/e888b4d0-d4ad-4251-a456-223a36581af4-kube-api-access-vfmwd\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:10:01.186385 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:10:01.186388    2555 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d902b49c-0cfa-41b0-94a7-7c667e90303e-util\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:10:01.186385 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:10:01.186397    2555 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d902b49c-0cfa-41b0-94a7-7c667e90303e-bundle\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:10:01.186385 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:10:01.186404    2555 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e888b4d0-d4ad-4251-a456-223a36581af4-util\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:10:01.186750 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:10:01.186412    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xfwg\" (UniqueName: \"kubernetes.io/projected/d902b49c-0cfa-41b0-94a7-7c667e90303e-kube-api-access-8xfwg\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:10:01.859068 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:10:01.859035    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xcpgl"
Sep 15 12:10:01.859530 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:10:01.859035    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xcpgl" event={"ID":"e888b4d0-d4ad-4251-a456-223a36581af4","Type":"ContainerDied","Data":"95f5e2c3b5effcece66dbfa45fbdb2f655d9fe0997b4a77389fbbb55ebaf00f2"}
Sep 15 12:10:01.859530 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:10:01.859161    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="95f5e2c3b5effcece66dbfa45fbdb2f655d9fe0997b4a77389fbbb55ebaf00f2"
Sep 15 12:10:01.861283 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:10:01.861245    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccv29f" event={"ID":"d902b49c-0cfa-41b0-94a7-7c667e90303e","Type":"ContainerDied","Data":"a7b4d76e1cbd860c1ec74fd32b7e77727e6bf4e88e2d0394fdcacd04214bf34e"}
Sep 15 12:10:01.861411 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:10:01.861301    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a7b4d76e1cbd860c1ec74fd32b7e77727e6bf4e88e2d0394fdcacd04214bf34e"
Sep 15 12:10:01.861411 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:10:01.861256    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccv29f"
Sep 15 12:10:22.970549 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:10:22.970510    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-r7tk9"]
Sep 15 12:10:22.970929 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:10:22.970807    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97b591c2-3999-44b9-a081-1374cb45a663" containerName="extract"
Sep 15 12:10:22.970929 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:10:22.970817    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="97b591c2-3999-44b9-a081-1374cb45a663" containerName="extract"
Sep 15 12:10:22.970929 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:10:22.970826    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97b591c2-3999-44b9-a081-1374cb45a663" containerName="util"
Sep 15 12:10:22.970929 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:10:22.970832    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="97b591c2-3999-44b9-a081-1374cb45a663" containerName="util"
Sep 15 12:10:22.970929 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:10:22.970839    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e888b4d0-d4ad-4251-a456-223a36581af4" containerName="extract"
Sep 15 12:10:22.970929 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:10:22.970845    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e888b4d0-d4ad-4251-a456-223a36581af4" containerName="extract"
Sep 15 12:10:22.970929 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:10:22.970853    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97b591c2-3999-44b9-a081-1374cb45a663" containerName="pull"
Sep 15 12:10:22.970929 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:10:22.970858    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="97b591c2-3999-44b9-a081-1374cb45a663" containerName="pull"
Sep 15 12:10:22.970929 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:10:22.970865    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cfe2b8b-0cb2-4ec9-90d7-db82647c7798" containerName="extract"
Sep 15 12:10:22.970929 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:10:22.970870    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cfe2b8b-0cb2-4ec9-90d7-db82647c7798" containerName="extract"
Sep 15 12:10:22.970929 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:10:22.970884    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cfe2b8b-0cb2-4ec9-90d7-db82647c7798" containerName="pull"
Sep 15 12:10:22.970929 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:10:22.970888    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cfe2b8b-0cb2-4ec9-90d7-db82647c7798" containerName="pull"
Sep 15 12:10:22.970929 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:10:22.970894    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d902b49c-0cfa-41b0-94a7-7c667e90303e" containerName="extract"
Sep 15 12:10:22.970929 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:10:22.970899    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="d902b49c-0cfa-41b0-94a7-7c667e90303e" containerName="extract"
Sep 15 12:10:22.970929 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:10:22.970906    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d902b49c-0cfa-41b0-94a7-7c667e90303e" containerName="pull"
Sep 15 12:10:22.970929 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:10:22.970912    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="d902b49c-0cfa-41b0-94a7-7c667e90303e" containerName="pull"
Sep 15 12:10:22.970929 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:10:22.970919    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cfe2b8b-0cb2-4ec9-90d7-db82647c7798" containerName="util"
Sep 15 12:10:22.970929 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:10:22.970923    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cfe2b8b-0cb2-4ec9-90d7-db82647c7798" containerName="util"
Sep 15 12:10:22.970929 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:10:22.970932    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e888b4d0-d4ad-4251-a456-223a36581af4" containerName="util"
Sep 15 12:10:22.970929 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:10:22.970937    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e888b4d0-d4ad-4251-a456-223a36581af4" containerName="util"
Sep 15 12:10:22.971526 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:10:22.970942    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e888b4d0-d4ad-4251-a456-223a36581af4" containerName="pull"
Sep 15 12:10:22.971526 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:10:22.970947    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e888b4d0-d4ad-4251-a456-223a36581af4" containerName="pull"
Sep 15 12:10:22.971526 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:10:22.970955    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d902b49c-0cfa-41b0-94a7-7c667e90303e" containerName="util"
Sep 15 12:10:22.971526 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:10:22.970959    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="d902b49c-0cfa-41b0-94a7-7c667e90303e" containerName="util"
Sep 15 12:10:22.971526 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:10:22.971002    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="d902b49c-0cfa-41b0-94a7-7c667e90303e" containerName="extract"
Sep 15 12:10:22.971526 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:10:22.971011    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="97b591c2-3999-44b9-a081-1374cb45a663" containerName="extract"
Sep 15 12:10:22.971526 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:10:22.971017    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="e888b4d0-d4ad-4251-a456-223a36581af4" containerName="extract"
Sep 15 12:10:22.971526 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:10:22.971023    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cfe2b8b-0cb2-4ec9-90d7-db82647c7798" containerName="extract"
Sep 15 12:10:22.973872 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:10:22.973853    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-r7tk9"
Sep 15 12:10:22.976430 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:10:22.976409    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 15 12:10:22.976548 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:10:22.976422    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 15 12:10:22.976548 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:10:22.976492    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-n2ln9\""
Sep 15 12:10:22.977372 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:10:22.977351    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 15 12:10:22.977455 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:10:22.977393    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 15 12:10:22.983639 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:10:22.983611    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-r7tk9"]
Sep 15 12:10:23.075561 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:10:23.075518    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqtd8\" (UniqueName: \"kubernetes.io/projected/88a41244-5e05-46f1-965c-c4bcd1f9ae1b-kube-api-access-wqtd8\") pod \"kuadrant-console-plugin-fbdd4f445-r7tk9\" (UID: \"88a41244-5e05-46f1-965c-c4bcd1f9ae1b\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-r7tk9"
Sep 15 12:10:23.075737 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:10:23.075572    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/88a41244-5e05-46f1-965c-c4bcd1f9ae1b-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-r7tk9\" (UID: \"88a41244-5e05-46f1-965c-c4bcd1f9ae1b\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-r7tk9"
Sep 15 12:10:23.075737 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:10:23.075652    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/88a41244-5e05-46f1-965c-c4bcd1f9ae1b-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-r7tk9\" (UID: \"88a41244-5e05-46f1-965c-c4bcd1f9ae1b\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-r7tk9"
Sep 15 12:10:23.176123 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:10:23.176080    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqtd8\" (UniqueName: \"kubernetes.io/projected/88a41244-5e05-46f1-965c-c4bcd1f9ae1b-kube-api-access-wqtd8\") pod \"kuadrant-console-plugin-fbdd4f445-r7tk9\" (UID: \"88a41244-5e05-46f1-965c-c4bcd1f9ae1b\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-r7tk9"
Sep 15 12:10:23.176123 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:10:23.176131    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/88a41244-5e05-46f1-965c-c4bcd1f9ae1b-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-r7tk9\" (UID: \"88a41244-5e05-46f1-965c-c4bcd1f9ae1b\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-r7tk9"
Sep 15 12:10:23.176366 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:10:23.176173    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/88a41244-5e05-46f1-965c-c4bcd1f9ae1b-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-r7tk9\" (UID: \"88a41244-5e05-46f1-965c-c4bcd1f9ae1b\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-r7tk9"
Sep 15 12:10:23.176939 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:10:23.176916    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/88a41244-5e05-46f1-965c-c4bcd1f9ae1b-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-r7tk9\" (UID: \"88a41244-5e05-46f1-965c-c4bcd1f9ae1b\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-r7tk9"
Sep 15 12:10:23.178762 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:10:23.178735    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/88a41244-5e05-46f1-965c-c4bcd1f9ae1b-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-r7tk9\" (UID: \"88a41244-5e05-46f1-965c-c4bcd1f9ae1b\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-r7tk9"
Sep 15 12:10:23.184728 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:10:23.184691    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqtd8\" (UniqueName: \"kubernetes.io/projected/88a41244-5e05-46f1-965c-c4bcd1f9ae1b-kube-api-access-wqtd8\") pod \"kuadrant-console-plugin-fbdd4f445-r7tk9\" (UID: \"88a41244-5e05-46f1-965c-c4bcd1f9ae1b\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-r7tk9"
Sep 15 12:10:23.284543 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:10:23.284465    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-r7tk9"
Sep 15 12:10:23.564561 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:10:23.564536    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-r7tk9"]
Sep 15 12:10:23.566847 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:10:23.566825    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod88a41244_5e05_46f1_965c_c4bcd1f9ae1b.slice/crio-fe02925a5a373e2a2af614d3b67de06a0b3b995a3890af0d314ad24129fb1618 WatchSource:0}: Error finding container fe02925a5a373e2a2af614d3b67de06a0b3b995a3890af0d314ad24129fb1618: Status 404 returned error can't find the container with id fe02925a5a373e2a2af614d3b67de06a0b3b995a3890af0d314ad24129fb1618
Sep 15 12:10:23.965693 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:10:23.965657    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-r7tk9" event={"ID":"88a41244-5e05-46f1-965c-c4bcd1f9ae1b","Type":"ContainerStarted","Data":"fe02925a5a373e2a2af614d3b67de06a0b3b995a3890af0d314ad24129fb1618"}
Sep 15 12:10:32.009493 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:10:32.009457    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-r7tk9" event={"ID":"88a41244-5e05-46f1-965c-c4bcd1f9ae1b","Type":"ContainerStarted","Data":"601b8ee28228d1bca91a7f4a4eeaa9fdc3d9c05206f90611074c1369973328aa"}
Sep 15 12:10:32.071478 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:10:32.071424    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-r7tk9" podStartSLOduration=1.779813281 podStartE2EDuration="10.071406876s" totalImagesPullingTime="8.291593595s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:10:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:10:32.070517175 +0000 UTC m=+488.861439689" watchObservedRunningTime="2026-09-15 12:10:32.071406876 +0000 UTC m=+488.862329426"
Sep 15 12:11:10.546855 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:11:10.546809    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5gkxs"]
Sep 15 12:11:10.568696 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:11:10.568658    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5gkxs"]
Sep 15 12:11:10.568850 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:11:10.568786    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-5gkxs"
Sep 15 12:11:10.571527 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:11:10.571507    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 15 12:11:10.578421 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:11:10.578396    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5gkxs"]
Sep 15 12:11:10.688570 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:11:10.688530    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7db9\" (UniqueName: \"kubernetes.io/projected/c6b055e4-6a78-4dfa-88de-0aba5339f311-kube-api-access-k7db9\") pod \"limitador-limitador-84994f7ffc-5gkxs\" (UID: \"c6b055e4-6a78-4dfa-88de-0aba5339f311\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-5gkxs"
Sep 15 12:11:10.688570 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:11:10.688577    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c6b055e4-6a78-4dfa-88de-0aba5339f311-config-file\") pod \"limitador-limitador-84994f7ffc-5gkxs\" (UID: \"c6b055e4-6a78-4dfa-88de-0aba5339f311\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-5gkxs"
Sep 15 12:11:10.789639 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:11:10.789589    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7db9\" (UniqueName: \"kubernetes.io/projected/c6b055e4-6a78-4dfa-88de-0aba5339f311-kube-api-access-k7db9\") pod \"limitador-limitador-84994f7ffc-5gkxs\" (UID: \"c6b055e4-6a78-4dfa-88de-0aba5339f311\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-5gkxs"
Sep 15 12:11:10.789639 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:11:10.789642    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c6b055e4-6a78-4dfa-88de-0aba5339f311-config-file\") pod \"limitador-limitador-84994f7ffc-5gkxs\" (UID: \"c6b055e4-6a78-4dfa-88de-0aba5339f311\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-5gkxs"
Sep 15 12:11:10.790294 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:11:10.790250    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c6b055e4-6a78-4dfa-88de-0aba5339f311-config-file\") pod \"limitador-limitador-84994f7ffc-5gkxs\" (UID: \"c6b055e4-6a78-4dfa-88de-0aba5339f311\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-5gkxs"
Sep 15 12:11:10.798008 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:11:10.797957    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7db9\" (UniqueName: \"kubernetes.io/projected/c6b055e4-6a78-4dfa-88de-0aba5339f311-kube-api-access-k7db9\") pod \"limitador-limitador-84994f7ffc-5gkxs\" (UID: \"c6b055e4-6a78-4dfa-88de-0aba5339f311\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-5gkxs"
Sep 15 12:11:10.880097 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:11:10.880060    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-5gkxs"
Sep 15 12:11:10.968109 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:11:10.968072    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-pbgbs"]
Sep 15 12:11:10.971777 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:11:10.971759    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-pbgbs"
Sep 15 12:11:10.975331 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:11:10.975305    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-k4g6t\""
Sep 15 12:11:10.980281 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:11:10.980242    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-pbgbs"]
Sep 15 12:11:11.092326 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:11:11.092211    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-889hp\" (UniqueName: \"kubernetes.io/projected/e0f1bee8-89ca-48f5-8944-744924afcad7-kube-api-access-889hp\") pod \"authorino-7bcfd8f9cd-pbgbs\" (UID: \"e0f1bee8-89ca-48f5-8944-744924afcad7\") " pod="kuadrant-system/authorino-7bcfd8f9cd-pbgbs"
Sep 15 12:11:11.163131 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:11:11.163106    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5gkxs"]
Sep 15 12:11:11.165414 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:11:11.165385    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6b055e4_6a78_4dfa_88de_0aba5339f311.slice/crio-2b09e3d6624c35662d800649b55655a7a2a50e6c44bf3622d98fe125c35542df WatchSource:0}: Error finding container 2b09e3d6624c35662d800649b55655a7a2a50e6c44bf3622d98fe125c35542df: Status 404 returned error can't find the container with id 2b09e3d6624c35662d800649b55655a7a2a50e6c44bf3622d98fe125c35542df
Sep 15 12:11:11.192651 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:11:11.192620    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-889hp\" (UniqueName: \"kubernetes.io/projected/e0f1bee8-89ca-48f5-8944-744924afcad7-kube-api-access-889hp\") pod \"authorino-7bcfd8f9cd-pbgbs\" (UID: \"e0f1bee8-89ca-48f5-8944-744924afcad7\") " pod="kuadrant-system/authorino-7bcfd8f9cd-pbgbs"
Sep 15 12:11:11.192939 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:11:11.192915    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-5gkxs" event={"ID":"c6b055e4-6a78-4dfa-88de-0aba5339f311","Type":"ContainerStarted","Data":"2b09e3d6624c35662d800649b55655a7a2a50e6c44bf3622d98fe125c35542df"}
Sep 15 12:11:11.201819 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:11:11.201791    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-889hp\" (UniqueName: \"kubernetes.io/projected/e0f1bee8-89ca-48f5-8944-744924afcad7-kube-api-access-889hp\") pod \"authorino-7bcfd8f9cd-pbgbs\" (UID: \"e0f1bee8-89ca-48f5-8944-744924afcad7\") " pod="kuadrant-system/authorino-7bcfd8f9cd-pbgbs"
Sep 15 12:11:11.282797 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:11:11.282759    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-pbgbs"
Sep 15 12:11:11.560716 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:11:11.560688    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-pbgbs"]
Sep 15 12:11:11.562409 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:11:11.562383    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode0f1bee8_89ca_48f5_8944_744924afcad7.slice/crio-47756740fa4141bc06cc8623d338b13a91d96690fe47280003dae5a610c3bba6 WatchSource:0}: Error finding container 47756740fa4141bc06cc8623d338b13a91d96690fe47280003dae5a610c3bba6: Status 404 returned error can't find the container with id 47756740fa4141bc06cc8623d338b13a91d96690fe47280003dae5a610c3bba6
Sep 15 12:11:12.208105 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:11:12.208060    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-pbgbs" event={"ID":"e0f1bee8-89ca-48f5-8944-744924afcad7","Type":"ContainerStarted","Data":"47756740fa4141bc06cc8623d338b13a91d96690fe47280003dae5a610c3bba6"}
Sep 15 12:11:14.222815 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:11:14.222776    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-pbgbs" event={"ID":"e0f1bee8-89ca-48f5-8944-744924afcad7","Type":"ContainerStarted","Data":"e2ff8f55dfb404ab0e94510df11e6011b17bdc0be8ee43a6b35f19654bfc34eb"}
Sep 15 12:11:14.224339 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:11:14.224310    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-5gkxs" event={"ID":"c6b055e4-6a78-4dfa-88de-0aba5339f311","Type":"ContainerStarted","Data":"5e63190099377658eb62b4e6ca73d5483f2770401e68f82c0785aa0db6fe6125"}
Sep 15 12:11:14.224464 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:11:14.224444    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-5gkxs"
Sep 15 12:11:14.239628 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:11:14.239578    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-pbgbs" podStartSLOduration=1.803237441 podStartE2EDuration="4.239561953s" totalImagesPullingTime="2.436324512s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:11:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:11:14.239254242 +0000 UTC m=+531.030176778" watchObservedRunningTime="2026-09-15 12:11:14.239561953 +0000 UTC m=+531.030484484"
Sep 15 12:11:14.259826 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:11:14.259710    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-5gkxs" podStartSLOduration=1.434461033 podStartE2EDuration="4.25968231s" totalImagesPullingTime="2.825221277s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:11:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:11:14.259107659 +0000 UTC m=+531.050030173" watchObservedRunningTime="2026-09-15 12:11:14.25968231 +0000 UTC m=+531.050604823"
Sep 15 12:11:14.385904 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:11:14.385862    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-pbgbs"]
Sep 15 12:11:16.234318 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:11:16.234223    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-pbgbs" podUID="e0f1bee8-89ca-48f5-8944-744924afcad7" containerName="authorino" containerID="cri-o://e2ff8f55dfb404ab0e94510df11e6011b17bdc0be8ee43a6b35f19654bfc34eb" gracePeriod=30
Sep 15 12:11:16.484012 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:11:16.483987    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-pbgbs"
Sep 15 12:11:16.533565 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:11:16.533471    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-889hp\" (UniqueName: \"kubernetes.io/projected/e0f1bee8-89ca-48f5-8944-744924afcad7-kube-api-access-889hp\") pod \"e0f1bee8-89ca-48f5-8944-744924afcad7\" (UID: \"e0f1bee8-89ca-48f5-8944-744924afcad7\") "
Sep 15 12:11:16.535594 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:11:16.535562    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0f1bee8-89ca-48f5-8944-744924afcad7-kube-api-access-889hp" (OuterVolumeSpecName: "kube-api-access-889hp") pod "e0f1bee8-89ca-48f5-8944-744924afcad7" (UID: "e0f1bee8-89ca-48f5-8944-744924afcad7"). InnerVolumeSpecName "kube-api-access-889hp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:11:16.634053 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:11:16.634010    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-889hp\" (UniqueName: \"kubernetes.io/projected/e0f1bee8-89ca-48f5-8944-744924afcad7-kube-api-access-889hp\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:11:17.240364 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:11:17.240332    2555 generic.go:356] "Generic (PLEG): container finished" podID="e0f1bee8-89ca-48f5-8944-744924afcad7" containerID="e2ff8f55dfb404ab0e94510df11e6011b17bdc0be8ee43a6b35f19654bfc34eb" exitCode=0
Sep 15 12:11:17.240832 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:11:17.240371    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-pbgbs"
Sep 15 12:11:17.240832 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:11:17.240411    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-pbgbs" event={"ID":"e0f1bee8-89ca-48f5-8944-744924afcad7","Type":"ContainerDied","Data":"e2ff8f55dfb404ab0e94510df11e6011b17bdc0be8ee43a6b35f19654bfc34eb"}
Sep 15 12:11:17.240832 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:11:17.240457    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-pbgbs" event={"ID":"e0f1bee8-89ca-48f5-8944-744924afcad7","Type":"ContainerDied","Data":"47756740fa4141bc06cc8623d338b13a91d96690fe47280003dae5a610c3bba6"}
Sep 15 12:11:17.240832 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:11:17.240493    2555 scope.go:117] "RemoveContainer" containerID="e2ff8f55dfb404ab0e94510df11e6011b17bdc0be8ee43a6b35f19654bfc34eb"
Sep 15 12:11:17.249657 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:11:17.249640    2555 scope.go:117] "RemoveContainer" containerID="e2ff8f55dfb404ab0e94510df11e6011b17bdc0be8ee43a6b35f19654bfc34eb"
Sep 15 12:11:17.249986 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:11:17.249944    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2ff8f55dfb404ab0e94510df11e6011b17bdc0be8ee43a6b35f19654bfc34eb\": container with ID starting with e2ff8f55dfb404ab0e94510df11e6011b17bdc0be8ee43a6b35f19654bfc34eb not found: ID does not exist" containerID="e2ff8f55dfb404ab0e94510df11e6011b17bdc0be8ee43a6b35f19654bfc34eb"
Sep 15 12:11:17.250029 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:11:17.249995    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2ff8f55dfb404ab0e94510df11e6011b17bdc0be8ee43a6b35f19654bfc34eb"} err="failed to get container status \"e2ff8f55dfb404ab0e94510df11e6011b17bdc0be8ee43a6b35f19654bfc34eb\": rpc error: code = NotFound desc = could not find container \"e2ff8f55dfb404ab0e94510df11e6011b17bdc0be8ee43a6b35f19654bfc34eb\": container with ID starting with e2ff8f55dfb404ab0e94510df11e6011b17bdc0be8ee43a6b35f19654bfc34eb not found: ID does not exist"
Sep 15 12:11:17.272030 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:11:17.271973    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-pbgbs"]
Sep 15 12:11:17.274436 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:11:17.274411    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-pbgbs"]
Sep 15 12:11:17.816682 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:11:17.816648    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0f1bee8-89ca-48f5-8944-744924afcad7" path="/var/lib/kubelet/pods/e0f1bee8-89ca-48f5-8944-744924afcad7/volumes"
Sep 15 12:11:25.230108 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:11:25.230074    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-5gkxs"
Sep 15 12:12:31.525629 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:12:31.525593    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5gkxs"]
Sep 15 12:12:46.769608 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:12:46.769516    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5gkxs"]
Sep 15 12:12:54.866008 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:12:54.865962    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5gkxs"]
Sep 15 12:13:21.025122 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:13:21.025083    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-p7blm"]
Sep 15 12:13:21.025564 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:13:21.025466    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0f1bee8-89ca-48f5-8944-744924afcad7" containerName="authorino"
Sep 15 12:13:21.025564 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:13:21.025479    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0f1bee8-89ca-48f5-8944-744924afcad7" containerName="authorino"
Sep 15 12:13:21.025564 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:13:21.025534    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0f1bee8-89ca-48f5-8944-744924afcad7" containerName="authorino"
Sep 15 12:13:21.028461 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:13:21.028443    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-p7blm"
Sep 15 12:13:21.035007 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:13:21.034983    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 15 12:13:21.035886 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:13:21.035863    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 15 12:13:21.036009 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:13:21.035885    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Sep 15 12:13:21.036080 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:13:21.036059    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-c9vjk\""
Sep 15 12:13:21.041485 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:13:21.041462    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-p7blm"]
Sep 15 12:13:21.073715 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:13:21.073679    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5v45f\" (UniqueName: \"kubernetes.io/projected/2409c9c4-0f91-41f5-b598-d9f342a626be-kube-api-access-5v45f\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-p7blm\" (UID: \"2409c9c4-0f91-41f5-b598-d9f342a626be\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-p7blm"
Sep 15 12:13:21.073899 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:13:21.073734    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2409c9c4-0f91-41f5-b598-d9f342a626be-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-p7blm\" (UID: \"2409c9c4-0f91-41f5-b598-d9f342a626be\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-p7blm"
Sep 15 12:13:21.073899 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:13:21.073819    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2409c9c4-0f91-41f5-b598-d9f342a626be-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-p7blm\" (UID: \"2409c9c4-0f91-41f5-b598-d9f342a626be\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-p7blm"
Sep 15 12:13:21.073899 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:13:21.073864    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2409c9c4-0f91-41f5-b598-d9f342a626be-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-p7blm\" (UID: \"2409c9c4-0f91-41f5-b598-d9f342a626be\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-p7blm"
Sep 15 12:13:21.073899 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:13:21.073886    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2409c9c4-0f91-41f5-b598-d9f342a626be-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-p7blm\" (UID: \"2409c9c4-0f91-41f5-b598-d9f342a626be\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-p7blm"
Sep 15 12:13:21.074077 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:13:21.073946    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2409c9c4-0f91-41f5-b598-d9f342a626be-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-p7blm\" (UID: \"2409c9c4-0f91-41f5-b598-d9f342a626be\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-p7blm"
Sep 15 12:13:21.174775 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:13:21.174742    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v45f\" (UniqueName: \"kubernetes.io/projected/2409c9c4-0f91-41f5-b598-d9f342a626be-kube-api-access-5v45f\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-p7blm\" (UID: \"2409c9c4-0f91-41f5-b598-d9f342a626be\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-p7blm"
Sep 15 12:13:21.174775 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:13:21.174781    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2409c9c4-0f91-41f5-b598-d9f342a626be-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-p7blm\" (UID: \"2409c9c4-0f91-41f5-b598-d9f342a626be\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-p7blm"
Sep 15 12:13:21.175011 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:13:21.174824    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2409c9c4-0f91-41f5-b598-d9f342a626be-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-p7blm\" (UID: \"2409c9c4-0f91-41f5-b598-d9f342a626be\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-p7blm"
Sep 15 12:13:21.175011 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:13:21.174846    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2409c9c4-0f91-41f5-b598-d9f342a626be-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-p7blm\" (UID: \"2409c9c4-0f91-41f5-b598-d9f342a626be\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-p7blm"
Sep 15 12:13:21.175011 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:13:21.174864    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2409c9c4-0f91-41f5-b598-d9f342a626be-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-p7blm\" (UID: \"2409c9c4-0f91-41f5-b598-d9f342a626be\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-p7blm"
Sep 15 12:13:21.175011 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:13:21.174900    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2409c9c4-0f91-41f5-b598-d9f342a626be-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-p7blm\" (UID: \"2409c9c4-0f91-41f5-b598-d9f342a626be\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-p7blm"
Sep 15 12:13:21.175345 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:13:21.175303    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2409c9c4-0f91-41f5-b598-d9f342a626be-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-p7blm\" (UID: \"2409c9c4-0f91-41f5-b598-d9f342a626be\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-p7blm"
Sep 15 12:13:21.175471 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:13:21.175348    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2409c9c4-0f91-41f5-b598-d9f342a626be-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-p7blm\" (UID: \"2409c9c4-0f91-41f5-b598-d9f342a626be\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-p7blm"
Sep 15 12:13:21.175471 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:13:21.175380    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2409c9c4-0f91-41f5-b598-d9f342a626be-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-p7blm\" (UID: \"2409c9c4-0f91-41f5-b598-d9f342a626be\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-p7blm"
Sep 15 12:13:21.177249 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:13:21.177221    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2409c9c4-0f91-41f5-b598-d9f342a626be-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-p7blm\" (UID: \"2409c9c4-0f91-41f5-b598-d9f342a626be\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-p7blm"
Sep 15 12:13:21.177366 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:13:21.177351    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2409c9c4-0f91-41f5-b598-d9f342a626be-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-p7blm\" (UID: \"2409c9c4-0f91-41f5-b598-d9f342a626be\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-p7blm"
Sep 15 12:13:21.184582 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:13:21.184556    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5v45f\" (UniqueName: \"kubernetes.io/projected/2409c9c4-0f91-41f5-b598-d9f342a626be-kube-api-access-5v45f\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-p7blm\" (UID: \"2409c9c4-0f91-41f5-b598-d9f342a626be\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-p7blm"
Sep 15 12:13:21.338357 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:13:21.338251    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-p7blm"
Sep 15 12:13:21.608583 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:13:21.608559    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-p7blm"]
Sep 15 12:13:21.610324 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:13:21.610294    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2409c9c4_0f91_41f5_b598_d9f342a626be.slice/crio-dcdf4123e73e1b77c067a681f79403b03c5e7ff3233d3cdd44f75edb48278de4 WatchSource:0}: Error finding container dcdf4123e73e1b77c067a681f79403b03c5e7ff3233d3cdd44f75edb48278de4: Status 404 returned error can't find the container with id dcdf4123e73e1b77c067a681f79403b03c5e7ff3233d3cdd44f75edb48278de4
Sep 15 12:13:21.611655 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:13:21.611634    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 12:13:21.842332 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:13:21.842289    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-p7blm" event={"ID":"2409c9c4-0f91-41f5-b598-d9f342a626be","Type":"ContainerStarted","Data":"dcdf4123e73e1b77c067a681f79403b03c5e7ff3233d3cdd44f75edb48278de4"}
Sep 15 12:13:21.973388 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:13:21.973351    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5gkxs"]
Sep 15 12:13:22.849544 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:13:22.849501    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-p7blm" event={"ID":"2409c9c4-0f91-41f5-b598-d9f342a626be","Type":"ContainerStarted","Data":"ab9740cac9887b7ba897ea9390b5f91aa209e3d679eedb20923bc99fce6bc44a"}
Sep 15 12:13:22.871272 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:13:22.871217    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-p7blm" podStartSLOduration=0.83144994 podStartE2EDuration="1.871200079s" totalImagesPullingTime="1.039750139s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:13:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:13:22.868183189 +0000 UTC m=+659.659105734" watchObservedRunningTime="2026-09-15 12:13:22.871200079 +0000 UTC m=+659.662122632"
Sep 15 12:13:30.456493 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:13:30.456450    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5gkxs"]
Sep 15 12:13:31.339020 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:13:31.338977    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-p7blm"
Sep 15 12:13:31.339220 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:13:31.339048    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-p7blm"
Sep 15 12:13:31.353227 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:13:31.353195    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-p7blm"
Sep 15 12:13:31.903825 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:13:31.903792    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-p7blm"
Sep 15 12:13:42.060779 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:13:42.060738    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5gkxs"]
Sep 15 12:13:55.806224 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:13:55.806182    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-wv5fb"]
Sep 15 12:13:55.813599 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:13:55.813559    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wv5fb"
Sep 15 12:13:55.816792 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:13:55.816764    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 15 12:13:55.827502 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:13:55.827474    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-wv5fb"]
Sep 15 12:13:55.968537 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:13:55.968500    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6fb1d655-76fb-4a3c-98b0-3ea71075915d-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wv5fb\" (UID: \"6fb1d655-76fb-4a3c-98b0-3ea71075915d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wv5fb"
Sep 15 12:13:55.968717 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:13:55.968543    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgwwt\" (UniqueName: \"kubernetes.io/projected/6fb1d655-76fb-4a3c-98b0-3ea71075915d-kube-api-access-wgwwt\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wv5fb\" (UID: \"6fb1d655-76fb-4a3c-98b0-3ea71075915d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wv5fb"
Sep 15 12:13:55.968717 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:13:55.968585    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6fb1d655-76fb-4a3c-98b0-3ea71075915d-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wv5fb\" (UID: \"6fb1d655-76fb-4a3c-98b0-3ea71075915d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wv5fb"
Sep 15 12:13:55.968717 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:13:55.968604    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6fb1d655-76fb-4a3c-98b0-3ea71075915d-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wv5fb\" (UID: \"6fb1d655-76fb-4a3c-98b0-3ea71075915d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wv5fb"
Sep 15 12:13:55.968717 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:13:55.968627    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6fb1d655-76fb-4a3c-98b0-3ea71075915d-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wv5fb\" (UID: \"6fb1d655-76fb-4a3c-98b0-3ea71075915d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wv5fb"
Sep 15 12:13:55.968717 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:13:55.968662    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fb1d655-76fb-4a3c-98b0-3ea71075915d-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wv5fb\" (UID: \"6fb1d655-76fb-4a3c-98b0-3ea71075915d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wv5fb"
Sep 15 12:13:56.069231 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:13:56.069146    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fb1d655-76fb-4a3c-98b0-3ea71075915d-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wv5fb\" (UID: \"6fb1d655-76fb-4a3c-98b0-3ea71075915d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wv5fb"
Sep 15 12:13:56.069231 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:13:56.069214    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6fb1d655-76fb-4a3c-98b0-3ea71075915d-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wv5fb\" (UID: \"6fb1d655-76fb-4a3c-98b0-3ea71075915d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wv5fb"
Sep 15 12:13:56.069538 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:13:56.069240    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgwwt\" (UniqueName: \"kubernetes.io/projected/6fb1d655-76fb-4a3c-98b0-3ea71075915d-kube-api-access-wgwwt\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wv5fb\" (UID: \"6fb1d655-76fb-4a3c-98b0-3ea71075915d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wv5fb"
Sep 15 12:13:56.069538 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:13:56.069388    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6fb1d655-76fb-4a3c-98b0-3ea71075915d-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wv5fb\" (UID: \"6fb1d655-76fb-4a3c-98b0-3ea71075915d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wv5fb"
Sep 15 12:13:56.069538 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:13:56.069419    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6fb1d655-76fb-4a3c-98b0-3ea71075915d-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wv5fb\" (UID: \"6fb1d655-76fb-4a3c-98b0-3ea71075915d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wv5fb"
Sep 15 12:13:56.069538 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:13:56.069442    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6fb1d655-76fb-4a3c-98b0-3ea71075915d-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wv5fb\" (UID: \"6fb1d655-76fb-4a3c-98b0-3ea71075915d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wv5fb"
Sep 15 12:13:56.069743 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:13:56.069619    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6fb1d655-76fb-4a3c-98b0-3ea71075915d-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wv5fb\" (UID: \"6fb1d655-76fb-4a3c-98b0-3ea71075915d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wv5fb"
Sep 15 12:13:56.069786 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:13:56.069771    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6fb1d655-76fb-4a3c-98b0-3ea71075915d-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wv5fb\" (UID: \"6fb1d655-76fb-4a3c-98b0-3ea71075915d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wv5fb"
Sep 15 12:13:56.069821 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:13:56.069784    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6fb1d655-76fb-4a3c-98b0-3ea71075915d-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wv5fb\" (UID: \"6fb1d655-76fb-4a3c-98b0-3ea71075915d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wv5fb"
Sep 15 12:13:56.071619 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:13:56.071595    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6fb1d655-76fb-4a3c-98b0-3ea71075915d-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wv5fb\" (UID: \"6fb1d655-76fb-4a3c-98b0-3ea71075915d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wv5fb"
Sep 15 12:13:56.071768 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:13:56.071743    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fb1d655-76fb-4a3c-98b0-3ea71075915d-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wv5fb\" (UID: \"6fb1d655-76fb-4a3c-98b0-3ea71075915d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wv5fb"
Sep 15 12:13:56.078821 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:13:56.078796    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgwwt\" (UniqueName: \"kubernetes.io/projected/6fb1d655-76fb-4a3c-98b0-3ea71075915d-kube-api-access-wgwwt\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wv5fb\" (UID: \"6fb1d655-76fb-4a3c-98b0-3ea71075915d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wv5fb"
Sep 15 12:13:56.126161 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:13:56.126109    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wv5fb"
Sep 15 12:13:56.401754 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:13:56.401727    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-wv5fb"]
Sep 15 12:13:56.403584 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:13:56.403555    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fb1d655_76fb_4a3c_98b0_3ea71075915d.slice/crio-6738f10266863b4fe233f3224b755732621d33443491b6ac984f42885d545343 WatchSource:0}: Error finding container 6738f10266863b4fe233f3224b755732621d33443491b6ac984f42885d545343: Status 404 returned error can't find the container with id 6738f10266863b4fe233f3224b755732621d33443491b6ac984f42885d545343
Sep 15 12:13:57.014077 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:13:57.013983    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wv5fb" event={"ID":"6fb1d655-76fb-4a3c-98b0-3ea71075915d","Type":"ContainerStarted","Data":"fa62030f1228bbcee19e56bfe7a9a463fe4faf94e8bbfe0821bab0f956b0dd28"}
Sep 15 12:13:57.014077 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:13:57.014039    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wv5fb" event={"ID":"6fb1d655-76fb-4a3c-98b0-3ea71075915d","Type":"ContainerStarted","Data":"6738f10266863b4fe233f3224b755732621d33443491b6ac984f42885d545343"}
Sep 15 12:13:57.035180 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:13:57.035123    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wv5fb" podStartSLOduration=1.749545358 podStartE2EDuration="2.035106431s" totalImagesPullingTime="285.561073ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:13:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:13:57.032697132 +0000 UTC m=+693.823619656" watchObservedRunningTime="2026-09-15 12:13:57.035106431 +0000 UTC m=+693.826028941"
Sep 15 12:13:57.660811 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:13:57.660766    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5gkxs"]
Sep 15 12:14:04.620327 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:14:04.620290    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-n7tsf"]
Sep 15 12:14:04.623781 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:14:04.623760    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-n7tsf"
Sep 15 12:14:04.626881 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:14:04.626864    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 15 12:14:04.641982 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:14:04.641947    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-n7tsf"]
Sep 15 12:14:04.649005 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:14:04.648975    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab42393b-e5a0-466c-8019-068e769519c0-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-n7tsf\" (UID: \"ab42393b-e5a0-466c-8019-068e769519c0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-n7tsf"
Sep 15 12:14:04.649178 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:14:04.649051    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab42393b-e5a0-466c-8019-068e769519c0-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-n7tsf\" (UID: \"ab42393b-e5a0-466c-8019-068e769519c0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-n7tsf"
Sep 15 12:14:04.649178 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:14:04.649087    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab42393b-e5a0-466c-8019-068e769519c0-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-n7tsf\" (UID: \"ab42393b-e5a0-466c-8019-068e769519c0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-n7tsf"
Sep 15 12:14:04.649178 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:14:04.649114    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab42393b-e5a0-466c-8019-068e769519c0-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-n7tsf\" (UID: \"ab42393b-e5a0-466c-8019-068e769519c0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-n7tsf"
Sep 15 12:14:04.649178 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:14:04.649140    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9s55\" (UniqueName: \"kubernetes.io/projected/ab42393b-e5a0-466c-8019-068e769519c0-kube-api-access-x9s55\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-n7tsf\" (UID: \"ab42393b-e5a0-466c-8019-068e769519c0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-n7tsf"
Sep 15 12:14:04.649178 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:14:04.649170    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab42393b-e5a0-466c-8019-068e769519c0-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-n7tsf\" (UID: \"ab42393b-e5a0-466c-8019-068e769519c0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-n7tsf"
Sep 15 12:14:04.750450 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:14:04.750408    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab42393b-e5a0-466c-8019-068e769519c0-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-n7tsf\" (UID: \"ab42393b-e5a0-466c-8019-068e769519c0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-n7tsf"
Sep 15 12:14:04.750450 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:14:04.750451    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab42393b-e5a0-466c-8019-068e769519c0-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-n7tsf\" (UID: \"ab42393b-e5a0-466c-8019-068e769519c0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-n7tsf"
Sep 15 12:14:04.750691 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:14:04.750555    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab42393b-e5a0-466c-8019-068e769519c0-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-n7tsf\" (UID: \"ab42393b-e5a0-466c-8019-068e769519c0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-n7tsf"
Sep 15 12:14:04.750691 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:14:04.750584    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9s55\" (UniqueName: \"kubernetes.io/projected/ab42393b-e5a0-466c-8019-068e769519c0-kube-api-access-x9s55\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-n7tsf\" (UID: \"ab42393b-e5a0-466c-8019-068e769519c0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-n7tsf"
Sep 15 12:14:04.750691 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:14:04.750625    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab42393b-e5a0-466c-8019-068e769519c0-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-n7tsf\" (UID: \"ab42393b-e5a0-466c-8019-068e769519c0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-n7tsf"
Sep 15 12:14:04.750691 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:14:04.750660    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab42393b-e5a0-466c-8019-068e769519c0-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-n7tsf\" (UID: \"ab42393b-e5a0-466c-8019-068e769519c0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-n7tsf"
Sep 15 12:14:04.750940 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:14:04.750917    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab42393b-e5a0-466c-8019-068e769519c0-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-n7tsf\" (UID: \"ab42393b-e5a0-466c-8019-068e769519c0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-n7tsf"
Sep 15 12:14:04.751039 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:14:04.751000    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab42393b-e5a0-466c-8019-068e769519c0-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-n7tsf\" (UID: \"ab42393b-e5a0-466c-8019-068e769519c0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-n7tsf"
Sep 15 12:14:04.751100 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:14:04.751060    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab42393b-e5a0-466c-8019-068e769519c0-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-n7tsf\" (UID: \"ab42393b-e5a0-466c-8019-068e769519c0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-n7tsf"
Sep 15 12:14:04.752729 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:14:04.752708    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab42393b-e5a0-466c-8019-068e769519c0-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-n7tsf\" (UID: \"ab42393b-e5a0-466c-8019-068e769519c0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-n7tsf"
Sep 15 12:14:04.753171 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:14:04.753146    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab42393b-e5a0-466c-8019-068e769519c0-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-n7tsf\" (UID: \"ab42393b-e5a0-466c-8019-068e769519c0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-n7tsf"
Sep 15 12:14:04.759113 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:14:04.759087    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9s55\" (UniqueName: \"kubernetes.io/projected/ab42393b-e5a0-466c-8019-068e769519c0-kube-api-access-x9s55\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-n7tsf\" (UID: \"ab42393b-e5a0-466c-8019-068e769519c0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-n7tsf"
Sep 15 12:14:04.933588 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:14:04.933548    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-n7tsf"
Sep 15 12:14:05.214067 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:14:05.214036    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-n7tsf"]
Sep 15 12:14:05.216363 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:14:05.216332    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podab42393b_e5a0_466c_8019_068e769519c0.slice/crio-80f24a8583a3df54a6a5c35e9b7a43226c11461163a72ace9a83c395c94accac WatchSource:0}: Error finding container 80f24a8583a3df54a6a5c35e9b7a43226c11461163a72ace9a83c395c94accac: Status 404 returned error can't find the container with id 80f24a8583a3df54a6a5c35e9b7a43226c11461163a72ace9a83c395c94accac
Sep 15 12:14:06.059334 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:14:06.059291    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-n7tsf" event={"ID":"ab42393b-e5a0-466c-8019-068e769519c0","Type":"ContainerStarted","Data":"bff2822a366be658f3df04d3471b7f0b3abe4131ad8c8cb49f1d84730d307214"}
Sep 15 12:14:06.059334 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:14:06.059340    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-n7tsf" event={"ID":"ab42393b-e5a0-466c-8019-068e769519c0","Type":"ContainerStarted","Data":"80f24a8583a3df54a6a5c35e9b7a43226c11461163a72ace9a83c395c94accac"}
Sep 15 12:14:06.079020 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:14:06.078960    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-n7tsf" podStartSLOduration=1.865283013 podStartE2EDuration="2.07891354s" totalImagesPullingTime="213.630527ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:14:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:14:06.077839372 +0000 UTC m=+702.868761883" watchObservedRunningTime="2026-09-15 12:14:06.07891354 +0000 UTC m=+702.869836077"
Sep 15 12:14:06.127222 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:14:06.127176    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wv5fb"
Sep 15 12:14:06.127416 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:14:06.127243    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wv5fb"
Sep 15 12:14:06.139897 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:14:06.139875    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wv5fb"
Sep 15 12:14:06.769022 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:14:06.768986    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5gkxs"]
Sep 15 12:14:07.076173 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:14:07.076086    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wv5fb"
Sep 15 12:14:14.934627 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:14:14.934582    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-n7tsf"
Sep 15 12:14:14.934627 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:14:14.934637    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-n7tsf"
Sep 15 12:14:14.947219 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:14:14.947190    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-n7tsf"
Sep 15 12:14:15.114956 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:14:15.114923    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-n7tsf"
Sep 15 12:14:58.381015 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:14:58.380978    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-55d8c6c4b9-6bw79"]
Sep 15 12:14:58.384308 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:14:58.384285    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-55d8c6c4b9-6bw79"
Sep 15 12:14:58.388447 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:14:58.388427    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-k4g6t\""
Sep 15 12:14:58.388557 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:14:58.388481    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 15 12:14:58.394239 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:14:58.394213    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-55d8c6c4b9-6bw79"]
Sep 15 12:14:58.413164 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:14:58.413132    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4a0ae0e8-2ce9-4a44-a6d3-5225f6b920f4-tls-cert\") pod \"authorino-55d8c6c4b9-6bw79\" (UID: \"4a0ae0e8-2ce9-4a44-a6d3-5225f6b920f4\") " pod="kuadrant-system/authorino-55d8c6c4b9-6bw79"
Sep 15 12:14:58.413339 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:14:58.413236    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79974\" (UniqueName: \"kubernetes.io/projected/4a0ae0e8-2ce9-4a44-a6d3-5225f6b920f4-kube-api-access-79974\") pod \"authorino-55d8c6c4b9-6bw79\" (UID: \"4a0ae0e8-2ce9-4a44-a6d3-5225f6b920f4\") " pod="kuadrant-system/authorino-55d8c6c4b9-6bw79"
Sep 15 12:14:58.514533 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:14:58.514499    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79974\" (UniqueName: \"kubernetes.io/projected/4a0ae0e8-2ce9-4a44-a6d3-5225f6b920f4-kube-api-access-79974\") pod \"authorino-55d8c6c4b9-6bw79\" (UID: \"4a0ae0e8-2ce9-4a44-a6d3-5225f6b920f4\") " pod="kuadrant-system/authorino-55d8c6c4b9-6bw79"
Sep 15 12:14:58.514533 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:14:58.514533    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4a0ae0e8-2ce9-4a44-a6d3-5225f6b920f4-tls-cert\") pod \"authorino-55d8c6c4b9-6bw79\" (UID: \"4a0ae0e8-2ce9-4a44-a6d3-5225f6b920f4\") " pod="kuadrant-system/authorino-55d8c6c4b9-6bw79"
Sep 15 12:14:58.517054 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:14:58.517019    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4a0ae0e8-2ce9-4a44-a6d3-5225f6b920f4-tls-cert\") pod \"authorino-55d8c6c4b9-6bw79\" (UID: \"4a0ae0e8-2ce9-4a44-a6d3-5225f6b920f4\") " pod="kuadrant-system/authorino-55d8c6c4b9-6bw79"
Sep 15 12:14:58.528466 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:14:58.528438    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-79974\" (UniqueName: \"kubernetes.io/projected/4a0ae0e8-2ce9-4a44-a6d3-5225f6b920f4-kube-api-access-79974\") pod \"authorino-55d8c6c4b9-6bw79\" (UID: \"4a0ae0e8-2ce9-4a44-a6d3-5225f6b920f4\") " pod="kuadrant-system/authorino-55d8c6c4b9-6bw79"
Sep 15 12:14:58.693877 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:14:58.693841    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-55d8c6c4b9-6bw79"
Sep 15 12:14:58.968402 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:14:58.968374    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-55d8c6c4b9-6bw79"]
Sep 15 12:14:58.970381 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:14:58.970349    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4a0ae0e8_2ce9_4a44_a6d3_5225f6b920f4.slice/crio-a8a5d7290766e19e60a9ccc3a96ad14f6ab4fefa65e73ce87dd2173a5cab328a WatchSource:0}: Error finding container a8a5d7290766e19e60a9ccc3a96ad14f6ab4fefa65e73ce87dd2173a5cab328a: Status 404 returned error can't find the container with id a8a5d7290766e19e60a9ccc3a96ad14f6ab4fefa65e73ce87dd2173a5cab328a
Sep 15 12:14:59.322867 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:14:59.322774    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-55d8c6c4b9-6bw79" event={"ID":"4a0ae0e8-2ce9-4a44-a6d3-5225f6b920f4","Type":"ContainerStarted","Data":"a8a5d7290766e19e60a9ccc3a96ad14f6ab4fefa65e73ce87dd2173a5cab328a"}
Sep 15 12:15:00.329936 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:00.329894    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-55d8c6c4b9-6bw79" event={"ID":"4a0ae0e8-2ce9-4a44-a6d3-5225f6b920f4","Type":"ContainerStarted","Data":"832e921fa35dd0ec6f10f34f82cf07c44f0b68fdfe1d689014d67efc60206e5d"}
Sep 15 12:15:00.354651 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:00.354590    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-55d8c6c4b9-6bw79" podStartSLOduration=1.8313847 podStartE2EDuration="2.35456859s" totalImagesPullingTime="523.18389ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:14:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:15:00.354223699 +0000 UTC m=+757.145146213" watchObservedRunningTime="2026-09-15 12:15:00.35456859 +0000 UTC m=+757.145491102"
Sep 15 12:15:23.075537 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:23.075500    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5gkxs"]
Sep 15 12:15:26.529372 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:26.529337    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7950-8e1b3f"]
Sep 15 12:15:26.532789 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:26.532772    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7950-8e1b3f"
Sep 15 12:15:26.536924 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:26.536903    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-q76n5\""
Sep 15 12:15:26.549534 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:26.549496    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7950-8e1b3f"]
Sep 15 12:15:26.659720 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:26.659679    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7wdj\" (UniqueName: \"kubernetes.io/projected/fc1027b6-66a3-4c38-94cd-dd6f82528f24-kube-api-access-p7wdj\") pod \"test-curl-7950-8e1b3f\" (UID: \"fc1027b6-66a3-4c38-94cd-dd6f82528f24\") " pod="opendatahub/test-curl-7950-8e1b3f"
Sep 15 12:15:26.760643 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:26.760592    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7wdj\" (UniqueName: \"kubernetes.io/projected/fc1027b6-66a3-4c38-94cd-dd6f82528f24-kube-api-access-p7wdj\") pod \"test-curl-7950-8e1b3f\" (UID: \"fc1027b6-66a3-4c38-94cd-dd6f82528f24\") " pod="opendatahub/test-curl-7950-8e1b3f"
Sep 15 12:15:26.774569 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:26.774543    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7wdj\" (UniqueName: \"kubernetes.io/projected/fc1027b6-66a3-4c38-94cd-dd6f82528f24-kube-api-access-p7wdj\") pod \"test-curl-7950-8e1b3f\" (UID: \"fc1027b6-66a3-4c38-94cd-dd6f82528f24\") " pod="opendatahub/test-curl-7950-8e1b3f"
Sep 15 12:15:26.842506 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:26.842415    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7950-8e1b3f"
Sep 15 12:15:27.113546 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:27.113521    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7950-8e1b3f"]
Sep 15 12:15:27.115248 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:15:27.115216    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfc1027b6_66a3_4c38_94cd_dd6f82528f24.slice/crio-05e174620c33080e63a5726b7f1c5552bd995093adc8785835498e61945998f9 WatchSource:0}: Error finding container 05e174620c33080e63a5726b7f1c5552bd995093adc8785835498e61945998f9: Status 404 returned error can't find the container with id 05e174620c33080e63a5726b7f1c5552bd995093adc8785835498e61945998f9
Sep 15 12:15:27.471577 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:27.471536    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7950-8e1b3f" event={"ID":"fc1027b6-66a3-4c38-94cd-dd6f82528f24","Type":"ContainerStarted","Data":"05e174620c33080e63a5726b7f1c5552bd995093adc8785835498e61945998f9"}
Sep 15 12:15:28.875426 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:28.875396    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-7d7eff3e-openshift-default-86654d9866-b5rpz"]
Sep 15 12:15:28.879958 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:28.879934    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7d7eff3e-openshift-default-86654d9866-b5rpz"
Sep 15 12:15:28.883315 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:28.883295    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 15 12:15:28.883315 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:28.883303    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-7d7eff3e-openshift-default-dockercfg-svv8r\""
Sep 15 12:15:28.895972 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:28.895947    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-7d7eff3e-openshift-default-86654d9866-b5rpz"]
Sep 15 12:15:28.977967 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:28.977928    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c776db8-f0ec-426e-abb7-09183b6e3c6b-istio-data\") pod \"e2e-ait-7d7eff3e-openshift-default-86654d9866-b5rpz\" (UID: \"6c776db8-f0ec-426e-abb7-09183b6e3c6b\") " pod="openshift-ingress/e2e-ait-7d7eff3e-openshift-default-86654d9866-b5rpz"
Sep 15 12:15:28.977967 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:28.977969    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c776db8-f0ec-426e-abb7-09183b6e3c6b-istio-envoy\") pod \"e2e-ait-7d7eff3e-openshift-default-86654d9866-b5rpz\" (UID: \"6c776db8-f0ec-426e-abb7-09183b6e3c6b\") " pod="openshift-ingress/e2e-ait-7d7eff3e-openshift-default-86654d9866-b5rpz"
Sep 15 12:15:28.978251 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:28.978000    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c776db8-f0ec-426e-abb7-09183b6e3c6b-istio-token\") pod \"e2e-ait-7d7eff3e-openshift-default-86654d9866-b5rpz\" (UID: \"6c776db8-f0ec-426e-abb7-09183b6e3c6b\") " pod="openshift-ingress/e2e-ait-7d7eff3e-openshift-default-86654d9866-b5rpz"
Sep 15 12:15:28.978251 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:28.978107    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c776db8-f0ec-426e-abb7-09183b6e3c6b-workload-socket\") pod \"e2e-ait-7d7eff3e-openshift-default-86654d9866-b5rpz\" (UID: \"6c776db8-f0ec-426e-abb7-09183b6e3c6b\") " pod="openshift-ingress/e2e-ait-7d7eff3e-openshift-default-86654d9866-b5rpz"
Sep 15 12:15:28.978251 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:28.978145    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c776db8-f0ec-426e-abb7-09183b6e3c6b-workload-certs\") pod \"e2e-ait-7d7eff3e-openshift-default-86654d9866-b5rpz\" (UID: \"6c776db8-f0ec-426e-abb7-09183b6e3c6b\") " pod="openshift-ingress/e2e-ait-7d7eff3e-openshift-default-86654d9866-b5rpz"
Sep 15 12:15:28.978251 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:28.978169    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c776db8-f0ec-426e-abb7-09183b6e3c6b-istiod-ca-cert\") pod \"e2e-ait-7d7eff3e-openshift-default-86654d9866-b5rpz\" (UID: \"6c776db8-f0ec-426e-abb7-09183b6e3c6b\") " pod="openshift-ingress/e2e-ait-7d7eff3e-openshift-default-86654d9866-b5rpz"
Sep 15 12:15:28.978251 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:28.978224    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m5bh\" (UniqueName: \"kubernetes.io/projected/6c776db8-f0ec-426e-abb7-09183b6e3c6b-kube-api-access-6m5bh\") pod \"e2e-ait-7d7eff3e-openshift-default-86654d9866-b5rpz\" (UID: \"6c776db8-f0ec-426e-abb7-09183b6e3c6b\") " pod="openshift-ingress/e2e-ait-7d7eff3e-openshift-default-86654d9866-b5rpz"
Sep 15 12:15:28.978481 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:28.978297    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c776db8-f0ec-426e-abb7-09183b6e3c6b-credential-socket\") pod \"e2e-ait-7d7eff3e-openshift-default-86654d9866-b5rpz\" (UID: \"6c776db8-f0ec-426e-abb7-09183b6e3c6b\") " pod="openshift-ingress/e2e-ait-7d7eff3e-openshift-default-86654d9866-b5rpz"
Sep 15 12:15:28.978481 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:28.978328    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c776db8-f0ec-426e-abb7-09183b6e3c6b-istio-podinfo\") pod \"e2e-ait-7d7eff3e-openshift-default-86654d9866-b5rpz\" (UID: \"6c776db8-f0ec-426e-abb7-09183b6e3c6b\") " pod="openshift-ingress/e2e-ait-7d7eff3e-openshift-default-86654d9866-b5rpz"
Sep 15 12:15:29.078875 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:29.078787    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c776db8-f0ec-426e-abb7-09183b6e3c6b-istio-envoy\") pod \"e2e-ait-7d7eff3e-openshift-default-86654d9866-b5rpz\" (UID: \"6c776db8-f0ec-426e-abb7-09183b6e3c6b\") " pod="openshift-ingress/e2e-ait-7d7eff3e-openshift-default-86654d9866-b5rpz"
Sep 15 12:15:29.078875 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:29.078840    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c776db8-f0ec-426e-abb7-09183b6e3c6b-istio-token\") pod \"e2e-ait-7d7eff3e-openshift-default-86654d9866-b5rpz\" (UID: \"6c776db8-f0ec-426e-abb7-09183b6e3c6b\") " pod="openshift-ingress/e2e-ait-7d7eff3e-openshift-default-86654d9866-b5rpz"
Sep 15 12:15:29.079103 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:29.078886    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c776db8-f0ec-426e-abb7-09183b6e3c6b-workload-socket\") pod \"e2e-ait-7d7eff3e-openshift-default-86654d9866-b5rpz\" (UID: \"6c776db8-f0ec-426e-abb7-09183b6e3c6b\") " pod="openshift-ingress/e2e-ait-7d7eff3e-openshift-default-86654d9866-b5rpz"
Sep 15 12:15:29.079103 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:29.078911    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c776db8-f0ec-426e-abb7-09183b6e3c6b-workload-certs\") pod \"e2e-ait-7d7eff3e-openshift-default-86654d9866-b5rpz\" (UID: \"6c776db8-f0ec-426e-abb7-09183b6e3c6b\") " pod="openshift-ingress/e2e-ait-7d7eff3e-openshift-default-86654d9866-b5rpz"
Sep 15 12:15:29.079103 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:29.078964    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c776db8-f0ec-426e-abb7-09183b6e3c6b-istiod-ca-cert\") pod \"e2e-ait-7d7eff3e-openshift-default-86654d9866-b5rpz\" (UID: \"6c776db8-f0ec-426e-abb7-09183b6e3c6b\") " pod="openshift-ingress/e2e-ait-7d7eff3e-openshift-default-86654d9866-b5rpz"
Sep 15 12:15:29.079103 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:29.079028    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m5bh\" (UniqueName: \"kubernetes.io/projected/6c776db8-f0ec-426e-abb7-09183b6e3c6b-kube-api-access-6m5bh\") pod \"e2e-ait-7d7eff3e-openshift-default-86654d9866-b5rpz\" (UID: \"6c776db8-f0ec-426e-abb7-09183b6e3c6b\") " pod="openshift-ingress/e2e-ait-7d7eff3e-openshift-default-86654d9866-b5rpz"
Sep 15 12:15:29.079103 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:29.079058    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c776db8-f0ec-426e-abb7-09183b6e3c6b-credential-socket\") pod \"e2e-ait-7d7eff3e-openshift-default-86654d9866-b5rpz\" (UID: \"6c776db8-f0ec-426e-abb7-09183b6e3c6b\") " pod="openshift-ingress/e2e-ait-7d7eff3e-openshift-default-86654d9866-b5rpz"
Sep 15 12:15:29.079103 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:29.079081    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c776db8-f0ec-426e-abb7-09183b6e3c6b-istio-podinfo\") pod \"e2e-ait-7d7eff3e-openshift-default-86654d9866-b5rpz\" (UID: \"6c776db8-f0ec-426e-abb7-09183b6e3c6b\") " pod="openshift-ingress/e2e-ait-7d7eff3e-openshift-default-86654d9866-b5rpz"
Sep 15 12:15:29.079426 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:29.079166    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c776db8-f0ec-426e-abb7-09183b6e3c6b-istio-data\") pod \"e2e-ait-7d7eff3e-openshift-default-86654d9866-b5rpz\" (UID: \"6c776db8-f0ec-426e-abb7-09183b6e3c6b\") " pod="openshift-ingress/e2e-ait-7d7eff3e-openshift-default-86654d9866-b5rpz"
Sep 15 12:15:29.079426 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:29.079313    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c776db8-f0ec-426e-abb7-09183b6e3c6b-workload-certs\") pod \"e2e-ait-7d7eff3e-openshift-default-86654d9866-b5rpz\" (UID: \"6c776db8-f0ec-426e-abb7-09183b6e3c6b\") " pod="openshift-ingress/e2e-ait-7d7eff3e-openshift-default-86654d9866-b5rpz"
Sep 15 12:15:29.079520 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:29.079415    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c776db8-f0ec-426e-abb7-09183b6e3c6b-credential-socket\") pod \"e2e-ait-7d7eff3e-openshift-default-86654d9866-b5rpz\" (UID: \"6c776db8-f0ec-426e-abb7-09183b6e3c6b\") " pod="openshift-ingress/e2e-ait-7d7eff3e-openshift-default-86654d9866-b5rpz"
Sep 15 12:15:29.079582 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:29.079521    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c776db8-f0ec-426e-abb7-09183b6e3c6b-istio-data\") pod \"e2e-ait-7d7eff3e-openshift-default-86654d9866-b5rpz\" (UID: \"6c776db8-f0ec-426e-abb7-09183b6e3c6b\") " pod="openshift-ingress/e2e-ait-7d7eff3e-openshift-default-86654d9866-b5rpz"
Sep 15 12:15:29.079683 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:29.079665    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c776db8-f0ec-426e-abb7-09183b6e3c6b-workload-socket\") pod \"e2e-ait-7d7eff3e-openshift-default-86654d9866-b5rpz\" (UID: \"6c776db8-f0ec-426e-abb7-09183b6e3c6b\") " pod="openshift-ingress/e2e-ait-7d7eff3e-openshift-default-86654d9866-b5rpz"
Sep 15 12:15:29.079759 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:29.079743    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c776db8-f0ec-426e-abb7-09183b6e3c6b-istiod-ca-cert\") pod \"e2e-ait-7d7eff3e-openshift-default-86654d9866-b5rpz\" (UID: \"6c776db8-f0ec-426e-abb7-09183b6e3c6b\") " pod="openshift-ingress/e2e-ait-7d7eff3e-openshift-default-86654d9866-b5rpz"
Sep 15 12:15:29.081030 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:29.081012    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c776db8-f0ec-426e-abb7-09183b6e3c6b-istio-envoy\") pod \"e2e-ait-7d7eff3e-openshift-default-86654d9866-b5rpz\" (UID: \"6c776db8-f0ec-426e-abb7-09183b6e3c6b\") " pod="openshift-ingress/e2e-ait-7d7eff3e-openshift-default-86654d9866-b5rpz"
Sep 15 12:15:29.081508 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:29.081492    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c776db8-f0ec-426e-abb7-09183b6e3c6b-istio-podinfo\") pod \"e2e-ait-7d7eff3e-openshift-default-86654d9866-b5rpz\" (UID: \"6c776db8-f0ec-426e-abb7-09183b6e3c6b\") " pod="openshift-ingress/e2e-ait-7d7eff3e-openshift-default-86654d9866-b5rpz"
Sep 15 12:15:29.087541 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:29.087522    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c776db8-f0ec-426e-abb7-09183b6e3c6b-istio-token\") pod \"e2e-ait-7d7eff3e-openshift-default-86654d9866-b5rpz\" (UID: \"6c776db8-f0ec-426e-abb7-09183b6e3c6b\") " pod="openshift-ingress/e2e-ait-7d7eff3e-openshift-default-86654d9866-b5rpz"
Sep 15 12:15:29.087678 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:29.087662    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m5bh\" (UniqueName: \"kubernetes.io/projected/6c776db8-f0ec-426e-abb7-09183b6e3c6b-kube-api-access-6m5bh\") pod \"e2e-ait-7d7eff3e-openshift-default-86654d9866-b5rpz\" (UID: \"6c776db8-f0ec-426e-abb7-09183b6e3c6b\") " pod="openshift-ingress/e2e-ait-7d7eff3e-openshift-default-86654d9866-b5rpz"
Sep 15 12:15:29.193084 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:29.193048    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7d7eff3e-openshift-default-86654d9866-b5rpz"
Sep 15 12:15:29.468907 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:29.468881    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-7d7eff3e-openshift-default-86654d9866-b5rpz"]
Sep 15 12:15:29.470020 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:15:29.469992    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c776db8_f0ec_426e_abb7_09183b6e3c6b.slice/crio-9e3c8ad0cd143dcd3f3cd99041136f5bc0f2d8d71f488f5d8811e5aa671f3550 WatchSource:0}: Error finding container 9e3c8ad0cd143dcd3f3cd99041136f5bc0f2d8d71f488f5d8811e5aa671f3550: Status 404 returned error can't find the container with id 9e3c8ad0cd143dcd3f3cd99041136f5bc0f2d8d71f488f5d8811e5aa671f3550
Sep 15 12:15:29.483730 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:29.483693    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7d7eff3e-openshift-default-86654d9866-b5rpz" event={"ID":"6c776db8-f0ec-426e-abb7-09183b6e3c6b","Type":"ContainerStarted","Data":"9e3c8ad0cd143dcd3f3cd99041136f5bc0f2d8d71f488f5d8811e5aa671f3550"}
Sep 15 12:15:29.485289 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:29.485248    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7950-8e1b3f" event={"ID":"fc1027b6-66a3-4c38-94cd-dd6f82528f24","Type":"ContainerStarted","Data":"3d4e0c0804e81e381c161e857dec3f9e21728ef5532fdf652df7390ff06e3f73"}
Sep 15 12:15:29.504566 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:29.504519    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7950-8e1b3f" podStartSLOduration=1.792256215 podStartE2EDuration="3.504502773s" totalImagesPullingTime="1.712246558s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:15:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:15:29.503966811 +0000 UTC m=+786.294889323" watchObservedRunningTime="2026-09-15 12:15:29.504502773 +0000 UTC m=+786.295425284"
Sep 15 12:15:30.154805 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:30.154760    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7950-8e1b3f"]
Sep 15 12:15:30.163518 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:30.163397    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7950-8e1b3f"]
Sep 15 12:15:30.490613 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:15:30.490533    2555 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7950-8e1b3f" container="test-curl-7950-8e1b3f"
Sep 15 12:15:30.987402 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:30.987362    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-0e10f0-openshift-default-5d9bcff6b6-86lrx"]
Sep 15 12:15:30.987919 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:30.987898    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc1027b6-66a3-4c38-94cd-dd6f82528f24" containerName="test-curl-7950-8e1b3f"
Sep 15 12:15:30.987919 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:30.987921    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc1027b6-66a3-4c38-94cd-dd6f82528f24" containerName="test-curl-7950-8e1b3f"
Sep 15 12:15:30.988061 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:30.988014    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc1027b6-66a3-4c38-94cd-dd6f82528f24" containerName="test-curl-7950-8e1b3f"
Sep 15 12:15:30.991359 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:30.991330    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-0e10f0-openshift-default-5d9bcff6b6-86lrx"
Sep 15 12:15:30.994581 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:30.994559    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-0e10f0-openshift-default-dockercfg-jw29x\""
Sep 15 12:15:31.006707 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:31.006652    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-0e10f0-openshift-default-5d9bcff6b6-86lrx"]
Sep 15 12:15:31.097649 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:31.097605    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc1353ca-8864-4cc7-93af-1a53f57fc7ab-istio-podinfo\") pod \"e2e-autores-0e10f0-openshift-default-5d9bcff6b6-86lrx\" (UID: \"bc1353ca-8864-4cc7-93af-1a53f57fc7ab\") " pod="openshift-ingress/e2e-autores-0e10f0-openshift-default-5d9bcff6b6-86lrx"
Sep 15 12:15:31.097847 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:31.097658    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc1353ca-8864-4cc7-93af-1a53f57fc7ab-workload-socket\") pod \"e2e-autores-0e10f0-openshift-default-5d9bcff6b6-86lrx\" (UID: \"bc1353ca-8864-4cc7-93af-1a53f57fc7ab\") " pod="openshift-ingress/e2e-autores-0e10f0-openshift-default-5d9bcff6b6-86lrx"
Sep 15 12:15:31.097847 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:31.097739    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc1353ca-8864-4cc7-93af-1a53f57fc7ab-istiod-ca-cert\") pod \"e2e-autores-0e10f0-openshift-default-5d9bcff6b6-86lrx\" (UID: \"bc1353ca-8864-4cc7-93af-1a53f57fc7ab\") " pod="openshift-ingress/e2e-autores-0e10f0-openshift-default-5d9bcff6b6-86lrx"
Sep 15 12:15:31.097847 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:31.097785    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc1353ca-8864-4cc7-93af-1a53f57fc7ab-workload-certs\") pod \"e2e-autores-0e10f0-openshift-default-5d9bcff6b6-86lrx\" (UID: \"bc1353ca-8864-4cc7-93af-1a53f57fc7ab\") " pod="openshift-ingress/e2e-autores-0e10f0-openshift-default-5d9bcff6b6-86lrx"
Sep 15 12:15:31.097847 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:31.097818    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc1353ca-8864-4cc7-93af-1a53f57fc7ab-istio-envoy\") pod \"e2e-autores-0e10f0-openshift-default-5d9bcff6b6-86lrx\" (UID: \"bc1353ca-8864-4cc7-93af-1a53f57fc7ab\") " pod="openshift-ingress/e2e-autores-0e10f0-openshift-default-5d9bcff6b6-86lrx"
Sep 15 12:15:31.097847 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:31.097839    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hb5l4\" (UniqueName: \"kubernetes.io/projected/bc1353ca-8864-4cc7-93af-1a53f57fc7ab-kube-api-access-hb5l4\") pod \"e2e-autores-0e10f0-openshift-default-5d9bcff6b6-86lrx\" (UID: \"bc1353ca-8864-4cc7-93af-1a53f57fc7ab\") " pod="openshift-ingress/e2e-autores-0e10f0-openshift-default-5d9bcff6b6-86lrx"
Sep 15 12:15:31.098123 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:31.097930    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc1353ca-8864-4cc7-93af-1a53f57fc7ab-credential-socket\") pod \"e2e-autores-0e10f0-openshift-default-5d9bcff6b6-86lrx\" (UID: \"bc1353ca-8864-4cc7-93af-1a53f57fc7ab\") " pod="openshift-ingress/e2e-autores-0e10f0-openshift-default-5d9bcff6b6-86lrx"
Sep 15 12:15:31.098123 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:31.097984    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc1353ca-8864-4cc7-93af-1a53f57fc7ab-istio-data\") pod \"e2e-autores-0e10f0-openshift-default-5d9bcff6b6-86lrx\" (UID: \"bc1353ca-8864-4cc7-93af-1a53f57fc7ab\") " pod="openshift-ingress/e2e-autores-0e10f0-openshift-default-5d9bcff6b6-86lrx"
Sep 15 12:15:31.098123 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:31.098011    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc1353ca-8864-4cc7-93af-1a53f57fc7ab-istio-token\") pod \"e2e-autores-0e10f0-openshift-default-5d9bcff6b6-86lrx\" (UID: \"bc1353ca-8864-4cc7-93af-1a53f57fc7ab\") " pod="openshift-ingress/e2e-autores-0e10f0-openshift-default-5d9bcff6b6-86lrx"
Sep 15 12:15:31.199315 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:31.199257    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc1353ca-8864-4cc7-93af-1a53f57fc7ab-istio-podinfo\") pod \"e2e-autores-0e10f0-openshift-default-5d9bcff6b6-86lrx\" (UID: \"bc1353ca-8864-4cc7-93af-1a53f57fc7ab\") " pod="openshift-ingress/e2e-autores-0e10f0-openshift-default-5d9bcff6b6-86lrx"
Sep 15 12:15:31.199793 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:31.199329    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc1353ca-8864-4cc7-93af-1a53f57fc7ab-workload-socket\") pod \"e2e-autores-0e10f0-openshift-default-5d9bcff6b6-86lrx\" (UID: \"bc1353ca-8864-4cc7-93af-1a53f57fc7ab\") " pod="openshift-ingress/e2e-autores-0e10f0-openshift-default-5d9bcff6b6-86lrx"
Sep 15 12:15:31.199793 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:31.199361    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc1353ca-8864-4cc7-93af-1a53f57fc7ab-istiod-ca-cert\") pod \"e2e-autores-0e10f0-openshift-default-5d9bcff6b6-86lrx\" (UID: \"bc1353ca-8864-4cc7-93af-1a53f57fc7ab\") " pod="openshift-ingress/e2e-autores-0e10f0-openshift-default-5d9bcff6b6-86lrx"
Sep 15 12:15:31.199793 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:31.199397    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc1353ca-8864-4cc7-93af-1a53f57fc7ab-workload-certs\") pod \"e2e-autores-0e10f0-openshift-default-5d9bcff6b6-86lrx\" (UID: \"bc1353ca-8864-4cc7-93af-1a53f57fc7ab\") " pod="openshift-ingress/e2e-autores-0e10f0-openshift-default-5d9bcff6b6-86lrx"
Sep 15 12:15:31.199793 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:31.199425    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc1353ca-8864-4cc7-93af-1a53f57fc7ab-istio-envoy\") pod \"e2e-autores-0e10f0-openshift-default-5d9bcff6b6-86lrx\" (UID: \"bc1353ca-8864-4cc7-93af-1a53f57fc7ab\") " pod="openshift-ingress/e2e-autores-0e10f0-openshift-default-5d9bcff6b6-86lrx"
Sep 15 12:15:31.199793 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:31.199450    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb5l4\" (UniqueName: \"kubernetes.io/projected/bc1353ca-8864-4cc7-93af-1a53f57fc7ab-kube-api-access-hb5l4\") pod \"e2e-autores-0e10f0-openshift-default-5d9bcff6b6-86lrx\" (UID: \"bc1353ca-8864-4cc7-93af-1a53f57fc7ab\") " pod="openshift-ingress/e2e-autores-0e10f0-openshift-default-5d9bcff6b6-86lrx"
Sep 15 12:15:31.199793 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:31.199494    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc1353ca-8864-4cc7-93af-1a53f57fc7ab-credential-socket\") pod \"e2e-autores-0e10f0-openshift-default-5d9bcff6b6-86lrx\" (UID: \"bc1353ca-8864-4cc7-93af-1a53f57fc7ab\") " pod="openshift-ingress/e2e-autores-0e10f0-openshift-default-5d9bcff6b6-86lrx"
Sep 15 12:15:31.199793 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:31.199535    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc1353ca-8864-4cc7-93af-1a53f57fc7ab-istio-data\") pod \"e2e-autores-0e10f0-openshift-default-5d9bcff6b6-86lrx\" (UID: \"bc1353ca-8864-4cc7-93af-1a53f57fc7ab\") " pod="openshift-ingress/e2e-autores-0e10f0-openshift-default-5d9bcff6b6-86lrx"
Sep 15 12:15:31.199793 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:31.199560    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc1353ca-8864-4cc7-93af-1a53f57fc7ab-istio-token\") pod \"e2e-autores-0e10f0-openshift-default-5d9bcff6b6-86lrx\" (UID: \"bc1353ca-8864-4cc7-93af-1a53f57fc7ab\") " pod="openshift-ingress/e2e-autores-0e10f0-openshift-default-5d9bcff6b6-86lrx"
Sep 15 12:15:31.201851 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:31.200454    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc1353ca-8864-4cc7-93af-1a53f57fc7ab-credential-socket\") pod \"e2e-autores-0e10f0-openshift-default-5d9bcff6b6-86lrx\" (UID: \"bc1353ca-8864-4cc7-93af-1a53f57fc7ab\") " pod="openshift-ingress/e2e-autores-0e10f0-openshift-default-5d9bcff6b6-86lrx"
Sep 15 12:15:31.201851 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:31.200752    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc1353ca-8864-4cc7-93af-1a53f57fc7ab-istio-data\") pod \"e2e-autores-0e10f0-openshift-default-5d9bcff6b6-86lrx\" (UID: \"bc1353ca-8864-4cc7-93af-1a53f57fc7ab\") " pod="openshift-ingress/e2e-autores-0e10f0-openshift-default-5d9bcff6b6-86lrx"
Sep 15 12:15:31.201851 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:31.200815    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc1353ca-8864-4cc7-93af-1a53f57fc7ab-workload-certs\") pod \"e2e-autores-0e10f0-openshift-default-5d9bcff6b6-86lrx\" (UID: \"bc1353ca-8864-4cc7-93af-1a53f57fc7ab\") " pod="openshift-ingress/e2e-autores-0e10f0-openshift-default-5d9bcff6b6-86lrx"
Sep 15 12:15:31.201851 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:31.200962    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc1353ca-8864-4cc7-93af-1a53f57fc7ab-istiod-ca-cert\") pod \"e2e-autores-0e10f0-openshift-default-5d9bcff6b6-86lrx\" (UID: \"bc1353ca-8864-4cc7-93af-1a53f57fc7ab\") " pod="openshift-ingress/e2e-autores-0e10f0-openshift-default-5d9bcff6b6-86lrx"
Sep 15 12:15:31.201851 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:31.200998    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc1353ca-8864-4cc7-93af-1a53f57fc7ab-workload-socket\") pod \"e2e-autores-0e10f0-openshift-default-5d9bcff6b6-86lrx\" (UID: \"bc1353ca-8864-4cc7-93af-1a53f57fc7ab\") " pod="openshift-ingress/e2e-autores-0e10f0-openshift-default-5d9bcff6b6-86lrx"
Sep 15 12:15:31.203913 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:31.202505    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc1353ca-8864-4cc7-93af-1a53f57fc7ab-istio-podinfo\") pod \"e2e-autores-0e10f0-openshift-default-5d9bcff6b6-86lrx\" (UID: \"bc1353ca-8864-4cc7-93af-1a53f57fc7ab\") " pod="openshift-ingress/e2e-autores-0e10f0-openshift-default-5d9bcff6b6-86lrx"
Sep 15 12:15:31.203913 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:31.202982    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc1353ca-8864-4cc7-93af-1a53f57fc7ab-istio-envoy\") pod \"e2e-autores-0e10f0-openshift-default-5d9bcff6b6-86lrx\" (UID: \"bc1353ca-8864-4cc7-93af-1a53f57fc7ab\") " pod="openshift-ingress/e2e-autores-0e10f0-openshift-default-5d9bcff6b6-86lrx"
Sep 15 12:15:31.209647 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:31.209621    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hb5l4\" (UniqueName: \"kubernetes.io/projected/bc1353ca-8864-4cc7-93af-1a53f57fc7ab-kube-api-access-hb5l4\") pod \"e2e-autores-0e10f0-openshift-default-5d9bcff6b6-86lrx\" (UID: \"bc1353ca-8864-4cc7-93af-1a53f57fc7ab\") " pod="openshift-ingress/e2e-autores-0e10f0-openshift-default-5d9bcff6b6-86lrx"
Sep 15 12:15:31.210168 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:31.210143    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc1353ca-8864-4cc7-93af-1a53f57fc7ab-istio-token\") pod \"e2e-autores-0e10f0-openshift-default-5d9bcff6b6-86lrx\" (UID: \"bc1353ca-8864-4cc7-93af-1a53f57fc7ab\") " pod="openshift-ingress/e2e-autores-0e10f0-openshift-default-5d9bcff6b6-86lrx"
Sep 15 12:15:31.306323 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:31.306223    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-0e10f0-openshift-default-5d9bcff6b6-86lrx"
Sep 15 12:15:31.496780 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:31.496738    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7950-8e1b3f" podUID="fc1027b6-66a3-4c38-94cd-dd6f82528f24" containerName="test-curl-7950-8e1b3f" containerID="cri-o://3d4e0c0804e81e381c161e857dec3f9e21728ef5532fdf652df7390ff06e3f73" gracePeriod=2
Sep 15 12:15:31.499276 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:15:31.499227    2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7950-8e1b3f\" is forbidden: User \"system:node:ip-10-0-140-17.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-17.ec2.internal' and this object" podUID="fc1027b6-66a3-4c38-94cd-dd6f82528f24" pod="opendatahub/test-curl-7950-8e1b3f"
Sep 15 12:15:31.637812 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:31.637773    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-0e10f0-openshift-default-5d9bcff6b6-86lrx"]
Sep 15 12:15:32.467613 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:15:32.467576    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc1353ca_8864_4cc7_93af_1a53f57fc7ab.slice/crio-757482f65eed99af581adc479bb4858714083398038a930068e59a769cff0917 WatchSource:0}: Error finding container 757482f65eed99af581adc479bb4858714083398038a930068e59a769cff0917: Status 404 returned error can't find the container with id 757482f65eed99af581adc479bb4858714083398038a930068e59a769cff0917
Sep 15 12:15:32.503443 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:32.503407    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-0e10f0-openshift-default-5d9bcff6b6-86lrx" event={"ID":"bc1353ca-8864-4cc7-93af-1a53f57fc7ab","Type":"ContainerStarted","Data":"757482f65eed99af581adc479bb4858714083398038a930068e59a769cff0917"}
Sep 15 12:15:32.530251 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:32.530207    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:15:32.530377 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:32.530301    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:15:32.530377 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:32.530335    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:15:33.019684 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:33.019652    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:15:33.019762 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:33.019720    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:15:33.019762 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:33.019746    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:15:33.512038 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:33.512006    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-0e10f0-openshift-default-5d9bcff6b6-86lrx" event={"ID":"bc1353ca-8864-4cc7-93af-1a53f57fc7ab","Type":"ContainerStarted","Data":"f331574f5a752a05d4b75cb1d7b2e8e134a2f87fb2aa62f396cff6d6a628f27f"}
Sep 15 12:15:33.513824 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:33.513801    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7d7eff3e-openshift-default-86654d9866-b5rpz" event={"ID":"6c776db8-f0ec-426e-abb7-09183b6e3c6b","Type":"ContainerStarted","Data":"4e44fd0feb698a461ec57cdd668eadf61641008e9852bd456e10db7ca418b955"}
Sep 15 12:15:33.536705 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:33.536646    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-0e10f0-openshift-default-5d9bcff6b6-86lrx" podStartSLOduration=2.987046784 podStartE2EDuration="3.536621332s" totalImagesPullingTime="549.574548ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:15:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:15:33.533835134 +0000 UTC m=+790.324757680" watchObservedRunningTime="2026-09-15 12:15:33.536621332 +0000 UTC m=+790.327543845"
Sep 15 12:15:33.557277 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:33.557209    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-7d7eff3e-openshift-default-86654d9866-b5rpz" podStartSLOduration=2.499372906 podStartE2EDuration="5.55718987s" totalImagesPullingTime="3.057816964s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:15:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:15:33.555049032 +0000 UTC m=+790.345971578" watchObservedRunningTime="2026-09-15 12:15:33.55718987 +0000 UTC m=+790.348112450"
Sep 15 12:15:33.684909 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:33.684882    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7950-8e1b3f"
Sep 15 12:15:33.823221 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:33.823141    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p7wdj\" (UniqueName: \"kubernetes.io/projected/fc1027b6-66a3-4c38-94cd-dd6f82528f24-kube-api-access-p7wdj\") pod \"fc1027b6-66a3-4c38-94cd-dd6f82528f24\" (UID: \"fc1027b6-66a3-4c38-94cd-dd6f82528f24\") "
Sep 15 12:15:33.825351 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:33.825326    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc1027b6-66a3-4c38-94cd-dd6f82528f24-kube-api-access-p7wdj" (OuterVolumeSpecName: "kube-api-access-p7wdj") pod "fc1027b6-66a3-4c38-94cd-dd6f82528f24" (UID: "fc1027b6-66a3-4c38-94cd-dd6f82528f24"). InnerVolumeSpecName "kube-api-access-p7wdj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:15:33.924525 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:33.924487    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7wdj\" (UniqueName: \"kubernetes.io/projected/fc1027b6-66a3-4c38-94cd-dd6f82528f24-kube-api-access-p7wdj\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:15:34.193597 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:34.193530    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-7d7eff3e-openshift-default-86654d9866-b5rpz"
Sep 15 12:15:34.199334 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:34.199302    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-7d7eff3e-openshift-default-86654d9866-b5rpz"
Sep 15 12:15:34.306835 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:34.306792    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-0e10f0-openshift-default-5d9bcff6b6-86lrx"
Sep 15 12:15:34.312101 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:34.312070    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-0e10f0-openshift-default-5d9bcff6b6-86lrx"
Sep 15 12:15:34.520322 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:34.520217    2555 generic.go:356] "Generic (PLEG): container finished" podID="fc1027b6-66a3-4c38-94cd-dd6f82528f24" containerID="3d4e0c0804e81e381c161e857dec3f9e21728ef5532fdf652df7390ff06e3f73" exitCode=137
Sep 15 12:15:34.520322 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:34.520289    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7950-8e1b3f"
Sep 15 12:15:34.520322 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:34.520320    2555 scope.go:117] "RemoveContainer" containerID="3d4e0c0804e81e381c161e857dec3f9e21728ef5532fdf652df7390ff06e3f73"
Sep 15 12:15:34.521050 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:34.520964    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-0e10f0-openshift-default-5d9bcff6b6-86lrx"
Sep 15 12:15:34.521050 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:34.521015    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-7d7eff3e-openshift-default-86654d9866-b5rpz"
Sep 15 12:15:34.521828 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:34.521806    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-7d7eff3e-openshift-default-86654d9866-b5rpz"
Sep 15 12:15:34.521917 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:34.521884    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-0e10f0-openshift-default-5d9bcff6b6-86lrx"
Sep 15 12:15:34.529652 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:34.529634    2555 scope.go:117] "RemoveContainer" containerID="3d4e0c0804e81e381c161e857dec3f9e21728ef5532fdf652df7390ff06e3f73"
Sep 15 12:15:34.529947 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:15:34.529930    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d4e0c0804e81e381c161e857dec3f9e21728ef5532fdf652df7390ff06e3f73\": container with ID starting with 3d4e0c0804e81e381c161e857dec3f9e21728ef5532fdf652df7390ff06e3f73 not found: ID does not exist" containerID="3d4e0c0804e81e381c161e857dec3f9e21728ef5532fdf652df7390ff06e3f73"
Sep 15 12:15:34.529993 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:34.529956    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d4e0c0804e81e381c161e857dec3f9e21728ef5532fdf652df7390ff06e3f73"} err="failed to get container status \"3d4e0c0804e81e381c161e857dec3f9e21728ef5532fdf652df7390ff06e3f73\": rpc error: code = NotFound desc = could not find container \"3d4e0c0804e81e381c161e857dec3f9e21728ef5532fdf652df7390ff06e3f73\": container with ID starting with 3d4e0c0804e81e381c161e857dec3f9e21728ef5532fdf652df7390ff06e3f73 not found: ID does not exist"
Sep 15 12:15:35.817379 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:35.817344    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc1027b6-66a3-4c38-94cd-dd6f82528f24" path="/var/lib/kubelet/pods/fc1027b6-66a3-4c38-94cd-dd6f82528f24/volumes"
Sep 15 12:15:37.069412 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:37.069373    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5gkxs"]
Sep 15 12:15:39.193098 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:39.193058    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-0e10f0-859b7f7d9d-hxr92"]
Sep 15 12:15:39.198010 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:39.197987    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-0e10f0-859b7f7d9d-hxr92"
Sep 15 12:15:39.202154 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:39.202133    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-0e10f0\""
Sep 15 12:15:39.215506 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:39.215478    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-0e10f0-859b7f7d9d-hxr92"]
Sep 15 12:15:39.274489 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:39.274443    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hdcn\" (UniqueName: \"kubernetes.io/projected/398ac422-ed36-46db-bcdd-cf23226c2110-kube-api-access-4hdcn\") pod \"payload-pre-processing-e2e-autores-0e10f0-859b7f7d9d-hxr92\" (UID: \"398ac422-ed36-46db-bcdd-cf23226c2110\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-0e10f0-859b7f7d9d-hxr92"
Sep 15 12:15:39.274698 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:39.274520    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/398ac422-ed36-46db-bcdd-cf23226c2110-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-0e10f0-859b7f7d9d-hxr92\" (UID: \"398ac422-ed36-46db-bcdd-cf23226c2110\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-0e10f0-859b7f7d9d-hxr92"
Sep 15 12:15:39.375869 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:39.375829    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hdcn\" (UniqueName: \"kubernetes.io/projected/398ac422-ed36-46db-bcdd-cf23226c2110-kube-api-access-4hdcn\") pod \"payload-pre-processing-e2e-autores-0e10f0-859b7f7d9d-hxr92\" (UID: \"398ac422-ed36-46db-bcdd-cf23226c2110\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-0e10f0-859b7f7d9d-hxr92"
Sep 15 12:15:39.376048 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:39.375921    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/398ac422-ed36-46db-bcdd-cf23226c2110-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-0e10f0-859b7f7d9d-hxr92\" (UID: \"398ac422-ed36-46db-bcdd-cf23226c2110\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-0e10f0-859b7f7d9d-hxr92"
Sep 15 12:15:39.376725 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:39.376703    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/398ac422-ed36-46db-bcdd-cf23226c2110-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-0e10f0-859b7f7d9d-hxr92\" (UID: \"398ac422-ed36-46db-bcdd-cf23226c2110\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-0e10f0-859b7f7d9d-hxr92"
Sep 15 12:15:39.386468 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:39.386438    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hdcn\" (UniqueName: \"kubernetes.io/projected/398ac422-ed36-46db-bcdd-cf23226c2110-kube-api-access-4hdcn\") pod \"payload-pre-processing-e2e-autores-0e10f0-859b7f7d9d-hxr92\" (UID: \"398ac422-ed36-46db-bcdd-cf23226c2110\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-0e10f0-859b7f7d9d-hxr92"
Sep 15 12:15:39.507872 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:39.507783    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-0e10f0-859b7f7d9d-hxr92"
Sep 15 12:15:39.790657 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:39.790632    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-0e10f0-859b7f7d9d-hxr92"]
Sep 15 12:15:39.792731 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:15:39.792702    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod398ac422_ed36_46db_bcdd_cf23226c2110.slice/crio-a301251fe4ea01cfc8a18a5693214aa82b47c9c57558df7b1562a2d66a003643 WatchSource:0}: Error finding container a301251fe4ea01cfc8a18a5693214aa82b47c9c57558df7b1562a2d66a003643: Status 404 returned error can't find the container with id a301251fe4ea01cfc8a18a5693214aa82b47c9c57558df7b1562a2d66a003643
Sep 15 12:15:40.361373 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:40.361338    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7d7eff3e-594cd8dfdb-kqvk8"]
Sep 15 12:15:40.366039 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:40.366020    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7d7eff3e-594cd8dfdb-kqvk8"
Sep 15 12:15:40.368663 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:40.368633    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-7d7eff3e-dockercfg-z2tvt\""
Sep 15 12:15:40.369128 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:40.369111    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-7d7eff3e\""
Sep 15 12:15:40.374444 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:40.374159    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7d7eff3e-594cd8dfdb-kqvk8"]
Sep 15 12:15:40.485212 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:40.485173    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/202b8fe6-5531-49d5-a84d-9b9e910cef18-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-7d7eff3e-594cd8dfdb-kqvk8\" (UID: \"202b8fe6-5531-49d5-a84d-9b9e910cef18\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7d7eff3e-594cd8dfdb-kqvk8"
Sep 15 12:15:40.485462 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:40.485229    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntj7z\" (UniqueName: \"kubernetes.io/projected/202b8fe6-5531-49d5-a84d-9b9e910cef18-kube-api-access-ntj7z\") pod \"payload-pre-processing-e2e-ait-7d7eff3e-594cd8dfdb-kqvk8\" (UID: \"202b8fe6-5531-49d5-a84d-9b9e910cef18\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7d7eff3e-594cd8dfdb-kqvk8"
Sep 15 12:15:40.555782 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:40.555742    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-0e10f0-859b7f7d9d-hxr92" event={"ID":"398ac422-ed36-46db-bcdd-cf23226c2110","Type":"ContainerStarted","Data":"a301251fe4ea01cfc8a18a5693214aa82b47c9c57558df7b1562a2d66a003643"}
Sep 15 12:15:40.586445 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:40.586408    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntj7z\" (UniqueName: \"kubernetes.io/projected/202b8fe6-5531-49d5-a84d-9b9e910cef18-kube-api-access-ntj7z\") pod \"payload-pre-processing-e2e-ait-7d7eff3e-594cd8dfdb-kqvk8\" (UID: \"202b8fe6-5531-49d5-a84d-9b9e910cef18\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7d7eff3e-594cd8dfdb-kqvk8"
Sep 15 12:15:40.586659 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:40.586511    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/202b8fe6-5531-49d5-a84d-9b9e910cef18-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-7d7eff3e-594cd8dfdb-kqvk8\" (UID: \"202b8fe6-5531-49d5-a84d-9b9e910cef18\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7d7eff3e-594cd8dfdb-kqvk8"
Sep 15 12:15:40.587256 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:40.587235    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/202b8fe6-5531-49d5-a84d-9b9e910cef18-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-7d7eff3e-594cd8dfdb-kqvk8\" (UID: \"202b8fe6-5531-49d5-a84d-9b9e910cef18\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7d7eff3e-594cd8dfdb-kqvk8"
Sep 15 12:15:40.594755 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:40.594725    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntj7z\" (UniqueName: \"kubernetes.io/projected/202b8fe6-5531-49d5-a84d-9b9e910cef18-kube-api-access-ntj7z\") pod \"payload-pre-processing-e2e-ait-7d7eff3e-594cd8dfdb-kqvk8\" (UID: \"202b8fe6-5531-49d5-a84d-9b9e910cef18\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7d7eff3e-594cd8dfdb-kqvk8"
Sep 15 12:15:40.678496 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:40.678459    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7d7eff3e-594cd8dfdb-kqvk8"
Sep 15 12:15:40.961598 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:40.961567    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7d7eff3e-594cd8dfdb-kqvk8"]
Sep 15 12:15:41.185740 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:15:41.185706    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod202b8fe6_5531_49d5_a84d_9b9e910cef18.slice/crio-c656abd8ea6a55b20f12f9ed7a2ea6427f1802c0673d848a7554d966bdac353c WatchSource:0}: Error finding container c656abd8ea6a55b20f12f9ed7a2ea6427f1802c0673d848a7554d966bdac353c: Status 404 returned error can't find the container with id c656abd8ea6a55b20f12f9ed7a2ea6427f1802c0673d848a7554d966bdac353c
Sep 15 12:15:41.562556 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:41.562450    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-0e10f0-859b7f7d9d-hxr92" event={"ID":"398ac422-ed36-46db-bcdd-cf23226c2110","Type":"ContainerStarted","Data":"d34b9c4567f44b838df1bdb54d25a5f991714ac16f50bde089164939ec190548"}
Sep 15 12:15:41.562556 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:41.562524    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-0e10f0-859b7f7d9d-hxr92"
Sep 15 12:15:41.563857 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:41.563804    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7d7eff3e-594cd8dfdb-kqvk8" event={"ID":"202b8fe6-5531-49d5-a84d-9b9e910cef18","Type":"ContainerStarted","Data":"c656abd8ea6a55b20f12f9ed7a2ea6427f1802c0673d848a7554d966bdac353c"}
Sep 15 12:15:41.583486 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:41.583433    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-0e10f0-859b7f7d9d-hxr92" podStartSLOduration=1.146074183 podStartE2EDuration="2.583414334s" totalImagesPullingTime="1.437340151s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:15:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:15:41.581369462 +0000 UTC m=+798.372291988" watchObservedRunningTime="2026-09-15 12:15:41.583414334 +0000 UTC m=+798.374336879"
Sep 15 12:15:43.578850 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:43.578814    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7d7eff3e-594cd8dfdb-kqvk8" event={"ID":"202b8fe6-5531-49d5-a84d-9b9e910cef18","Type":"ContainerStarted","Data":"c270dc34805b2cff421235ffddb40a9eaa6d2b02895aa407e161deffab3eca25"}
Sep 15 12:15:43.579355 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:43.578999    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-7d7eff3e-594cd8dfdb-kqvk8"
Sep 15 12:15:43.603450 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:43.603400    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-7d7eff3e-594cd8dfdb-kqvk8" podStartSLOduration=2.199031351 podStartE2EDuration="3.603381655s" totalImagesPullingTime="1.404350304s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:15:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:15:43.601011466 +0000 UTC m=+800.391933989" watchObservedRunningTime="2026-09-15 12:15:43.603381655 +0000 UTC m=+800.394304165"
Sep 15 12:15:49.348371 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:49.348240    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-69d7ba-c88fbdb46-m92g7"]
Sep 15 12:15:49.352308 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:49.352281    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-69d7ba-c88fbdb46-m92g7"
Sep 15 12:15:49.355870 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:49.355843    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-69d7ba\""
Sep 15 12:15:49.355987 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:49.355910    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-69d7ba-dockercfg-4rwxh\""
Sep 15 12:15:49.363058 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:49.363033    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-69d7ba-c88fbdb46-m92g7"]
Sep 15 12:15:49.480292 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:49.479879    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27308750-531b-4905-9f23-bd738daec2d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-69d7ba-c88fbdb46-m92g7\" (UID: \"27308750-531b-4905-9f23-bd738daec2d0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-69d7ba-c88fbdb46-m92g7"
Sep 15 12:15:49.480292 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:49.480030    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5cgc\" (UniqueName: \"kubernetes.io/projected/27308750-531b-4905-9f23-bd738daec2d0-kube-api-access-s5cgc\") pod \"payload-pre-processing-e2e-inf-b-69d7ba-c88fbdb46-m92g7\" (UID: \"27308750-531b-4905-9f23-bd738daec2d0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-69d7ba-c88fbdb46-m92g7"
Sep 15 12:15:49.580753 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:49.580711    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27308750-531b-4905-9f23-bd738daec2d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-69d7ba-c88fbdb46-m92g7\" (UID: \"27308750-531b-4905-9f23-bd738daec2d0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-69d7ba-c88fbdb46-m92g7"
Sep 15 12:15:49.580966 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:49.580823    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5cgc\" (UniqueName: \"kubernetes.io/projected/27308750-531b-4905-9f23-bd738daec2d0-kube-api-access-s5cgc\") pod \"payload-pre-processing-e2e-inf-b-69d7ba-c88fbdb46-m92g7\" (UID: \"27308750-531b-4905-9f23-bd738daec2d0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-69d7ba-c88fbdb46-m92g7"
Sep 15 12:15:49.581450 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:49.581429    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27308750-531b-4905-9f23-bd738daec2d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-69d7ba-c88fbdb46-m92g7\" (UID: \"27308750-531b-4905-9f23-bd738daec2d0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-69d7ba-c88fbdb46-m92g7"
Sep 15 12:15:49.589649 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:49.589616    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5cgc\" (UniqueName: \"kubernetes.io/projected/27308750-531b-4905-9f23-bd738daec2d0-kube-api-access-s5cgc\") pod \"payload-pre-processing-e2e-inf-b-69d7ba-c88fbdb46-m92g7\" (UID: \"27308750-531b-4905-9f23-bd738daec2d0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-69d7ba-c88fbdb46-m92g7"
Sep 15 12:15:49.663179 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:49.663140    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-69d7ba-c88fbdb46-m92g7"
Sep 15 12:15:49.936702 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:49.936671    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-69d7ba-c88fbdb46-m92g7"]
Sep 15 12:15:49.938545 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:15:49.938502    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27308750_531b_4905_9f23_bd738daec2d0.slice/crio-4599467381f46217cecc13d225967fd489e2315f6f9332811cd66b6a8d8ddf0e WatchSource:0}: Error finding container 4599467381f46217cecc13d225967fd489e2315f6f9332811cd66b6a8d8ddf0e: Status 404 returned error can't find the container with id 4599467381f46217cecc13d225967fd489e2315f6f9332811cd66b6a8d8ddf0e
Sep 15 12:15:50.623746 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:50.623703    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-69d7ba-c88fbdb46-m92g7" event={"ID":"27308750-531b-4905-9f23-bd738daec2d0","Type":"ContainerStarted","Data":"8e697429fcaf852c8ccc14c4d15f15c7a7df8b5f7e054404313b90bf5e451b92"}
Sep 15 12:15:50.624166 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:50.623797    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-69d7ba-c88fbdb46-m92g7" event={"ID":"27308750-531b-4905-9f23-bd738daec2d0","Type":"ContainerStarted","Data":"4599467381f46217cecc13d225967fd489e2315f6f9332811cd66b6a8d8ddf0e"}
Sep 15 12:15:50.624166 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:50.623827    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-69d7ba-c88fbdb46-m92g7"
Sep 15 12:15:52.579499 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:52.579454    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-0e10f0-859b7f7d9d-hxr92"
Sep 15 12:15:52.602755 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:52.602680    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-69d7ba-c88fbdb46-m92g7" podStartSLOduration=3.602633844 podStartE2EDuration="3.602633844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:15:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:15:50.668299391 +0000 UTC m=+807.459221922" watchObservedRunningTime="2026-09-15 12:15:52.602633844 +0000 UTC m=+809.393556356"
Sep 15 12:15:54.595049 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:15:54.595014    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-7d7eff3e-594cd8dfdb-kqvk8"
Sep 15 12:16:00.397754 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:00.392517    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7d7eff3e-t894k"]
Sep 15 12:16:00.397754 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:00.397079    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7d7eff3e-t894k"
Sep 15 12:16:00.410068 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:00.399963    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tbvkt\""
Sep 15 12:16:00.410068 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:00.400146    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 12:16:00.410068 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:00.406445    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7d7eff3e-t894k"]
Sep 15 12:16:00.471111 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:00.471078    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5gkxs"]
Sep 15 12:16:00.500977 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:00.500915    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ffc0d2a-a645-467d-980e-bcc75b97d50a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7d7eff3e-t894k\" (UID: \"2ffc0d2a-a645-467d-980e-bcc75b97d50a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7d7eff3e-t894k"
Sep 15 12:16:00.602334 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:00.602230    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ffc0d2a-a645-467d-980e-bcc75b97d50a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7d7eff3e-t894k\" (UID: \"2ffc0d2a-a645-467d-980e-bcc75b97d50a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7d7eff3e-t894k"
Sep 15 12:16:00.603088 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:00.603049    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ffc0d2a-a645-467d-980e-bcc75b97d50a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7d7eff3e-t894k\" (UID: \"2ffc0d2a-a645-467d-980e-bcc75b97d50a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7d7eff3e-t894k"
Sep 15 12:16:00.698960 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:00.698865    2555 generic.go:356] "Generic (PLEG): container finished" podID="bc1353ca-8864-4cc7-93af-1a53f57fc7ab" containerID="f331574f5a752a05d4b75cb1d7b2e8e134a2f87fb2aa62f396cff6d6a628f27f" exitCode=0
Sep 15 12:16:00.699099 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:00.698944    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-0e10f0-openshift-default-5d9bcff6b6-86lrx" event={"ID":"bc1353ca-8864-4cc7-93af-1a53f57fc7ab","Type":"ContainerDied","Data":"f331574f5a752a05d4b75cb1d7b2e8e134a2f87fb2aa62f396cff6d6a628f27f"}
Sep 15 12:16:00.699505 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:00.699486    2555 scope.go:117] "RemoveContainer" containerID="f331574f5a752a05d4b75cb1d7b2e8e134a2f87fb2aa62f396cff6d6a628f27f"
Sep 15 12:16:00.700847 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:00.700819    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:16:00.700906 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:00.700888    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:16:00.700941 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:00.700919    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:16:00.712877 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:00.712849    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7d7eff3e-t894k"
Sep 15 12:16:01.012245 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:01.012215    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7d7eff3e-t894k"]
Sep 15 12:16:01.014848 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:16:01.014819    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ffc0d2a_a645_467d_980e_bcc75b97d50a.slice/crio-df4596d2196be6075e0f0d7a4b9ae16e16359963180d1d984e4eb7441e9bca84 WatchSource:0}: Error finding container df4596d2196be6075e0f0d7a4b9ae16e16359963180d1d984e4eb7441e9bca84: Status 404 returned error can't find the container with id df4596d2196be6075e0f0d7a4b9ae16e16359963180d1d984e4eb7441e9bca84
Sep 15 12:16:01.632874 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:01.632837    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-69d7ba-c88fbdb46-m92g7"
Sep 15 12:16:01.706045 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:01.706006    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7d7eff3e-t894k" event={"ID":"2ffc0d2a-a645-467d-980e-bcc75b97d50a","Type":"ContainerStarted","Data":"df4596d2196be6075e0f0d7a4b9ae16e16359963180d1d984e4eb7441e9bca84"}
Sep 15 12:16:01.708113 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:01.708086    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-0e10f0-openshift-default-5d9bcff6b6-86lrx" event={"ID":"bc1353ca-8864-4cc7-93af-1a53f57fc7ab","Type":"ContainerStarted","Data":"d321d233d3b3fd56cb7150af58f6ab96ff51c70b3df9b0086deff32420868855"}
Sep 15 12:16:01.709734 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:01.709710    2555 generic.go:356] "Generic (PLEG): container finished" podID="6c776db8-f0ec-426e-abb7-09183b6e3c6b" containerID="4e44fd0feb698a461ec57cdd668eadf61641008e9852bd456e10db7ca418b955" exitCode=0
Sep 15 12:16:01.709829 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:01.709745    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7d7eff3e-openshift-default-86654d9866-b5rpz" event={"ID":"6c776db8-f0ec-426e-abb7-09183b6e3c6b","Type":"ContainerDied","Data":"4e44fd0feb698a461ec57cdd668eadf61641008e9852bd456e10db7ca418b955"}
Sep 15 12:16:01.710064 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:01.710051    2555 scope.go:117] "RemoveContainer" containerID="4e44fd0feb698a461ec57cdd668eadf61641008e9852bd456e10db7ca418b955"
Sep 15 12:16:01.711101 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:01.711073    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:16:01.711201 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:01.711155    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:16:01.711278 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:01.711204    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:16:02.306830 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:02.306790    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-0e10f0-openshift-default-5d9bcff6b6-86lrx"
Sep 15 12:16:02.307966 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:02.307922    2555 patch_prober.go:28] interesting pod/e2e-autores-0e10f0-openshift-default-5d9bcff6b6-86lrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Sep 15 12:16:02.308104 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:02.307999    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-0e10f0-openshift-default-5d9bcff6b6-86lrx" podUID="bc1353ca-8864-4cc7-93af-1a53f57fc7ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Sep 15 12:16:02.718109 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:02.718066    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7d7eff3e-openshift-default-86654d9866-b5rpz" event={"ID":"6c776db8-f0ec-426e-abb7-09183b6e3c6b","Type":"ContainerStarted","Data":"621c63581e13fde66d336162aa53b797b94a475e6d816eab3ab44396161d9afb"}
Sep 15 12:16:03.194799 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:03.194756    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-7d7eff3e-openshift-default-86654d9866-b5rpz"
Sep 15 12:16:03.195026 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:03.194948    2555 patch_prober.go:28] interesting pod/e2e-ait-7d7eff3e-openshift-default-86654d9866-b5rpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Sep 15 12:16:03.195026 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:03.195012    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7d7eff3e-openshift-default-86654d9866-b5rpz" podUID="6c776db8-f0ec-426e-abb7-09183b6e3c6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Sep 15 12:16:03.307144 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:03.307103    2555 patch_prober.go:28] interesting pod/e2e-autores-0e10f0-openshift-default-5d9bcff6b6-86lrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Sep 15 12:16:03.307329 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:03.307171    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-0e10f0-openshift-default-5d9bcff6b6-86lrx" podUID="bc1353ca-8864-4cc7-93af-1a53f57fc7ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Sep 15 12:16:04.193684 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:04.193632    2555 patch_prober.go:28] interesting pod/e2e-ait-7d7eff3e-openshift-default-86654d9866-b5rpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Sep 15 12:16:04.194203 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:04.193703    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7d7eff3e-openshift-default-86654d9866-b5rpz" podUID="6c776db8-f0ec-426e-abb7-09183b6e3c6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Sep 15 12:16:04.307391 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:04.307358    2555 patch_prober.go:28] interesting pod/e2e-autores-0e10f0-openshift-default-5d9bcff6b6-86lrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Sep 15 12:16:04.307557 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:04.307415    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-0e10f0-openshift-default-5d9bcff6b6-86lrx" podUID="bc1353ca-8864-4cc7-93af-1a53f57fc7ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Sep 15 12:16:04.521483 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:04.521398    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-7d7eff3e-openshift-default-86654d9866-b5rpz"
Sep 15 12:16:04.521483 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:04.521469    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-0e10f0-openshift-default-5d9bcff6b6-86lrx"
Sep 15 12:16:04.733143 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:04.733112    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7d7eff3e-t894k" event={"ID":"2ffc0d2a-a645-467d-980e-bcc75b97d50a","Type":"ContainerStarted","Data":"2fc84f48edadb897c1c7971821d696f27dbec59923459fcbe8f612a443f29f25"}
Sep 15 12:16:05.194040 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:05.193989    2555 patch_prober.go:28] interesting pod/e2e-ait-7d7eff3e-openshift-default-86654d9866-b5rpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Sep 15 12:16:05.194610 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:05.194075    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7d7eff3e-openshift-default-86654d9866-b5rpz" podUID="6c776db8-f0ec-426e-abb7-09183b6e3c6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Sep 15 12:16:05.306708 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:05.306677    2555 patch_prober.go:28] interesting pod/e2e-autores-0e10f0-openshift-default-5d9bcff6b6-86lrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Sep 15 12:16:05.306937 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:05.306916    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-0e10f0-openshift-default-5d9bcff6b6-86lrx" podUID="bc1353ca-8864-4cc7-93af-1a53f57fc7ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Sep 15 12:16:05.739416 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:05.739382    2555 generic.go:356] "Generic (PLEG): container finished" podID="2ffc0d2a-a645-467d-980e-bcc75b97d50a" containerID="2fc84f48edadb897c1c7971821d696f27dbec59923459fcbe8f612a443f29f25" exitCode=0
Sep 15 12:16:05.739621 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:05.739470    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7d7eff3e-t894k" event={"ID":"2ffc0d2a-a645-467d-980e-bcc75b97d50a","Type":"ContainerDied","Data":"2fc84f48edadb897c1c7971821d696f27dbec59923459fcbe8f612a443f29f25"}
Sep 15 12:16:06.194437 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:06.194396    2555 patch_prober.go:28] interesting pod/e2e-ait-7d7eff3e-openshift-default-86654d9866-b5rpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Sep 15 12:16:06.195065 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:06.195028    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7d7eff3e-openshift-default-86654d9866-b5rpz" podUID="6c776db8-f0ec-426e-abb7-09183b6e3c6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Sep 15 12:16:06.307648 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:06.307602    2555 patch_prober.go:28] interesting pod/e2e-autores-0e10f0-openshift-default-5d9bcff6b6-86lrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Sep 15 12:16:06.307823 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:06.307660    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-0e10f0-openshift-default-5d9bcff6b6-86lrx" podUID="bc1353ca-8864-4cc7-93af-1a53f57fc7ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Sep 15 12:16:06.993257 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:06.993111    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7d7eff3e-t894k"
Sep 15 12:16:07.121616 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:07.121520    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ffc0d2a-a645-467d-980e-bcc75b97d50a-maas-api-service-ca\") pod \"2ffc0d2a-a645-467d-980e-bcc75b97d50a\" (UID: \"2ffc0d2a-a645-467d-980e-bcc75b97d50a\") "
Sep 15 12:16:07.121836 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:07.121813    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ffc0d2a-a645-467d-980e-bcc75b97d50a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2ffc0d2a-a645-467d-980e-bcc75b97d50a" (UID: "2ffc0d2a-a645-467d-980e-bcc75b97d50a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:16:07.193503 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:07.193470    2555 patch_prober.go:28] interesting pod/e2e-ait-7d7eff3e-openshift-default-86654d9866-b5rpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Sep 15 12:16:07.193691 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:07.193529    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7d7eff3e-openshift-default-86654d9866-b5rpz" podUID="6c776db8-f0ec-426e-abb7-09183b6e3c6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Sep 15 12:16:07.222716 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:07.222674    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ffc0d2a-a645-467d-980e-bcc75b97d50a-maas-api-service-ca\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:16:07.311800 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:07.311771    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-0e10f0-openshift-default-5d9bcff6b6-86lrx"
Sep 15 12:16:07.312821 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:07.312803    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-0e10f0-openshift-default-5d9bcff6b6-86lrx"
Sep 15 12:16:07.756534 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:07.756491    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7d7eff3e-t894k"
Sep 15 12:16:07.757523 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:07.757488    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7d7eff3e-t894k" event={"ID":"2ffc0d2a-a645-467d-980e-bcc75b97d50a","Type":"ContainerDied","Data":"df4596d2196be6075e0f0d7a4b9ae16e16359963180d1d984e4eb7441e9bca84"}
Sep 15 12:16:07.757641 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:07.757550    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df4596d2196be6075e0f0d7a4b9ae16e16359963180d1d984e4eb7441e9bca84"
Sep 15 12:16:08.238639 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:08.238594    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-7d7eff3e-openshift-default-86654d9866-b5rpz"
Sep 15 12:16:08.242777 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:08.242755    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-7d7eff3e-openshift-default-86654d9866-b5rpz"
Sep 15 12:16:14.791703 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:14.791670    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-680229/simulator-w0-680229-kserve-6bbc79c558-r5j9c"]
Sep 15 12:16:14.792302 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:14.792207    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ffc0d2a-a645-467d-980e-bcc75b97d50a" containerName="revoke-keys"
Sep 15 12:16:14.792302 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:14.792223    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ffc0d2a-a645-467d-980e-bcc75b97d50a" containerName="revoke-keys"
Sep 15 12:16:14.792444 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:14.792361    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ffc0d2a-a645-467d-980e-bcc75b97d50a" containerName="revoke-keys"
Sep 15 12:16:14.795419 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:14.795401    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-680229/simulator-w0-680229-kserve-6bbc79c558-r5j9c"
Sep 15 12:16:14.799211 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:14.799187    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-680229\"/\"kube-root-ca.crt\""
Sep 15 12:16:14.799358 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:14.799287    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-680229\"/\"simulator-w0-680229-kserve-self-signed-certs\""
Sep 15 12:16:14.799358 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:14.799308    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-680229\"/\"default-dockercfg-jh667\""
Sep 15 12:16:14.799487 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:14.799431    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-680229\"/\"openshift-service-ca.crt\""
Sep 15 12:16:14.805302 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:14.805280    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-680229/simulator-w0-680229-kserve-6bbc79c558-r5j9c"]
Sep 15 12:16:14.889871 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:14.889831    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3409087-fe02-4ea0-9662-f682cd4fda20-tls-certs\") pod \"simulator-w0-680229-kserve-6bbc79c558-r5j9c\" (UID: \"b3409087-fe02-4ea0-9662-f682cd4fda20\") " pod="e2e-models-e2e-worker-w0-680229/simulator-w0-680229-kserve-6bbc79c558-r5j9c"
Sep 15 12:16:14.890066 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:14.889900    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3409087-fe02-4ea0-9662-f682cd4fda20-home\") pod \"simulator-w0-680229-kserve-6bbc79c558-r5j9c\" (UID: \"b3409087-fe02-4ea0-9662-f682cd4fda20\") " pod="e2e-models-e2e-worker-w0-680229/simulator-w0-680229-kserve-6bbc79c558-r5j9c"
Sep 15 12:16:14.890066 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:14.889927    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3409087-fe02-4ea0-9662-f682cd4fda20-model-cache\") pod \"simulator-w0-680229-kserve-6bbc79c558-r5j9c\" (UID: \"b3409087-fe02-4ea0-9662-f682cd4fda20\") " pod="e2e-models-e2e-worker-w0-680229/simulator-w0-680229-kserve-6bbc79c558-r5j9c"
Sep 15 12:16:14.890066 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:14.889989    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3409087-fe02-4ea0-9662-f682cd4fda20-tmp-dir\") pod \"simulator-w0-680229-kserve-6bbc79c558-r5j9c\" (UID: \"b3409087-fe02-4ea0-9662-f682cd4fda20\") " pod="e2e-models-e2e-worker-w0-680229/simulator-w0-680229-kserve-6bbc79c558-r5j9c"
Sep 15 12:16:14.890066 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:14.890028    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sgxx\" (UniqueName: \"kubernetes.io/projected/b3409087-fe02-4ea0-9662-f682cd4fda20-kube-api-access-6sgxx\") pod \"simulator-w0-680229-kserve-6bbc79c558-r5j9c\" (UID: \"b3409087-fe02-4ea0-9662-f682cd4fda20\") " pod="e2e-models-e2e-worker-w0-680229/simulator-w0-680229-kserve-6bbc79c558-r5j9c"
Sep 15 12:16:14.890379 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:14.890098    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3409087-fe02-4ea0-9662-f682cd4fda20-dshm\") pod \"simulator-w0-680229-kserve-6bbc79c558-r5j9c\" (UID: \"b3409087-fe02-4ea0-9662-f682cd4fda20\") " pod="e2e-models-e2e-worker-w0-680229/simulator-w0-680229-kserve-6bbc79c558-r5j9c"
Sep 15 12:16:14.990673 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:14.990637    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sgxx\" (UniqueName: \"kubernetes.io/projected/b3409087-fe02-4ea0-9662-f682cd4fda20-kube-api-access-6sgxx\") pod \"simulator-w0-680229-kserve-6bbc79c558-r5j9c\" (UID: \"b3409087-fe02-4ea0-9662-f682cd4fda20\") " pod="e2e-models-e2e-worker-w0-680229/simulator-w0-680229-kserve-6bbc79c558-r5j9c"
Sep 15 12:16:14.990673 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:14.990677    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3409087-fe02-4ea0-9662-f682cd4fda20-dshm\") pod \"simulator-w0-680229-kserve-6bbc79c558-r5j9c\" (UID: \"b3409087-fe02-4ea0-9662-f682cd4fda20\") " pod="e2e-models-e2e-worker-w0-680229/simulator-w0-680229-kserve-6bbc79c558-r5j9c"
Sep 15 12:16:14.990938 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:14.990718    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3409087-fe02-4ea0-9662-f682cd4fda20-tls-certs\") pod \"simulator-w0-680229-kserve-6bbc79c558-r5j9c\" (UID: \"b3409087-fe02-4ea0-9662-f682cd4fda20\") " pod="e2e-models-e2e-worker-w0-680229/simulator-w0-680229-kserve-6bbc79c558-r5j9c"
Sep 15 12:16:14.990938 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:14.990740    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3409087-fe02-4ea0-9662-f682cd4fda20-home\") pod \"simulator-w0-680229-kserve-6bbc79c558-r5j9c\" (UID: \"b3409087-fe02-4ea0-9662-f682cd4fda20\") " pod="e2e-models-e2e-worker-w0-680229/simulator-w0-680229-kserve-6bbc79c558-r5j9c"
Sep 15 12:16:14.990938 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:14.990757    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3409087-fe02-4ea0-9662-f682cd4fda20-model-cache\") pod \"simulator-w0-680229-kserve-6bbc79c558-r5j9c\" (UID: \"b3409087-fe02-4ea0-9662-f682cd4fda20\") " pod="e2e-models-e2e-worker-w0-680229/simulator-w0-680229-kserve-6bbc79c558-r5j9c"
Sep 15 12:16:14.990938 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:14.990789    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3409087-fe02-4ea0-9662-f682cd4fda20-tmp-dir\") pod \"simulator-w0-680229-kserve-6bbc79c558-r5j9c\" (UID: \"b3409087-fe02-4ea0-9662-f682cd4fda20\") " pod="e2e-models-e2e-worker-w0-680229/simulator-w0-680229-kserve-6bbc79c558-r5j9c"
Sep 15 12:16:14.991219 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:14.991195    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3409087-fe02-4ea0-9662-f682cd4fda20-model-cache\") pod \"simulator-w0-680229-kserve-6bbc79c558-r5j9c\" (UID: \"b3409087-fe02-4ea0-9662-f682cd4fda20\") " pod="e2e-models-e2e-worker-w0-680229/simulator-w0-680229-kserve-6bbc79c558-r5j9c"
Sep 15 12:16:14.991328 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:14.991253    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3409087-fe02-4ea0-9662-f682cd4fda20-tmp-dir\") pod \"simulator-w0-680229-kserve-6bbc79c558-r5j9c\" (UID: \"b3409087-fe02-4ea0-9662-f682cd4fda20\") " pod="e2e-models-e2e-worker-w0-680229/simulator-w0-680229-kserve-6bbc79c558-r5j9c"
Sep 15 12:16:14.991397 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:14.991362    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3409087-fe02-4ea0-9662-f682cd4fda20-home\") pod \"simulator-w0-680229-kserve-6bbc79c558-r5j9c\" (UID: \"b3409087-fe02-4ea0-9662-f682cd4fda20\") " pod="e2e-models-e2e-worker-w0-680229/simulator-w0-680229-kserve-6bbc79c558-r5j9c"
Sep 15 12:16:14.993144 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:14.993118    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3409087-fe02-4ea0-9662-f682cd4fda20-dshm\") pod \"simulator-w0-680229-kserve-6bbc79c558-r5j9c\" (UID: \"b3409087-fe02-4ea0-9662-f682cd4fda20\") " pod="e2e-models-e2e-worker-w0-680229/simulator-w0-680229-kserve-6bbc79c558-r5j9c"
Sep 15 12:16:14.993418 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:14.993398    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3409087-fe02-4ea0-9662-f682cd4fda20-tls-certs\") pod \"simulator-w0-680229-kserve-6bbc79c558-r5j9c\" (UID: \"b3409087-fe02-4ea0-9662-f682cd4fda20\") " pod="e2e-models-e2e-worker-w0-680229/simulator-w0-680229-kserve-6bbc79c558-r5j9c"
Sep 15 12:16:14.999938 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:14.999908    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sgxx\" (UniqueName: \"kubernetes.io/projected/b3409087-fe02-4ea0-9662-f682cd4fda20-kube-api-access-6sgxx\") pod \"simulator-w0-680229-kserve-6bbc79c558-r5j9c\" (UID: \"b3409087-fe02-4ea0-9662-f682cd4fda20\") " pod="e2e-models-e2e-worker-w0-680229/simulator-w0-680229-kserve-6bbc79c558-r5j9c"
Sep 15 12:16:15.105957 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:15.105859    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-680229/simulator-w0-680229-kserve-6bbc79c558-r5j9c"
Sep 15 12:16:15.405587 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:15.405557    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-680229/simulator-w0-680229-kserve-6bbc79c558-r5j9c"]
Sep 15 12:16:15.409601 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:16:15.409572    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb3409087_fe02_4ea0_9662_f682cd4fda20.slice/crio-fe589709f37b19d4178e720b5953dbdd4a971e379b7400fd7e3220cb11840ac5 WatchSource:0}: Error finding container fe589709f37b19d4178e720b5953dbdd4a971e379b7400fd7e3220cb11840ac5: Status 404 returned error can't find the container with id fe589709f37b19d4178e720b5953dbdd4a971e379b7400fd7e3220cb11840ac5
Sep 15 12:16:15.800236 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:15.800146    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-680229/simulator-w0-680229-kserve-6bbc79c558-r5j9c" event={"ID":"b3409087-fe02-4ea0-9662-f682cd4fda20","Type":"ContainerStarted","Data":"743d8529096a17348a805e86b88a62169f6358809573d5e9eece6ef1d1bc0a3a"}
Sep 15 12:16:15.800236 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:15.800198    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-680229/simulator-w0-680229-kserve-6bbc79c558-r5j9c" event={"ID":"b3409087-fe02-4ea0-9662-f682cd4fda20","Type":"ContainerStarted","Data":"fe589709f37b19d4178e720b5953dbdd4a971e379b7400fd7e3220cb11840ac5"}
Sep 15 12:16:15.825089 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:15.824496    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-680229/simulator-w0-680229-kserve-6bbc79c558-r5j9c" podStartSLOduration=1.8244762570000002 podStartE2EDuration="1.824476257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:16:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:16:15.823175999 +0000 UTC m=+832.614098531" watchObservedRunningTime="2026-09-15 12:16:15.824476257 +0000 UTC m=+832.615398769"
Sep 15 12:16:17.094945 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:17.094911    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7d7eff3e-594cd8dfdb-kqvk8"]
Sep 15 12:16:17.095688 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:17.095657    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-7d7eff3e-594cd8dfdb-kqvk8" podUID="202b8fe6-5531-49d5-a84d-9b9e910cef18" containerName="payload-pre-processing" containerID="cri-o://c270dc34805b2cff421235ffddb40a9eaa6d2b02895aa407e161deffab3eca25" gracePeriod=30
Sep 15 12:16:17.282400 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:17.282365    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7d7eff3e-t894k"]
Sep 15 12:16:17.284598 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:17.284570    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7d7eff3e-t894k"]
Sep 15 12:16:17.362049 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:17.362026    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7d7eff3e-594cd8dfdb-kqvk8"
Sep 15 12:16:17.427689 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:17.427653    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-7d7eff3e-openshift-default-86654d9866-b5rpz"]
Sep 15 12:16:17.427935 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:17.427910    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-7d7eff3e-openshift-default-86654d9866-b5rpz" podUID="6c776db8-f0ec-426e-abb7-09183b6e3c6b" containerName="istio-proxy" containerID="cri-o://621c63581e13fde66d336162aa53b797b94a475e6d816eab3ab44396161d9afb" gracePeriod=30
Sep 15 12:16:17.510390 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:17.510355    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ntj7z\" (UniqueName: \"kubernetes.io/projected/202b8fe6-5531-49d5-a84d-9b9e910cef18-kube-api-access-ntj7z\") pod \"202b8fe6-5531-49d5-a84d-9b9e910cef18\" (UID: \"202b8fe6-5531-49d5-a84d-9b9e910cef18\") "
Sep 15 12:16:17.510390 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:17.510397    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/202b8fe6-5531-49d5-a84d-9b9e910cef18-plugins-config-volume\") pod \"202b8fe6-5531-49d5-a84d-9b9e910cef18\" (UID: \"202b8fe6-5531-49d5-a84d-9b9e910cef18\") "
Sep 15 12:16:17.510805 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:17.510782    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/202b8fe6-5531-49d5-a84d-9b9e910cef18-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "202b8fe6-5531-49d5-a84d-9b9e910cef18" (UID: "202b8fe6-5531-49d5-a84d-9b9e910cef18"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:16:17.512660 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:17.512638    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/202b8fe6-5531-49d5-a84d-9b9e910cef18-kube-api-access-ntj7z" (OuterVolumeSpecName: "kube-api-access-ntj7z") pod "202b8fe6-5531-49d5-a84d-9b9e910cef18" (UID: "202b8fe6-5531-49d5-a84d-9b9e910cef18"). InnerVolumeSpecName "kube-api-access-ntj7z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:16:17.611375 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:17.611284    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntj7z\" (UniqueName: \"kubernetes.io/projected/202b8fe6-5531-49d5-a84d-9b9e910cef18-kube-api-access-ntj7z\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:16:17.611375 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:17.611314    2555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/202b8fe6-5531-49d5-a84d-9b9e910cef18-plugins-config-volume\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:16:17.811799 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:17.811763    2555 generic.go:356] "Generic (PLEG): container finished" podID="202b8fe6-5531-49d5-a84d-9b9e910cef18" containerID="c270dc34805b2cff421235ffddb40a9eaa6d2b02895aa407e161deffab3eca25" exitCode=0
Sep 15 12:16:17.811964 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:17.811884    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7d7eff3e-594cd8dfdb-kqvk8"
Sep 15 12:16:17.822054 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:17.820842    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ffc0d2a-a645-467d-980e-bcc75b97d50a" path="/var/lib/kubelet/pods/2ffc0d2a-a645-467d-980e-bcc75b97d50a/volumes"
Sep 15 12:16:17.822054 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:17.821407    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7d7eff3e-594cd8dfdb-kqvk8" event={"ID":"202b8fe6-5531-49d5-a84d-9b9e910cef18","Type":"ContainerDied","Data":"c270dc34805b2cff421235ffddb40a9eaa6d2b02895aa407e161deffab3eca25"}
Sep 15 12:16:17.822054 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:17.821465    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7d7eff3e-594cd8dfdb-kqvk8" event={"ID":"202b8fe6-5531-49d5-a84d-9b9e910cef18","Type":"ContainerDied","Data":"c656abd8ea6a55b20f12f9ed7a2ea6427f1802c0673d848a7554d966bdac353c"}
Sep 15 12:16:17.822054 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:17.821506    2555 scope.go:117] "RemoveContainer" containerID="c270dc34805b2cff421235ffddb40a9eaa6d2b02895aa407e161deffab3eca25"
Sep 15 12:16:17.832294 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:17.832249    2555 scope.go:117] "RemoveContainer" containerID="c270dc34805b2cff421235ffddb40a9eaa6d2b02895aa407e161deffab3eca25"
Sep 15 12:16:17.832614 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:16:17.832594    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c270dc34805b2cff421235ffddb40a9eaa6d2b02895aa407e161deffab3eca25\": container with ID starting with c270dc34805b2cff421235ffddb40a9eaa6d2b02895aa407e161deffab3eca25 not found: ID does not exist" containerID="c270dc34805b2cff421235ffddb40a9eaa6d2b02895aa407e161deffab3eca25"
Sep 15 12:16:17.832675 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:17.832622    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c270dc34805b2cff421235ffddb40a9eaa6d2b02895aa407e161deffab3eca25"} err="failed to get container status \"c270dc34805b2cff421235ffddb40a9eaa6d2b02895aa407e161deffab3eca25\": rpc error: code = NotFound desc = could not find container \"c270dc34805b2cff421235ffddb40a9eaa6d2b02895aa407e161deffab3eca25\": container with ID starting with c270dc34805b2cff421235ffddb40a9eaa6d2b02895aa407e161deffab3eca25 not found: ID does not exist"
Sep 15 12:16:17.839533 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:17.839507    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7d7eff3e-594cd8dfdb-kqvk8"]
Sep 15 12:16:17.841751 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:17.841727    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7d7eff3e-594cd8dfdb-kqvk8"]
Sep 15 12:16:18.423225 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:18.423193    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-1a5ccf40-gw-openshift-default-768d8f646d-w85gz"]
Sep 15 12:16:18.423755 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:18.423677    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="202b8fe6-5531-49d5-a84d-9b9e910cef18" containerName="payload-pre-processing"
Sep 15 12:16:18.423755 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:18.423696    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="202b8fe6-5531-49d5-a84d-9b9e910cef18" containerName="payload-pre-processing"
Sep 15 12:16:18.423840 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:18.423791    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="202b8fe6-5531-49d5-a84d-9b9e910cef18" containerName="payload-pre-processing"
Sep 15 12:16:18.428271 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:18.428243    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-1a5ccf40-gw-openshift-default-768d8f646d-w85gz"
Sep 15 12:16:18.430698 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:18.430679    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-1a5ccf40-gw-openshift-default-dockercfg-gwf7k\""
Sep 15 12:16:18.442455 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:18.442430    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-1a5ccf40-gw-openshift-default-768d8f646d-w85gz"]
Sep 15 12:16:18.515838 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:18.515784    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k76h4\" (UniqueName: \"kubernetes.io/projected/6fd3be91-8806-4522-b50e-97594dc9447b-kube-api-access-k76h4\") pod \"e2e-migrate-1a5ccf40-gw-openshift-default-768d8f646d-w85gz\" (UID: \"6fd3be91-8806-4522-b50e-97594dc9447b\") " pod="openshift-ingress/e2e-migrate-1a5ccf40-gw-openshift-default-768d8f646d-w85gz"
Sep 15 12:16:18.516052 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:18.515867    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fd3be91-8806-4522-b50e-97594dc9447b-workload-certs\") pod \"e2e-migrate-1a5ccf40-gw-openshift-default-768d8f646d-w85gz\" (UID: \"6fd3be91-8806-4522-b50e-97594dc9447b\") " pod="openshift-ingress/e2e-migrate-1a5ccf40-gw-openshift-default-768d8f646d-w85gz"
Sep 15 12:16:18.516052 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:18.515897    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fd3be91-8806-4522-b50e-97594dc9447b-workload-socket\") pod \"e2e-migrate-1a5ccf40-gw-openshift-default-768d8f646d-w85gz\" (UID: \"6fd3be91-8806-4522-b50e-97594dc9447b\") " pod="openshift-ingress/e2e-migrate-1a5ccf40-gw-openshift-default-768d8f646d-w85gz"
Sep 15 12:16:18.516052 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:18.515983    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fd3be91-8806-4522-b50e-97594dc9447b-istio-podinfo\") pod \"e2e-migrate-1a5ccf40-gw-openshift-default-768d8f646d-w85gz\" (UID: \"6fd3be91-8806-4522-b50e-97594dc9447b\") " pod="openshift-ingress/e2e-migrate-1a5ccf40-gw-openshift-default-768d8f646d-w85gz"
Sep 15 12:16:18.516052 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:18.516030    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fd3be91-8806-4522-b50e-97594dc9447b-istio-data\") pod \"e2e-migrate-1a5ccf40-gw-openshift-default-768d8f646d-w85gz\" (UID: \"6fd3be91-8806-4522-b50e-97594dc9447b\") " pod="openshift-ingress/e2e-migrate-1a5ccf40-gw-openshift-default-768d8f646d-w85gz"
Sep 15 12:16:18.516311 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:18.516057    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6fd3be91-8806-4522-b50e-97594dc9447b-istio-token\") pod \"e2e-migrate-1a5ccf40-gw-openshift-default-768d8f646d-w85gz\" (UID: \"6fd3be91-8806-4522-b50e-97594dc9447b\") " pod="openshift-ingress/e2e-migrate-1a5ccf40-gw-openshift-default-768d8f646d-w85gz"
Sep 15 12:16:18.516311 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:18.516099    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fd3be91-8806-4522-b50e-97594dc9447b-istio-envoy\") pod \"e2e-migrate-1a5ccf40-gw-openshift-default-768d8f646d-w85gz\" (UID: \"6fd3be91-8806-4522-b50e-97594dc9447b\") " pod="openshift-ingress/e2e-migrate-1a5ccf40-gw-openshift-default-768d8f646d-w85gz"
Sep 15 12:16:18.516311 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:18.516156    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fd3be91-8806-4522-b50e-97594dc9447b-credential-socket\") pod \"e2e-migrate-1a5ccf40-gw-openshift-default-768d8f646d-w85gz\" (UID: \"6fd3be91-8806-4522-b50e-97594dc9447b\") " pod="openshift-ingress/e2e-migrate-1a5ccf40-gw-openshift-default-768d8f646d-w85gz"
Sep 15 12:16:18.516311 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:18.516215    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fd3be91-8806-4522-b50e-97594dc9447b-istiod-ca-cert\") pod \"e2e-migrate-1a5ccf40-gw-openshift-default-768d8f646d-w85gz\" (UID: \"6fd3be91-8806-4522-b50e-97594dc9447b\") " pod="openshift-ingress/e2e-migrate-1a5ccf40-gw-openshift-default-768d8f646d-w85gz"
Sep 15 12:16:18.616787 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:18.616749    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fd3be91-8806-4522-b50e-97594dc9447b-istio-envoy\") pod \"e2e-migrate-1a5ccf40-gw-openshift-default-768d8f646d-w85gz\" (UID: \"6fd3be91-8806-4522-b50e-97594dc9447b\") " pod="openshift-ingress/e2e-migrate-1a5ccf40-gw-openshift-default-768d8f646d-w85gz"
Sep 15 12:16:18.616787 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:18.616792    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fd3be91-8806-4522-b50e-97594dc9447b-credential-socket\") pod \"e2e-migrate-1a5ccf40-gw-openshift-default-768d8f646d-w85gz\" (UID: \"6fd3be91-8806-4522-b50e-97594dc9447b\") " pod="openshift-ingress/e2e-migrate-1a5ccf40-gw-openshift-default-768d8f646d-w85gz"
Sep 15 12:16:18.617026 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:18.616810    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fd3be91-8806-4522-b50e-97594dc9447b-istiod-ca-cert\") pod \"e2e-migrate-1a5ccf40-gw-openshift-default-768d8f646d-w85gz\" (UID: \"6fd3be91-8806-4522-b50e-97594dc9447b\") " pod="openshift-ingress/e2e-migrate-1a5ccf40-gw-openshift-default-768d8f646d-w85gz"
Sep 15 12:16:18.617026 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:18.616857    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k76h4\" (UniqueName: \"kubernetes.io/projected/6fd3be91-8806-4522-b50e-97594dc9447b-kube-api-access-k76h4\") pod \"e2e-migrate-1a5ccf40-gw-openshift-default-768d8f646d-w85gz\" (UID: \"6fd3be91-8806-4522-b50e-97594dc9447b\") " pod="openshift-ingress/e2e-migrate-1a5ccf40-gw-openshift-default-768d8f646d-w85gz"
Sep 15 12:16:18.617026 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:18.616886    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fd3be91-8806-4522-b50e-97594dc9447b-workload-certs\") pod \"e2e-migrate-1a5ccf40-gw-openshift-default-768d8f646d-w85gz\" (UID: \"6fd3be91-8806-4522-b50e-97594dc9447b\") " pod="openshift-ingress/e2e-migrate-1a5ccf40-gw-openshift-default-768d8f646d-w85gz"
Sep 15 12:16:18.617026 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:18.616912    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fd3be91-8806-4522-b50e-97594dc9447b-workload-socket\") pod \"e2e-migrate-1a5ccf40-gw-openshift-default-768d8f646d-w85gz\" (UID: \"6fd3be91-8806-4522-b50e-97594dc9447b\") " pod="openshift-ingress/e2e-migrate-1a5ccf40-gw-openshift-default-768d8f646d-w85gz"
Sep 15 12:16:18.617026 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:18.616940    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fd3be91-8806-4522-b50e-97594dc9447b-istio-podinfo\") pod \"e2e-migrate-1a5ccf40-gw-openshift-default-768d8f646d-w85gz\" (UID: \"6fd3be91-8806-4522-b50e-97594dc9447b\") " pod="openshift-ingress/e2e-migrate-1a5ccf40-gw-openshift-default-768d8f646d-w85gz"
Sep 15 12:16:18.617026 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:18.616979    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fd3be91-8806-4522-b50e-97594dc9447b-istio-data\") pod \"e2e-migrate-1a5ccf40-gw-openshift-default-768d8f646d-w85gz\" (UID: \"6fd3be91-8806-4522-b50e-97594dc9447b\") " pod="openshift-ingress/e2e-migrate-1a5ccf40-gw-openshift-default-768d8f646d-w85gz"
Sep 15 12:16:18.617026 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:18.617009    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6fd3be91-8806-4522-b50e-97594dc9447b-istio-token\") pod \"e2e-migrate-1a5ccf40-gw-openshift-default-768d8f646d-w85gz\" (UID: \"6fd3be91-8806-4522-b50e-97594dc9447b\") " pod="openshift-ingress/e2e-migrate-1a5ccf40-gw-openshift-default-768d8f646d-w85gz"
Sep 15 12:16:18.617442 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:18.617256    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fd3be91-8806-4522-b50e-97594dc9447b-credential-socket\") pod \"e2e-migrate-1a5ccf40-gw-openshift-default-768d8f646d-w85gz\" (UID: \"6fd3be91-8806-4522-b50e-97594dc9447b\") " pod="openshift-ingress/e2e-migrate-1a5ccf40-gw-openshift-default-768d8f646d-w85gz"
Sep 15 12:16:18.617442 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:18.617397    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fd3be91-8806-4522-b50e-97594dc9447b-workload-socket\") pod \"e2e-migrate-1a5ccf40-gw-openshift-default-768d8f646d-w85gz\" (UID: \"6fd3be91-8806-4522-b50e-97594dc9447b\") " pod="openshift-ingress/e2e-migrate-1a5ccf40-gw-openshift-default-768d8f646d-w85gz"
Sep 15 12:16:18.617555 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:18.617477    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fd3be91-8806-4522-b50e-97594dc9447b-istio-data\") pod \"e2e-migrate-1a5ccf40-gw-openshift-default-768d8f646d-w85gz\" (UID: \"6fd3be91-8806-4522-b50e-97594dc9447b\") " pod="openshift-ingress/e2e-migrate-1a5ccf40-gw-openshift-default-768d8f646d-w85gz"
Sep 15 12:16:18.617670 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:18.617647    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fd3be91-8806-4522-b50e-97594dc9447b-workload-certs\") pod \"e2e-migrate-1a5ccf40-gw-openshift-default-768d8f646d-w85gz\" (UID: \"6fd3be91-8806-4522-b50e-97594dc9447b\") " pod="openshift-ingress/e2e-migrate-1a5ccf40-gw-openshift-default-768d8f646d-w85gz"
Sep 15 12:16:18.617729 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:18.617669    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fd3be91-8806-4522-b50e-97594dc9447b-istiod-ca-cert\") pod \"e2e-migrate-1a5ccf40-gw-openshift-default-768d8f646d-w85gz\" (UID: \"6fd3be91-8806-4522-b50e-97594dc9447b\") " pod="openshift-ingress/e2e-migrate-1a5ccf40-gw-openshift-default-768d8f646d-w85gz"
Sep 15 12:16:18.619023 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:18.618995    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fd3be91-8806-4522-b50e-97594dc9447b-istio-envoy\") pod \"e2e-migrate-1a5ccf40-gw-openshift-default-768d8f646d-w85gz\" (UID: \"6fd3be91-8806-4522-b50e-97594dc9447b\") " pod="openshift-ingress/e2e-migrate-1a5ccf40-gw-openshift-default-768d8f646d-w85gz"
Sep 15 12:16:18.619322 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:18.619304    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fd3be91-8806-4522-b50e-97594dc9447b-istio-podinfo\") pod \"e2e-migrate-1a5ccf40-gw-openshift-default-768d8f646d-w85gz\" (UID: \"6fd3be91-8806-4522-b50e-97594dc9447b\") " pod="openshift-ingress/e2e-migrate-1a5ccf40-gw-openshift-default-768d8f646d-w85gz"
Sep 15 12:16:18.626021 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:18.625998    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6fd3be91-8806-4522-b50e-97594dc9447b-istio-token\") pod \"e2e-migrate-1a5ccf40-gw-openshift-default-768d8f646d-w85gz\" (UID: \"6fd3be91-8806-4522-b50e-97594dc9447b\") " pod="openshift-ingress/e2e-migrate-1a5ccf40-gw-openshift-default-768d8f646d-w85gz"
Sep 15 12:16:18.628560 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:18.628537    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k76h4\" (UniqueName: \"kubernetes.io/projected/6fd3be91-8806-4522-b50e-97594dc9447b-kube-api-access-k76h4\") pod \"e2e-migrate-1a5ccf40-gw-openshift-default-768d8f646d-w85gz\" (UID: \"6fd3be91-8806-4522-b50e-97594dc9447b\") " pod="openshift-ingress/e2e-migrate-1a5ccf40-gw-openshift-default-768d8f646d-w85gz"
Sep 15 12:16:18.738104 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:18.738007    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-1a5ccf40-gw-openshift-default-768d8f646d-w85gz"
Sep 15 12:16:19.015272 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:19.015228    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-1a5ccf40-gw-openshift-default-768d8f646d-w85gz"]
Sep 15 12:16:19.015934 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:16:19.015907    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fd3be91_8806_4522_b50e_97594dc9447b.slice/crio-9349c31159b9f7a0679176e4bede653daa1efbcf8e7de447776b94f35ed30e99 WatchSource:0}: Error finding container 9349c31159b9f7a0679176e4bede653daa1efbcf8e7de447776b94f35ed30e99: Status 404 returned error can't find the container with id 9349c31159b9f7a0679176e4bede653daa1efbcf8e7de447776b94f35ed30e99
Sep 15 12:16:19.018457 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:19.018426    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:16:19.018524 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:19.018508    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:16:19.018564 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:19.018540    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:16:19.678066 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:19.678030    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5gkxs"]
Sep 15 12:16:19.819303 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:19.819252    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="202b8fe6-5531-49d5-a84d-9b9e910cef18" path="/var/lib/kubelet/pods/202b8fe6-5531-49d5-a84d-9b9e910cef18/volumes"
Sep 15 12:16:19.827371 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:19.827323    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-1a5ccf40-gw-openshift-default-768d8f646d-w85gz" event={"ID":"6fd3be91-8806-4522-b50e-97594dc9447b","Type":"ContainerStarted","Data":"6a69fb0a4528e19768a22778d4e4850a90a2ec0e6fddf7c237d086220cc3d8e7"}
Sep 15 12:16:19.827556 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:19.827387    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-1a5ccf40-gw-openshift-default-768d8f646d-w85gz" event={"ID":"6fd3be91-8806-4522-b50e-97594dc9447b","Type":"ContainerStarted","Data":"9349c31159b9f7a0679176e4bede653daa1efbcf8e7de447776b94f35ed30e99"}
Sep 15 12:16:19.851669 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:19.851612    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-1a5ccf40-gw-openshift-default-768d8f646d-w85gz" podStartSLOduration=1.8515938090000001 podStartE2EDuration="1.851593809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:16:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:16:19.849790489 +0000 UTC m=+836.640713025" watchObservedRunningTime="2026-09-15 12:16:19.851593809 +0000 UTC m=+836.642516316"
Sep 15 12:16:20.738549 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:20.738511    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-1a5ccf40-gw-openshift-default-768d8f646d-w85gz"
Sep 15 12:16:20.743871 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:20.743844    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-1a5ccf40-gw-openshift-default-768d8f646d-w85gz"
Sep 15 12:16:20.833165 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:20.833131    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-1a5ccf40-gw-openshift-default-768d8f646d-w85gz"
Sep 15 12:16:20.834188 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:20.834170    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-1a5ccf40-gw-openshift-default-768d8f646d-w85gz"
Sep 15 12:16:22.667623 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:22.667601    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7d7eff3e-openshift-default-86654d9866-b5rpz"
Sep 15 12:16:22.755700 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:22.755612    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c776db8-f0ec-426e-abb7-09183b6e3c6b-workload-certs\") pod \"6c776db8-f0ec-426e-abb7-09183b6e3c6b\" (UID: \"6c776db8-f0ec-426e-abb7-09183b6e3c6b\") "
Sep 15 12:16:22.755700 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:22.755676    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6m5bh\" (UniqueName: \"kubernetes.io/projected/6c776db8-f0ec-426e-abb7-09183b6e3c6b-kube-api-access-6m5bh\") pod \"6c776db8-f0ec-426e-abb7-09183b6e3c6b\" (UID: \"6c776db8-f0ec-426e-abb7-09183b6e3c6b\") "
Sep 15 12:16:22.755700 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:22.755700    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c776db8-f0ec-426e-abb7-09183b6e3c6b-istio-podinfo\") pod \"6c776db8-f0ec-426e-abb7-09183b6e3c6b\" (UID: \"6c776db8-f0ec-426e-abb7-09183b6e3c6b\") "
Sep 15 12:16:22.755931 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:22.755736    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c776db8-f0ec-426e-abb7-09183b6e3c6b-istiod-ca-cert\") pod \"6c776db8-f0ec-426e-abb7-09183b6e3c6b\" (UID: \"6c776db8-f0ec-426e-abb7-09183b6e3c6b\") "
Sep 15 12:16:22.755931 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:22.755800    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c776db8-f0ec-426e-abb7-09183b6e3c6b-istio-data\") pod \"6c776db8-f0ec-426e-abb7-09183b6e3c6b\" (UID: \"6c776db8-f0ec-426e-abb7-09183b6e3c6b\") "
Sep 15 12:16:22.755931 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:22.755834    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c776db8-f0ec-426e-abb7-09183b6e3c6b-istio-token\") pod \"6c776db8-f0ec-426e-abb7-09183b6e3c6b\" (UID: \"6c776db8-f0ec-426e-abb7-09183b6e3c6b\") "
Sep 15 12:16:22.755931 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:22.755878    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c776db8-f0ec-426e-abb7-09183b6e3c6b-workload-socket\") pod \"6c776db8-f0ec-426e-abb7-09183b6e3c6b\" (UID: \"6c776db8-f0ec-426e-abb7-09183b6e3c6b\") "
Sep 15 12:16:22.755931 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:22.755909    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c776db8-f0ec-426e-abb7-09183b6e3c6b-istio-envoy\") pod \"6c776db8-f0ec-426e-abb7-09183b6e3c6b\" (UID: \"6c776db8-f0ec-426e-abb7-09183b6e3c6b\") "
Sep 15 12:16:22.756182 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:22.755934    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c776db8-f0ec-426e-abb7-09183b6e3c6b-credential-socket\") pod \"6c776db8-f0ec-426e-abb7-09183b6e3c6b\" (UID: \"6c776db8-f0ec-426e-abb7-09183b6e3c6b\") "
Sep 15 12:16:22.756343 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:22.755926    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c776db8-f0ec-426e-abb7-09183b6e3c6b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6c776db8-f0ec-426e-abb7-09183b6e3c6b" (UID: "6c776db8-f0ec-426e-abb7-09183b6e3c6b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:16:22.756438 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:22.756161    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c776db8-f0ec-426e-abb7-09183b6e3c6b-istio-data" (OuterVolumeSpecName: "istio-data") pod "6c776db8-f0ec-426e-abb7-09183b6e3c6b" (UID: "6c776db8-f0ec-426e-abb7-09183b6e3c6b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:16:22.756438 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:22.756300    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c776db8-f0ec-426e-abb7-09183b6e3c6b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6c776db8-f0ec-426e-abb7-09183b6e3c6b" (UID: "6c776db8-f0ec-426e-abb7-09183b6e3c6b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:16:22.756545 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:22.756447    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c776db8-f0ec-426e-abb7-09183b6e3c6b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6c776db8-f0ec-426e-abb7-09183b6e3c6b" (UID: "6c776db8-f0ec-426e-abb7-09183b6e3c6b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:16:22.756545 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:22.756498    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c776db8-f0ec-426e-abb7-09183b6e3c6b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6c776db8-f0ec-426e-abb7-09183b6e3c6b" (UID: "6c776db8-f0ec-426e-abb7-09183b6e3c6b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:16:22.758144 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:22.758103    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6c776db8-f0ec-426e-abb7-09183b6e3c6b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6c776db8-f0ec-426e-abb7-09183b6e3c6b" (UID: "6c776db8-f0ec-426e-abb7-09183b6e3c6b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 12:16:22.758312 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:22.758248    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c776db8-f0ec-426e-abb7-09183b6e3c6b-istio-token" (OuterVolumeSpecName: "istio-token") pod "6c776db8-f0ec-426e-abb7-09183b6e3c6b" (UID: "6c776db8-f0ec-426e-abb7-09183b6e3c6b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:16:22.758394 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:22.758333    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c776db8-f0ec-426e-abb7-09183b6e3c6b-kube-api-access-6m5bh" (OuterVolumeSpecName: "kube-api-access-6m5bh") pod "6c776db8-f0ec-426e-abb7-09183b6e3c6b" (UID: "6c776db8-f0ec-426e-abb7-09183b6e3c6b"). InnerVolumeSpecName "kube-api-access-6m5bh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:16:22.758455 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:22.758425    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c776db8-f0ec-426e-abb7-09183b6e3c6b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6c776db8-f0ec-426e-abb7-09183b6e3c6b" (UID: "6c776db8-f0ec-426e-abb7-09183b6e3c6b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:16:22.845839 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:22.845806    2555 generic.go:356] "Generic (PLEG): container finished" podID="6c776db8-f0ec-426e-abb7-09183b6e3c6b" containerID="621c63581e13fde66d336162aa53b797b94a475e6d816eab3ab44396161d9afb" exitCode=0
Sep 15 12:16:22.846027 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:22.845854    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7d7eff3e-openshift-default-86654d9866-b5rpz"
Sep 15 12:16:22.846027 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:22.845896    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7d7eff3e-openshift-default-86654d9866-b5rpz" event={"ID":"6c776db8-f0ec-426e-abb7-09183b6e3c6b","Type":"ContainerDied","Data":"621c63581e13fde66d336162aa53b797b94a475e6d816eab3ab44396161d9afb"}
Sep 15 12:16:22.846027 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:22.845967    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7d7eff3e-openshift-default-86654d9866-b5rpz" event={"ID":"6c776db8-f0ec-426e-abb7-09183b6e3c6b","Type":"ContainerDied","Data":"9e3c8ad0cd143dcd3f3cd99041136f5bc0f2d8d71f488f5d8811e5aa671f3550"}
Sep 15 12:16:22.846027 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:22.845997    2555 scope.go:117] "RemoveContainer" containerID="621c63581e13fde66d336162aa53b797b94a475e6d816eab3ab44396161d9afb"
Sep 15 12:16:22.858913 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:22.858882    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6m5bh\" (UniqueName: \"kubernetes.io/projected/6c776db8-f0ec-426e-abb7-09183b6e3c6b-kube-api-access-6m5bh\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:16:22.858913 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:22.858913    2555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c776db8-f0ec-426e-abb7-09183b6e3c6b-istio-podinfo\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:16:22.858913 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:22.858931    2555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c776db8-f0ec-426e-abb7-09183b6e3c6b-istiod-ca-cert\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:16:22.859187 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:22.858944    2555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c776db8-f0ec-426e-abb7-09183b6e3c6b-istio-data\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:16:22.859187 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:22.858956    2555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c776db8-f0ec-426e-abb7-09183b6e3c6b-istio-token\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:16:22.859187 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:22.858967    2555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c776db8-f0ec-426e-abb7-09183b6e3c6b-workload-socket\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:16:22.859187 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:22.858979    2555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c776db8-f0ec-426e-abb7-09183b6e3c6b-istio-envoy\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:16:22.859187 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:22.858991    2555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c776db8-f0ec-426e-abb7-09183b6e3c6b-credential-socket\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:16:22.859187 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:22.859004    2555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c776db8-f0ec-426e-abb7-09183b6e3c6b-workload-certs\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:16:22.862628 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:22.862608    2555 scope.go:117] "RemoveContainer" containerID="4e44fd0feb698a461ec57cdd668eadf61641008e9852bd456e10db7ca418b955"
Sep 15 12:16:22.872833 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:22.872805    2555 scope.go:117] "RemoveContainer" containerID="621c63581e13fde66d336162aa53b797b94a475e6d816eab3ab44396161d9afb"
Sep 15 12:16:22.873179 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:16:22.873158    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"621c63581e13fde66d336162aa53b797b94a475e6d816eab3ab44396161d9afb\": container with ID starting with 621c63581e13fde66d336162aa53b797b94a475e6d816eab3ab44396161d9afb not found: ID does not exist" containerID="621c63581e13fde66d336162aa53b797b94a475e6d816eab3ab44396161d9afb"
Sep 15 12:16:22.873286 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:22.873193    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"621c63581e13fde66d336162aa53b797b94a475e6d816eab3ab44396161d9afb"} err="failed to get container status \"621c63581e13fde66d336162aa53b797b94a475e6d816eab3ab44396161d9afb\": rpc error: code = NotFound desc = could not find container \"621c63581e13fde66d336162aa53b797b94a475e6d816eab3ab44396161d9afb\": container with ID starting with 621c63581e13fde66d336162aa53b797b94a475e6d816eab3ab44396161d9afb not found: ID does not exist"
Sep 15 12:16:22.873286 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:22.873220    2555 scope.go:117] "RemoveContainer" containerID="4e44fd0feb698a461ec57cdd668eadf61641008e9852bd456e10db7ca418b955"
Sep 15 12:16:22.873685 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:16:22.873668    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e44fd0feb698a461ec57cdd668eadf61641008e9852bd456e10db7ca418b955\": container with ID starting with 4e44fd0feb698a461ec57cdd668eadf61641008e9852bd456e10db7ca418b955 not found: ID does not exist" containerID="4e44fd0feb698a461ec57cdd668eadf61641008e9852bd456e10db7ca418b955"
Sep 15 12:16:22.873759 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:22.873694    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e44fd0feb698a461ec57cdd668eadf61641008e9852bd456e10db7ca418b955"} err="failed to get container status \"4e44fd0feb698a461ec57cdd668eadf61641008e9852bd456e10db7ca418b955\": rpc error: code = NotFound desc = could not find container \"4e44fd0feb698a461ec57cdd668eadf61641008e9852bd456e10db7ca418b955\": container with ID starting with 4e44fd0feb698a461ec57cdd668eadf61641008e9852bd456e10db7ca418b955 not found: ID does not exist"
Sep 15 12:16:22.883271 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:22.883232    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-7d7eff3e-openshift-default-86654d9866-b5rpz"]
Sep 15 12:16:22.894591 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:22.894562    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-7d7eff3e-openshift-default-86654d9866-b5rpz"]
Sep 15 12:16:23.489248 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:23.489210    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-1a5ccf40-5655c54db6-z6zf2"]
Sep 15 12:16:23.489644 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:23.489630    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c776db8-f0ec-426e-abb7-09183b6e3c6b" containerName="istio-proxy"
Sep 15 12:16:23.489698 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:23.489647    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c776db8-f0ec-426e-abb7-09183b6e3c6b" containerName="istio-proxy"
Sep 15 12:16:23.489737 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:23.489723    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c776db8-f0ec-426e-abb7-09183b6e3c6b" containerName="istio-proxy"
Sep 15 12:16:23.489737 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:23.489733    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c776db8-f0ec-426e-abb7-09183b6e3c6b" containerName="istio-proxy"
Sep 15 12:16:23.494096 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:23.494078    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1a5ccf40-5655c54db6-z6zf2"
Sep 15 12:16:23.496745 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:23.496721    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-1a5ccf40\""
Sep 15 12:16:23.496839 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:23.496746    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-1a5ccf40-dockercfg-dmlst\""
Sep 15 12:16:23.507834 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:23.507807    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-1a5ccf40-5655c54db6-z6zf2"]
Sep 15 12:16:23.531051 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:23.531020    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-1a5ccf40-569d8db498-52gvc"]
Sep 15 12:16:23.531409 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:23.531397    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c776db8-f0ec-426e-abb7-09183b6e3c6b" containerName="istio-proxy"
Sep 15 12:16:23.531471 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:23.531411    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c776db8-f0ec-426e-abb7-09183b6e3c6b" containerName="istio-proxy"
Sep 15 12:16:23.534487 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:23.534471    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-1a5ccf40-569d8db498-52gvc"
Sep 15 12:16:23.546113 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:23.546083    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-1a5ccf40-569d8db498-52gvc"]
Sep 15 12:16:23.565707 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:23.565678    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f25f50e-c69d-41a3-aa4f-0cd0b3615e16-plugins-config-volume\") pod \"payload-processing-e2e-migrate-1a5ccf40-569d8db498-52gvc\" (UID: \"7f25f50e-c69d-41a3-aa4f-0cd0b3615e16\") " pod="openshift-ingress/payload-processing-e2e-migrate-1a5ccf40-569d8db498-52gvc"
Sep 15 12:16:23.565853 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:23.565729    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bea5d09-cd4c-4aea-9dfe-cd42a070bfbe-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-1a5ccf40-5655c54db6-z6zf2\" (UID: \"5bea5d09-cd4c-4aea-9dfe-cd42a070bfbe\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-1a5ccf40-5655c54db6-z6zf2"
Sep 15 12:16:23.565853 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:23.565757    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvqvj\" (UniqueName: \"kubernetes.io/projected/7f25f50e-c69d-41a3-aa4f-0cd0b3615e16-kube-api-access-gvqvj\") pod \"payload-processing-e2e-migrate-1a5ccf40-569d8db498-52gvc\" (UID: \"7f25f50e-c69d-41a3-aa4f-0cd0b3615e16\") " pod="openshift-ingress/payload-processing-e2e-migrate-1a5ccf40-569d8db498-52gvc"
Sep 15 12:16:23.565853 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:23.565790    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sq6gh\" (UniqueName: \"kubernetes.io/projected/5bea5d09-cd4c-4aea-9dfe-cd42a070bfbe-kube-api-access-sq6gh\") pod \"payload-pre-processing-e2e-migrate-1a5ccf40-5655c54db6-z6zf2\" (UID: \"5bea5d09-cd4c-4aea-9dfe-cd42a070bfbe\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-1a5ccf40-5655c54db6-z6zf2"
Sep 15 12:16:23.667074 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:23.667032    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f25f50e-c69d-41a3-aa4f-0cd0b3615e16-plugins-config-volume\") pod \"payload-processing-e2e-migrate-1a5ccf40-569d8db498-52gvc\" (UID: \"7f25f50e-c69d-41a3-aa4f-0cd0b3615e16\") " pod="openshift-ingress/payload-processing-e2e-migrate-1a5ccf40-569d8db498-52gvc"
Sep 15 12:16:23.667247 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:23.667092    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bea5d09-cd4c-4aea-9dfe-cd42a070bfbe-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-1a5ccf40-5655c54db6-z6zf2\" (UID: \"5bea5d09-cd4c-4aea-9dfe-cd42a070bfbe\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-1a5ccf40-5655c54db6-z6zf2"
Sep 15 12:16:23.667247 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:23.667108    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvqvj\" (UniqueName: \"kubernetes.io/projected/7f25f50e-c69d-41a3-aa4f-0cd0b3615e16-kube-api-access-gvqvj\") pod \"payload-processing-e2e-migrate-1a5ccf40-569d8db498-52gvc\" (UID: \"7f25f50e-c69d-41a3-aa4f-0cd0b3615e16\") " pod="openshift-ingress/payload-processing-e2e-migrate-1a5ccf40-569d8db498-52gvc"
Sep 15 12:16:23.667377 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:23.667293    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sq6gh\" (UniqueName: \"kubernetes.io/projected/5bea5d09-cd4c-4aea-9dfe-cd42a070bfbe-kube-api-access-sq6gh\") pod \"payload-pre-processing-e2e-migrate-1a5ccf40-5655c54db6-z6zf2\" (UID: \"5bea5d09-cd4c-4aea-9dfe-cd42a070bfbe\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-1a5ccf40-5655c54db6-z6zf2"
Sep 15 12:16:23.668038 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:23.668016    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bea5d09-cd4c-4aea-9dfe-cd42a070bfbe-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-1a5ccf40-5655c54db6-z6zf2\" (UID: \"5bea5d09-cd4c-4aea-9dfe-cd42a070bfbe\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-1a5ccf40-5655c54db6-z6zf2"
Sep 15 12:16:23.668305 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:23.668063    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f25f50e-c69d-41a3-aa4f-0cd0b3615e16-plugins-config-volume\") pod \"payload-processing-e2e-migrate-1a5ccf40-569d8db498-52gvc\" (UID: \"7f25f50e-c69d-41a3-aa4f-0cd0b3615e16\") " pod="openshift-ingress/payload-processing-e2e-migrate-1a5ccf40-569d8db498-52gvc"
Sep 15 12:16:23.676664 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:23.676638    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvqvj\" (UniqueName: \"kubernetes.io/projected/7f25f50e-c69d-41a3-aa4f-0cd0b3615e16-kube-api-access-gvqvj\") pod \"payload-processing-e2e-migrate-1a5ccf40-569d8db498-52gvc\" (UID: \"7f25f50e-c69d-41a3-aa4f-0cd0b3615e16\") " pod="openshift-ingress/payload-processing-e2e-migrate-1a5ccf40-569d8db498-52gvc"
Sep 15 12:16:23.676899 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:23.676879    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sq6gh\" (UniqueName: \"kubernetes.io/projected/5bea5d09-cd4c-4aea-9dfe-cd42a070bfbe-kube-api-access-sq6gh\") pod \"payload-pre-processing-e2e-migrate-1a5ccf40-5655c54db6-z6zf2\" (UID: \"5bea5d09-cd4c-4aea-9dfe-cd42a070bfbe\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-1a5ccf40-5655c54db6-z6zf2"
Sep 15 12:16:23.806618 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:23.806538    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-1a5ccf40-dockercfg-dmlst\""
Sep 15 12:16:23.814391 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:23.814358    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1a5ccf40-5655c54db6-z6zf2"
Sep 15 12:16:23.817879 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:23.817848    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c776db8-f0ec-426e-abb7-09183b6e3c6b" path="/var/lib/kubelet/pods/6c776db8-f0ec-426e-abb7-09183b6e3c6b/volumes"
Sep 15 12:16:23.844541 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:23.844514    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-1a5ccf40-569d8db498-52gvc"
Sep 15 12:16:24.099219 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:24.099187    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-1a5ccf40-5655c54db6-z6zf2"]
Sep 15 12:16:24.100908 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:16:24.100877    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bea5d09_cd4c_4aea_9dfe_cd42a070bfbe.slice/crio-9462abe2ce4947dd1acc21462f7239a5e00ea5c50839be6bc74e3673d547a239 WatchSource:0}: Error finding container 9462abe2ce4947dd1acc21462f7239a5e00ea5c50839be6bc74e3673d547a239: Status 404 returned error can't find the container with id 9462abe2ce4947dd1acc21462f7239a5e00ea5c50839be6bc74e3673d547a239
Sep 15 12:16:24.127969 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:24.127946    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-1a5ccf40-569d8db498-52gvc"]
Sep 15 12:16:24.133940 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:16:24.133906    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f25f50e_c69d_41a3_aa4f_0cd0b3615e16.slice/crio-0fd6835b48c461a19c35cd1b4517b0164812ce198cef85848f8390be38b3381f WatchSource:0}: Error finding container 0fd6835b48c461a19c35cd1b4517b0164812ce198cef85848f8390be38b3381f: Status 404 returned error can't find the container with id 0fd6835b48c461a19c35cd1b4517b0164812ce198cef85848f8390be38b3381f
Sep 15 12:16:24.861359 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:24.861305    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-1a5ccf40-569d8db498-52gvc" event={"ID":"7f25f50e-c69d-41a3-aa4f-0cd0b3615e16","Type":"ContainerStarted","Data":"0aed4771387b91b2583e4e3c1d9b9bffd4a67678db123646e99c19ab2a67a10e"}
Sep 15 12:16:24.861871 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:24.861373    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-1a5ccf40-569d8db498-52gvc" event={"ID":"7f25f50e-c69d-41a3-aa4f-0cd0b3615e16","Type":"ContainerStarted","Data":"0fd6835b48c461a19c35cd1b4517b0164812ce198cef85848f8390be38b3381f"}
Sep 15 12:16:24.861871 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:24.861457    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-1a5ccf40-569d8db498-52gvc"
Sep 15 12:16:24.862986 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:24.862958    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1a5ccf40-5655c54db6-z6zf2" event={"ID":"5bea5d09-cd4c-4aea-9dfe-cd42a070bfbe","Type":"ContainerStarted","Data":"69f6322348a9d56b5b168ae8f08e370b52db7bde33c443d7f3bf626bfca8b234"}
Sep 15 12:16:24.863121 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:24.863007    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1a5ccf40-5655c54db6-z6zf2" event={"ID":"5bea5d09-cd4c-4aea-9dfe-cd42a070bfbe","Type":"ContainerStarted","Data":"9462abe2ce4947dd1acc21462f7239a5e00ea5c50839be6bc74e3673d547a239"}
Sep 15 12:16:24.863121 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:24.863034    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1a5ccf40-5655c54db6-z6zf2"
Sep 15 12:16:24.877366 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:24.877305    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-1a5ccf40-569d8db498-52gvc" podStartSLOduration=1.877286038 podStartE2EDuration="1.877286038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:16:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:16:24.876031339 +0000 UTC m=+841.666953871" watchObservedRunningTime="2026-09-15 12:16:24.877286038 +0000 UTC m=+841.668208548"
Sep 15 12:16:24.892037 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:24.891984    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1a5ccf40-5655c54db6-z6zf2" podStartSLOduration=1.8919686470000001 podStartE2EDuration="1.891968647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:16:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:16:24.890521345 +0000 UTC m=+841.681443892" watchObservedRunningTime="2026-09-15 12:16:24.891968647 +0000 UTC m=+841.682891159"
Sep 15 12:16:25.105985 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:25.105956    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-680229/simulator-w0-680229-kserve-6bbc79c558-r5j9c"
Sep 15 12:16:25.106154 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:25.106007    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-680229/simulator-w0-680229-kserve-6bbc79c558-r5j9c"
Sep 15 12:16:25.119028 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:25.118944    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-680229/simulator-w0-680229-kserve-6bbc79c558-r5j9c"
Sep 15 12:16:25.880168 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:25.880139    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-680229/simulator-w0-680229-kserve-6bbc79c558-r5j9c"
Sep 15 12:16:35.871125 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:35.871089    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-1a5ccf40-569d8db498-52gvc"
Sep 15 12:16:35.871549 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:35.871176    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1a5ccf40-5655c54db6-z6zf2"
Sep 15 12:16:36.874474 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:36.874432    2555 patch_prober.go:28] interesting pod/e2e-migrate-1a5ccf40-gw-openshift-default-768d8f646d-w85gz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 15 12:16:36.874951 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:36.874518    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-1a5ccf40-gw-openshift-default-768d8f646d-w85gz" podUID="6fd3be91-8806-4522-b50e-97594dc9447b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 15 12:16:37.761929 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:37.761890    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5gkxs"]
Sep 15 12:16:38.942513 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:38.942422    2555 generic.go:356] "Generic (PLEG): container finished" podID="6fd3be91-8806-4522-b50e-97594dc9447b" containerID="6a69fb0a4528e19768a22778d4e4850a90a2ec0e6fddf7c237d086220cc3d8e7" exitCode=0
Sep 15 12:16:38.942928 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:38.942502    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-1a5ccf40-gw-openshift-default-768d8f646d-w85gz" event={"ID":"6fd3be91-8806-4522-b50e-97594dc9447b","Type":"ContainerDied","Data":"6a69fb0a4528e19768a22778d4e4850a90a2ec0e6fddf7c237d086220cc3d8e7"}
Sep 15 12:16:38.942972 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:38.942954    2555 scope.go:117] "RemoveContainer" containerID="6a69fb0a4528e19768a22778d4e4850a90a2ec0e6fddf7c237d086220cc3d8e7"
Sep 15 12:16:38.944148 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:38.944124    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:16:38.944219 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:38.944206    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:16:38.944271 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:38.944244    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:16:39.493186 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:39.493153    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-9ba48f/simulator-w4-9ba48f-kserve-67d55d8789-vvwbq"]
Sep 15 12:16:39.496715 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:39.496695    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9ba48f/simulator-w4-9ba48f-kserve-67d55d8789-vvwbq"
Sep 15 12:16:39.499441 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:39.499419    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-9ba48f\"/\"kube-root-ca.crt\""
Sep 15 12:16:39.500481 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:39.500457    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-9ba48f\"/\"openshift-service-ca.crt\""
Sep 15 12:16:39.500481 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:39.500470    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-9ba48f\"/\"default-dockercfg-mgxdh\""
Sep 15 12:16:39.500620 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:39.500602    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-9ba48f\"/\"simulator-w4-9ba48f-kserve-self-signed-certs\""
Sep 15 12:16:39.505838 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:39.505814    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-9ba48f/simulator-w4-9ba48f-kserve-67d55d8789-vvwbq"]
Sep 15 12:16:39.620734 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:39.620699    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5485dc6-da0b-4a35-a7ae-c7047dce4faa-home\") pod \"simulator-w4-9ba48f-kserve-67d55d8789-vvwbq\" (UID: \"a5485dc6-da0b-4a35-a7ae-c7047dce4faa\") " pod="e2e-models-e2e-worker-w4-9ba48f/simulator-w4-9ba48f-kserve-67d55d8789-vvwbq"
Sep 15 12:16:39.620928 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:39.620745    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5485dc6-da0b-4a35-a7ae-c7047dce4faa-tmp-dir\") pod \"simulator-w4-9ba48f-kserve-67d55d8789-vvwbq\" (UID: \"a5485dc6-da0b-4a35-a7ae-c7047dce4faa\") " pod="e2e-models-e2e-worker-w4-9ba48f/simulator-w4-9ba48f-kserve-67d55d8789-vvwbq"
Sep 15 12:16:39.620928 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:39.620772    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5485dc6-da0b-4a35-a7ae-c7047dce4faa-tls-certs\") pod \"simulator-w4-9ba48f-kserve-67d55d8789-vvwbq\" (UID: \"a5485dc6-da0b-4a35-a7ae-c7047dce4faa\") " pod="e2e-models-e2e-worker-w4-9ba48f/simulator-w4-9ba48f-kserve-67d55d8789-vvwbq"
Sep 15 12:16:39.620928 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:39.620816    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5485dc6-da0b-4a35-a7ae-c7047dce4faa-dshm\") pod \"simulator-w4-9ba48f-kserve-67d55d8789-vvwbq\" (UID: \"a5485dc6-da0b-4a35-a7ae-c7047dce4faa\") " pod="e2e-models-e2e-worker-w4-9ba48f/simulator-w4-9ba48f-kserve-67d55d8789-vvwbq"
Sep 15 12:16:39.620928 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:39.620860    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5485dc6-da0b-4a35-a7ae-c7047dce4faa-model-cache\") pod \"simulator-w4-9ba48f-kserve-67d55d8789-vvwbq\" (UID: \"a5485dc6-da0b-4a35-a7ae-c7047dce4faa\") " pod="e2e-models-e2e-worker-w4-9ba48f/simulator-w4-9ba48f-kserve-67d55d8789-vvwbq"
Sep 15 12:16:39.620928 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:39.620883    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfz99\" (UniqueName: \"kubernetes.io/projected/a5485dc6-da0b-4a35-a7ae-c7047dce4faa-kube-api-access-pfz99\") pod \"simulator-w4-9ba48f-kserve-67d55d8789-vvwbq\" (UID: \"a5485dc6-da0b-4a35-a7ae-c7047dce4faa\") " pod="e2e-models-e2e-worker-w4-9ba48f/simulator-w4-9ba48f-kserve-67d55d8789-vvwbq"
Sep 15 12:16:39.722359 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:39.722304    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5485dc6-da0b-4a35-a7ae-c7047dce4faa-home\") pod \"simulator-w4-9ba48f-kserve-67d55d8789-vvwbq\" (UID: \"a5485dc6-da0b-4a35-a7ae-c7047dce4faa\") " pod="e2e-models-e2e-worker-w4-9ba48f/simulator-w4-9ba48f-kserve-67d55d8789-vvwbq"
Sep 15 12:16:39.722560 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:39.722374    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5485dc6-da0b-4a35-a7ae-c7047dce4faa-tmp-dir\") pod \"simulator-w4-9ba48f-kserve-67d55d8789-vvwbq\" (UID: \"a5485dc6-da0b-4a35-a7ae-c7047dce4faa\") " pod="e2e-models-e2e-worker-w4-9ba48f/simulator-w4-9ba48f-kserve-67d55d8789-vvwbq"
Sep 15 12:16:39.722560 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:39.722401    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5485dc6-da0b-4a35-a7ae-c7047dce4faa-tls-certs\") pod \"simulator-w4-9ba48f-kserve-67d55d8789-vvwbq\" (UID: \"a5485dc6-da0b-4a35-a7ae-c7047dce4faa\") " pod="e2e-models-e2e-worker-w4-9ba48f/simulator-w4-9ba48f-kserve-67d55d8789-vvwbq"
Sep 15 12:16:39.722560 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:39.722425    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5485dc6-da0b-4a35-a7ae-c7047dce4faa-dshm\") pod \"simulator-w4-9ba48f-kserve-67d55d8789-vvwbq\" (UID: \"a5485dc6-da0b-4a35-a7ae-c7047dce4faa\") " pod="e2e-models-e2e-worker-w4-9ba48f/simulator-w4-9ba48f-kserve-67d55d8789-vvwbq"
Sep 15 12:16:39.722560 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:39.722449    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5485dc6-da0b-4a35-a7ae-c7047dce4faa-model-cache\") pod \"simulator-w4-9ba48f-kserve-67d55d8789-vvwbq\" (UID: \"a5485dc6-da0b-4a35-a7ae-c7047dce4faa\") " pod="e2e-models-e2e-worker-w4-9ba48f/simulator-w4-9ba48f-kserve-67d55d8789-vvwbq"
Sep 15 12:16:39.722560 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:39.722472    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfz99\" (UniqueName: \"kubernetes.io/projected/a5485dc6-da0b-4a35-a7ae-c7047dce4faa-kube-api-access-pfz99\") pod \"simulator-w4-9ba48f-kserve-67d55d8789-vvwbq\" (UID: \"a5485dc6-da0b-4a35-a7ae-c7047dce4faa\") " pod="e2e-models-e2e-worker-w4-9ba48f/simulator-w4-9ba48f-kserve-67d55d8789-vvwbq"
Sep 15 12:16:39.722800 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:39.722724    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5485dc6-da0b-4a35-a7ae-c7047dce4faa-home\") pod \"simulator-w4-9ba48f-kserve-67d55d8789-vvwbq\" (UID: \"a5485dc6-da0b-4a35-a7ae-c7047dce4faa\") " pod="e2e-models-e2e-worker-w4-9ba48f/simulator-w4-9ba48f-kserve-67d55d8789-vvwbq"
Sep 15 12:16:39.722859 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:39.722802    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5485dc6-da0b-4a35-a7ae-c7047dce4faa-tmp-dir\") pod \"simulator-w4-9ba48f-kserve-67d55d8789-vvwbq\" (UID: \"a5485dc6-da0b-4a35-a7ae-c7047dce4faa\") " pod="e2e-models-e2e-worker-w4-9ba48f/simulator-w4-9ba48f-kserve-67d55d8789-vvwbq"
Sep 15 12:16:39.722948 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:39.722920    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5485dc6-da0b-4a35-a7ae-c7047dce4faa-model-cache\") pod \"simulator-w4-9ba48f-kserve-67d55d8789-vvwbq\" (UID: \"a5485dc6-da0b-4a35-a7ae-c7047dce4faa\") " pod="e2e-models-e2e-worker-w4-9ba48f/simulator-w4-9ba48f-kserve-67d55d8789-vvwbq"
Sep 15 12:16:39.724744 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:39.724715    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5485dc6-da0b-4a35-a7ae-c7047dce4faa-dshm\") pod \"simulator-w4-9ba48f-kserve-67d55d8789-vvwbq\" (UID: \"a5485dc6-da0b-4a35-a7ae-c7047dce4faa\") " pod="e2e-models-e2e-worker-w4-9ba48f/simulator-w4-9ba48f-kserve-67d55d8789-vvwbq"
Sep 15 12:16:39.724977 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:39.724957    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5485dc6-da0b-4a35-a7ae-c7047dce4faa-tls-certs\") pod \"simulator-w4-9ba48f-kserve-67d55d8789-vvwbq\" (UID: \"a5485dc6-da0b-4a35-a7ae-c7047dce4faa\") " pod="e2e-models-e2e-worker-w4-9ba48f/simulator-w4-9ba48f-kserve-67d55d8789-vvwbq"
Sep 15 12:16:39.731117 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:39.731091    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfz99\" (UniqueName: \"kubernetes.io/projected/a5485dc6-da0b-4a35-a7ae-c7047dce4faa-kube-api-access-pfz99\") pod \"simulator-w4-9ba48f-kserve-67d55d8789-vvwbq\" (UID: \"a5485dc6-da0b-4a35-a7ae-c7047dce4faa\") " pod="e2e-models-e2e-worker-w4-9ba48f/simulator-w4-9ba48f-kserve-67d55d8789-vvwbq"
Sep 15 12:16:39.807817 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:39.807731    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9ba48f/simulator-w4-9ba48f-kserve-67d55d8789-vvwbq"
Sep 15 12:16:39.974861 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:39.974825    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-1a5ccf40-gw-openshift-default-768d8f646d-w85gz" event={"ID":"6fd3be91-8806-4522-b50e-97594dc9447b","Type":"ContainerStarted","Data":"d9e1a65a9dd907d4ca02eb89cb1513416b7fce6c4314ccdb87698ade5bfdb954"}
Sep 15 12:16:40.101042 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:40.101016    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-9ba48f/simulator-w4-9ba48f-kserve-67d55d8789-vvwbq"]
Sep 15 12:16:40.102122 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:16:40.102096    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda5485dc6_da0b_4a35_a7ae_c7047dce4faa.slice/crio-97c5f22bbc5bc2e2bdb47202389cf8f7e4e3aa7ec9fbe407a29962b4d3d1bffb WatchSource:0}: Error finding container 97c5f22bbc5bc2e2bdb47202389cf8f7e4e3aa7ec9fbe407a29962b4d3d1bffb: Status 404 returned error can't find the container with id 97c5f22bbc5bc2e2bdb47202389cf8f7e4e3aa7ec9fbe407a29962b4d3d1bffb
Sep 15 12:16:40.738501 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:40.738463    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-1a5ccf40-gw-openshift-default-768d8f646d-w85gz"
Sep 15 12:16:40.738751 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:40.738725    2555 patch_prober.go:28] interesting pod/e2e-migrate-1a5ccf40-gw-openshift-default-768d8f646d-w85gz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Sep 15 12:16:40.738861 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:40.738784    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-1a5ccf40-gw-openshift-default-768d8f646d-w85gz" podUID="6fd3be91-8806-4522-b50e-97594dc9447b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Sep 15 12:16:40.984184 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:40.984144    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9ba48f/simulator-w4-9ba48f-kserve-67d55d8789-vvwbq" event={"ID":"a5485dc6-da0b-4a35-a7ae-c7047dce4faa","Type":"ContainerStarted","Data":"dcc848dc24bf022799722a25b0f69d9deb970514cae6f5ae9e3d67d889185a48"}
Sep 15 12:16:40.984626 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:40.984199    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9ba48f/simulator-w4-9ba48f-kserve-67d55d8789-vvwbq" event={"ID":"a5485dc6-da0b-4a35-a7ae-c7047dce4faa","Type":"ContainerStarted","Data":"97c5f22bbc5bc2e2bdb47202389cf8f7e4e3aa7ec9fbe407a29962b4d3d1bffb"}
Sep 15 12:16:41.739357 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:41.739316    2555 patch_prober.go:28] interesting pod/e2e-migrate-1a5ccf40-gw-openshift-default-768d8f646d-w85gz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Sep 15 12:16:41.739551 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:41.739376    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-1a5ccf40-gw-openshift-default-768d8f646d-w85gz" podUID="6fd3be91-8806-4522-b50e-97594dc9447b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Sep 15 12:16:42.738488 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:42.738450    2555 patch_prober.go:28] interesting pod/e2e-migrate-1a5ccf40-gw-openshift-default-768d8f646d-w85gz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Sep 15 12:16:42.738886 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:42.738507    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-1a5ccf40-gw-openshift-default-768d8f646d-w85gz" podUID="6fd3be91-8806-4522-b50e-97594dc9447b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Sep 15 12:16:43.739038 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:43.738994    2555 patch_prober.go:28] interesting pod/e2e-migrate-1a5ccf40-gw-openshift-default-768d8f646d-w85gz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Sep 15 12:16:43.739471 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:43.739061    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-1a5ccf40-gw-openshift-default-768d8f646d-w85gz" podUID="6fd3be91-8806-4522-b50e-97594dc9447b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Sep 15 12:16:44.739414 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:44.739372    2555 patch_prober.go:28] interesting pod/e2e-migrate-1a5ccf40-gw-openshift-default-768d8f646d-w85gz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Sep 15 12:16:44.739806 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:44.739436    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-1a5ccf40-gw-openshift-default-768d8f646d-w85gz" podUID="6fd3be91-8806-4522-b50e-97594dc9447b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Sep 15 12:16:45.742834 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:45.742802    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-1a5ccf40-gw-openshift-default-768d8f646d-w85gz"
Sep 15 12:16:45.743208 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:45.743061    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-1a5ccf40-gw-openshift-default-768d8f646d-w85gz"
Sep 15 12:16:45.743797 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:45.743780    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-1a5ccf40-gw-openshift-default-768d8f646d-w85gz"
Sep 15 12:16:45.766446 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:45.766383    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-9ba48f/simulator-w4-9ba48f-kserve-67d55d8789-vvwbq" podStartSLOduration=6.766362175 podStartE2EDuration="6.766362175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:16:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:16:41.001567695 +0000 UTC m=+857.792490205" watchObservedRunningTime="2026-09-15 12:16:45.766362175 +0000 UTC m=+862.557284689"
Sep 15 12:16:49.807872 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:49.807829    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9ba48f/simulator-w4-9ba48f-kserve-67d55d8789-vvwbq"
Sep 15 12:16:49.808292 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:49.807889    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-9ba48f/simulator-w4-9ba48f-kserve-67d55d8789-vvwbq"
Sep 15 12:16:49.822656 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:49.822628    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-9ba48f/simulator-w4-9ba48f-kserve-67d55d8789-vvwbq"
Sep 15 12:16:50.046113 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:50.046083    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9ba48f/simulator-w4-9ba48f-kserve-67d55d8789-vvwbq"
Sep 15 12:16:51.863978 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:51.863942    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5gkxs"]
Sep 15 12:16:53.151847 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:53.151810    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-1a5ccf40-569d8db498-52gvc"]
Sep 15 12:16:53.152343 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:53.152078    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-1a5ccf40-569d8db498-52gvc" podUID="7f25f50e-c69d-41a3-aa4f-0cd0b3615e16" containerName="payload-processing" containerID="cri-o://0aed4771387b91b2583e4e3c1d9b9bffd4a67678db123646e99c19ab2a67a10e" gracePeriod=30
Sep 15 12:16:53.157828 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:53.157791    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-1a5ccf40-5655c54db6-z6zf2"]
Sep 15 12:16:53.158073 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:53.158020    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1a5ccf40-5655c54db6-z6zf2" podUID="5bea5d09-cd4c-4aea-9dfe-cd42a070bfbe" containerName="payload-pre-processing" containerID="cri-o://69f6322348a9d56b5b168ae8f08e370b52db7bde33c443d7f3bf626bfca8b234" gracePeriod=30
Sep 15 12:16:53.411198 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:53.410061    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-1a5ccf40-569d8db498-52gvc"
Sep 15 12:16:53.446464 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:53.446248    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1a5ccf40-5655c54db6-z6zf2"
Sep 15 12:16:53.449166 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:53.449141    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-1a5ccf40-gw-openshift-default-768d8f646d-w85gz"]
Sep 15 12:16:53.449460 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:53.449430    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-1a5ccf40-gw-openshift-default-768d8f646d-w85gz" podUID="6fd3be91-8806-4522-b50e-97594dc9447b" containerName="istio-proxy" containerID="cri-o://d9e1a65a9dd907d4ca02eb89cb1513416b7fce6c4314ccdb87698ade5bfdb954" gracePeriod=30
Sep 15 12:16:53.560303 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:53.560243    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bea5d09-cd4c-4aea-9dfe-cd42a070bfbe-plugins-config-volume\") pod \"5bea5d09-cd4c-4aea-9dfe-cd42a070bfbe\" (UID: \"5bea5d09-cd4c-4aea-9dfe-cd42a070bfbe\") "
Sep 15 12:16:53.560503 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:53.560416    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sq6gh\" (UniqueName: \"kubernetes.io/projected/5bea5d09-cd4c-4aea-9dfe-cd42a070bfbe-kube-api-access-sq6gh\") pod \"5bea5d09-cd4c-4aea-9dfe-cd42a070bfbe\" (UID: \"5bea5d09-cd4c-4aea-9dfe-cd42a070bfbe\") "
Sep 15 12:16:53.560503 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:53.560465    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f25f50e-c69d-41a3-aa4f-0cd0b3615e16-plugins-config-volume\") pod \"7f25f50e-c69d-41a3-aa4f-0cd0b3615e16\" (UID: \"7f25f50e-c69d-41a3-aa4f-0cd0b3615e16\") "
Sep 15 12:16:53.560635 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:53.560569    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gvqvj\" (UniqueName: \"kubernetes.io/projected/7f25f50e-c69d-41a3-aa4f-0cd0b3615e16-kube-api-access-gvqvj\") pod \"7f25f50e-c69d-41a3-aa4f-0cd0b3615e16\" (UID: \"7f25f50e-c69d-41a3-aa4f-0cd0b3615e16\") "
Sep 15 12:16:53.560700 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:53.560642    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bea5d09-cd4c-4aea-9dfe-cd42a070bfbe-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5bea5d09-cd4c-4aea-9dfe-cd42a070bfbe" (UID: "5bea5d09-cd4c-4aea-9dfe-cd42a070bfbe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:16:53.560889 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:53.560867    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f25f50e-c69d-41a3-aa4f-0cd0b3615e16-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7f25f50e-c69d-41a3-aa4f-0cd0b3615e16" (UID: "7f25f50e-c69d-41a3-aa4f-0cd0b3615e16"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:16:53.560997 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:53.560977    2555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bea5d09-cd4c-4aea-9dfe-cd42a070bfbe-plugins-config-volume\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:16:53.560997 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:53.560996    2555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f25f50e-c69d-41a3-aa4f-0cd0b3615e16-plugins-config-volume\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:16:53.562777 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:53.562749    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f25f50e-c69d-41a3-aa4f-0cd0b3615e16-kube-api-access-gvqvj" (OuterVolumeSpecName: "kube-api-access-gvqvj") pod "7f25f50e-c69d-41a3-aa4f-0cd0b3615e16" (UID: "7f25f50e-c69d-41a3-aa4f-0cd0b3615e16"). InnerVolumeSpecName "kube-api-access-gvqvj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:16:53.563219 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:53.563195    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bea5d09-cd4c-4aea-9dfe-cd42a070bfbe-kube-api-access-sq6gh" (OuterVolumeSpecName: "kube-api-access-sq6gh") pod "5bea5d09-cd4c-4aea-9dfe-cd42a070bfbe" (UID: "5bea5d09-cd4c-4aea-9dfe-cd42a070bfbe"). InnerVolumeSpecName "kube-api-access-sq6gh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:16:53.662078 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:53.661982    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvqvj\" (UniqueName: \"kubernetes.io/projected/7f25f50e-c69d-41a3-aa4f-0cd0b3615e16-kube-api-access-gvqvj\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:16:53.662078 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:53.662019    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sq6gh\" (UniqueName: \"kubernetes.io/projected/5bea5d09-cd4c-4aea-9dfe-cd42a070bfbe-kube-api-access-sq6gh\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:16:54.058321 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:54.058208    2555 generic.go:356] "Generic (PLEG): container finished" podID="7f25f50e-c69d-41a3-aa4f-0cd0b3615e16" containerID="0aed4771387b91b2583e4e3c1d9b9bffd4a67678db123646e99c19ab2a67a10e" exitCode=0
Sep 15 12:16:54.058321 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:54.058255    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-1a5ccf40-569d8db498-52gvc"
Sep 15 12:16:54.058321 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:54.058292    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-1a5ccf40-569d8db498-52gvc" event={"ID":"7f25f50e-c69d-41a3-aa4f-0cd0b3615e16","Type":"ContainerDied","Data":"0aed4771387b91b2583e4e3c1d9b9bffd4a67678db123646e99c19ab2a67a10e"}
Sep 15 12:16:54.058567 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:54.058347    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-1a5ccf40-569d8db498-52gvc" event={"ID":"7f25f50e-c69d-41a3-aa4f-0cd0b3615e16","Type":"ContainerDied","Data":"0fd6835b48c461a19c35cd1b4517b0164812ce198cef85848f8390be38b3381f"}
Sep 15 12:16:54.058567 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:54.058376    2555 scope.go:117] "RemoveContainer" containerID="0aed4771387b91b2583e4e3c1d9b9bffd4a67678db123646e99c19ab2a67a10e"
Sep 15 12:16:54.059973 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:54.059945    2555 generic.go:356] "Generic (PLEG): container finished" podID="5bea5d09-cd4c-4aea-9dfe-cd42a070bfbe" containerID="69f6322348a9d56b5b168ae8f08e370b52db7bde33c443d7f3bf626bfca8b234" exitCode=0
Sep 15 12:16:54.060101 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:54.060021    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1a5ccf40-5655c54db6-z6zf2" event={"ID":"5bea5d09-cd4c-4aea-9dfe-cd42a070bfbe","Type":"ContainerDied","Data":"69f6322348a9d56b5b168ae8f08e370b52db7bde33c443d7f3bf626bfca8b234"}
Sep 15 12:16:54.060101 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:54.060067    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1a5ccf40-5655c54db6-z6zf2" event={"ID":"5bea5d09-cd4c-4aea-9dfe-cd42a070bfbe","Type":"ContainerDied","Data":"9462abe2ce4947dd1acc21462f7239a5e00ea5c50839be6bc74e3673d547a239"}
Sep 15 12:16:54.060101 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:54.060097    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1a5ccf40-5655c54db6-z6zf2"
Sep 15 12:16:54.068718 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:54.068698    2555 scope.go:117] "RemoveContainer" containerID="0aed4771387b91b2583e4e3c1d9b9bffd4a67678db123646e99c19ab2a67a10e"
Sep 15 12:16:54.069067 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:16:54.069020    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0aed4771387b91b2583e4e3c1d9b9bffd4a67678db123646e99c19ab2a67a10e\": container with ID starting with 0aed4771387b91b2583e4e3c1d9b9bffd4a67678db123646e99c19ab2a67a10e not found: ID does not exist" containerID="0aed4771387b91b2583e4e3c1d9b9bffd4a67678db123646e99c19ab2a67a10e"
Sep 15 12:16:54.069135 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:54.069077    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0aed4771387b91b2583e4e3c1d9b9bffd4a67678db123646e99c19ab2a67a10e"} err="failed to get container status \"0aed4771387b91b2583e4e3c1d9b9bffd4a67678db123646e99c19ab2a67a10e\": rpc error: code = NotFound desc = could not find container \"0aed4771387b91b2583e4e3c1d9b9bffd4a67678db123646e99c19ab2a67a10e\": container with ID starting with 0aed4771387b91b2583e4e3c1d9b9bffd4a67678db123646e99c19ab2a67a10e not found: ID does not exist"
Sep 15 12:16:54.069135 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:54.069093    2555 scope.go:117] "RemoveContainer" containerID="69f6322348a9d56b5b168ae8f08e370b52db7bde33c443d7f3bf626bfca8b234"
Sep 15 12:16:54.077684 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:54.077658    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-1a5ccf40-5655c54db6-z6zf2"]
Sep 15 12:16:54.077945 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:54.077922    2555 scope.go:117] "RemoveContainer" containerID="69f6322348a9d56b5b168ae8f08e370b52db7bde33c443d7f3bf626bfca8b234"
Sep 15 12:16:54.078236 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:16:54.078216    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69f6322348a9d56b5b168ae8f08e370b52db7bde33c443d7f3bf626bfca8b234\": container with ID starting with 69f6322348a9d56b5b168ae8f08e370b52db7bde33c443d7f3bf626bfca8b234 not found: ID does not exist" containerID="69f6322348a9d56b5b168ae8f08e370b52db7bde33c443d7f3bf626bfca8b234"
Sep 15 12:16:54.078417 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:54.078244    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69f6322348a9d56b5b168ae8f08e370b52db7bde33c443d7f3bf626bfca8b234"} err="failed to get container status \"69f6322348a9d56b5b168ae8f08e370b52db7bde33c443d7f3bf626bfca8b234\": rpc error: code = NotFound desc = could not find container \"69f6322348a9d56b5b168ae8f08e370b52db7bde33c443d7f3bf626bfca8b234\": container with ID starting with 69f6322348a9d56b5b168ae8f08e370b52db7bde33c443d7f3bf626bfca8b234 not found: ID does not exist"
Sep 15 12:16:54.083238 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:54.083215    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-1a5ccf40-5655c54db6-z6zf2"]
Sep 15 12:16:54.093482 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:54.093453    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-1a5ccf40-569d8db498-52gvc"]
Sep 15 12:16:54.096971 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:54.096949    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-1a5ccf40-569d8db498-52gvc"]
Sep 15 12:16:55.817789 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:55.817756    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5bea5d09-cd4c-4aea-9dfe-cd42a070bfbe" path="/var/lib/kubelet/pods/5bea5d09-cd4c-4aea-9dfe-cd42a070bfbe/volumes"
Sep 15 12:16:55.818213 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:55.818196    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f25f50e-c69d-41a3-aa4f-0cd0b3615e16" path="/var/lib/kubelet/pods/7f25f50e-c69d-41a3-aa4f-0cd0b3615e16/volumes"
Sep 15 12:16:58.710171 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:58.710144    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-1a5ccf40-gw-openshift-default-768d8f646d-w85gz"
Sep 15 12:16:58.807556 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:58.807460    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fd3be91-8806-4522-b50e-97594dc9447b-istiod-ca-cert\") pod \"6fd3be91-8806-4522-b50e-97594dc9447b\" (UID: \"6fd3be91-8806-4522-b50e-97594dc9447b\") "
Sep 15 12:16:58.807556 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:58.807518    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fd3be91-8806-4522-b50e-97594dc9447b-workload-socket\") pod \"6fd3be91-8806-4522-b50e-97594dc9447b\" (UID: \"6fd3be91-8806-4522-b50e-97594dc9447b\") "
Sep 15 12:16:58.807556 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:58.807547    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fd3be91-8806-4522-b50e-97594dc9447b-istio-podinfo\") pod \"6fd3be91-8806-4522-b50e-97594dc9447b\" (UID: \"6fd3be91-8806-4522-b50e-97594dc9447b\") "
Sep 15 12:16:58.807841 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:58.807569    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fd3be91-8806-4522-b50e-97594dc9447b-credential-socket\") pod \"6fd3be91-8806-4522-b50e-97594dc9447b\" (UID: \"6fd3be91-8806-4522-b50e-97594dc9447b\") "
Sep 15 12:16:58.807841 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:58.807627    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6fd3be91-8806-4522-b50e-97594dc9447b-istio-token\") pod \"6fd3be91-8806-4522-b50e-97594dc9447b\" (UID: \"6fd3be91-8806-4522-b50e-97594dc9447b\") "
Sep 15 12:16:58.807841 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:58.807793    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fd3be91-8806-4522-b50e-97594dc9447b-workload-certs\") pod \"6fd3be91-8806-4522-b50e-97594dc9447b\" (UID: \"6fd3be91-8806-4522-b50e-97594dc9447b\") "
Sep 15 12:16:58.808010 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:58.807848    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fd3be91-8806-4522-b50e-97594dc9447b-istio-data\") pod \"6fd3be91-8806-4522-b50e-97594dc9447b\" (UID: \"6fd3be91-8806-4522-b50e-97594dc9447b\") "
Sep 15 12:16:58.808010 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:58.807861    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fd3be91-8806-4522-b50e-97594dc9447b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6fd3be91-8806-4522-b50e-97594dc9447b" (UID: "6fd3be91-8806-4522-b50e-97594dc9447b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:16:58.808010 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:58.807875    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fd3be91-8806-4522-b50e-97594dc9447b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6fd3be91-8806-4522-b50e-97594dc9447b" (UID: "6fd3be91-8806-4522-b50e-97594dc9447b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:16:58.808010 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:58.807863    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fd3be91-8806-4522-b50e-97594dc9447b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6fd3be91-8806-4522-b50e-97594dc9447b" (UID: "6fd3be91-8806-4522-b50e-97594dc9447b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:16:58.808010 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:58.807878    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k76h4\" (UniqueName: \"kubernetes.io/projected/6fd3be91-8806-4522-b50e-97594dc9447b-kube-api-access-k76h4\") pod \"6fd3be91-8806-4522-b50e-97594dc9447b\" (UID: \"6fd3be91-8806-4522-b50e-97594dc9447b\") "
Sep 15 12:16:58.808010 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:58.807954    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fd3be91-8806-4522-b50e-97594dc9447b-istio-envoy\") pod \"6fd3be91-8806-4522-b50e-97594dc9447b\" (UID: \"6fd3be91-8806-4522-b50e-97594dc9447b\") "
Sep 15 12:16:58.808356 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:58.808061    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fd3be91-8806-4522-b50e-97594dc9447b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6fd3be91-8806-4522-b50e-97594dc9447b" (UID: "6fd3be91-8806-4522-b50e-97594dc9447b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:16:58.808356 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:58.808080    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fd3be91-8806-4522-b50e-97594dc9447b-istio-data" (OuterVolumeSpecName: "istio-data") pod "6fd3be91-8806-4522-b50e-97594dc9447b" (UID: "6fd3be91-8806-4522-b50e-97594dc9447b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:16:58.808356 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:58.808334    2555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fd3be91-8806-4522-b50e-97594dc9447b-istiod-ca-cert\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:16:58.808356 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:58.808355    2555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fd3be91-8806-4522-b50e-97594dc9447b-workload-socket\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:16:58.808580 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:58.808369    2555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fd3be91-8806-4522-b50e-97594dc9447b-credential-socket\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:16:58.808580 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:58.808383    2555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fd3be91-8806-4522-b50e-97594dc9447b-workload-certs\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:16:58.808580 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:58.808396    2555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fd3be91-8806-4522-b50e-97594dc9447b-istio-data\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:16:58.809913 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:58.809879    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6fd3be91-8806-4522-b50e-97594dc9447b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6fd3be91-8806-4522-b50e-97594dc9447b" (UID: "6fd3be91-8806-4522-b50e-97594dc9447b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 12:16:58.809913 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:58.809902    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fd3be91-8806-4522-b50e-97594dc9447b-kube-api-access-k76h4" (OuterVolumeSpecName: "kube-api-access-k76h4") pod "6fd3be91-8806-4522-b50e-97594dc9447b" (UID: "6fd3be91-8806-4522-b50e-97594dc9447b"). InnerVolumeSpecName "kube-api-access-k76h4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:16:58.810078 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:58.809900    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fd3be91-8806-4522-b50e-97594dc9447b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6fd3be91-8806-4522-b50e-97594dc9447b" (UID: "6fd3be91-8806-4522-b50e-97594dc9447b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:16:58.810078 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:58.810048    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fd3be91-8806-4522-b50e-97594dc9447b-istio-token" (OuterVolumeSpecName: "istio-token") pod "6fd3be91-8806-4522-b50e-97594dc9447b" (UID: "6fd3be91-8806-4522-b50e-97594dc9447b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:16:58.909329 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:58.909288    2555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6fd3be91-8806-4522-b50e-97594dc9447b-istio-token\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:16:58.909329 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:58.909324    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k76h4\" (UniqueName: \"kubernetes.io/projected/6fd3be91-8806-4522-b50e-97594dc9447b-kube-api-access-k76h4\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:16:58.909329 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:58.909334    2555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fd3be91-8806-4522-b50e-97594dc9447b-istio-envoy\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:16:58.909613 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:58.909343    2555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fd3be91-8806-4522-b50e-97594dc9447b-istio-podinfo\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:16:59.094509 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:59.094425    2555 generic.go:356] "Generic (PLEG): container finished" podID="6fd3be91-8806-4522-b50e-97594dc9447b" containerID="d9e1a65a9dd907d4ca02eb89cb1513416b7fce6c4314ccdb87698ade5bfdb954" exitCode=0
Sep 15 12:16:59.094509 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:59.094466    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-1a5ccf40-gw-openshift-default-768d8f646d-w85gz" event={"ID":"6fd3be91-8806-4522-b50e-97594dc9447b","Type":"ContainerDied","Data":"d9e1a65a9dd907d4ca02eb89cb1513416b7fce6c4314ccdb87698ade5bfdb954"}
Sep 15 12:16:59.094710 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:59.094522    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-1a5ccf40-gw-openshift-default-768d8f646d-w85gz" event={"ID":"6fd3be91-8806-4522-b50e-97594dc9447b","Type":"ContainerDied","Data":"9349c31159b9f7a0679176e4bede653daa1efbcf8e7de447776b94f35ed30e99"}
Sep 15 12:16:59.094710 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:59.094547    2555 scope.go:117] "RemoveContainer" containerID="d9e1a65a9dd907d4ca02eb89cb1513416b7fce6c4314ccdb87698ade5bfdb954"
Sep 15 12:16:59.094710 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:59.094478    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-1a5ccf40-gw-openshift-default-768d8f646d-w85gz"
Sep 15 12:16:59.104503 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:59.104483    2555 scope.go:117] "RemoveContainer" containerID="6a69fb0a4528e19768a22778d4e4850a90a2ec0e6fddf7c237d086220cc3d8e7"
Sep 15 12:16:59.115527 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:59.115502    2555 scope.go:117] "RemoveContainer" containerID="d9e1a65a9dd907d4ca02eb89cb1513416b7fce6c4314ccdb87698ade5bfdb954"
Sep 15 12:16:59.115891 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:16:59.115861    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9e1a65a9dd907d4ca02eb89cb1513416b7fce6c4314ccdb87698ade5bfdb954\": container with ID starting with d9e1a65a9dd907d4ca02eb89cb1513416b7fce6c4314ccdb87698ade5bfdb954 not found: ID does not exist" containerID="d9e1a65a9dd907d4ca02eb89cb1513416b7fce6c4314ccdb87698ade5bfdb954"
Sep 15 12:16:59.115984 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:59.115903    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9e1a65a9dd907d4ca02eb89cb1513416b7fce6c4314ccdb87698ade5bfdb954"} err="failed to get container status \"d9e1a65a9dd907d4ca02eb89cb1513416b7fce6c4314ccdb87698ade5bfdb954\": rpc error: code = NotFound desc = could not find container \"d9e1a65a9dd907d4ca02eb89cb1513416b7fce6c4314ccdb87698ade5bfdb954\": container with ID starting with d9e1a65a9dd907d4ca02eb89cb1513416b7fce6c4314ccdb87698ade5bfdb954 not found: ID does not exist"
Sep 15 12:16:59.115984 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:59.115931    2555 scope.go:117] "RemoveContainer" containerID="6a69fb0a4528e19768a22778d4e4850a90a2ec0e6fddf7c237d086220cc3d8e7"
Sep 15 12:16:59.116307 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:16:59.116286    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a69fb0a4528e19768a22778d4e4850a90a2ec0e6fddf7c237d086220cc3d8e7\": container with ID starting with 6a69fb0a4528e19768a22778d4e4850a90a2ec0e6fddf7c237d086220cc3d8e7 not found: ID does not exist" containerID="6a69fb0a4528e19768a22778d4e4850a90a2ec0e6fddf7c237d086220cc3d8e7"
Sep 15 12:16:59.116398 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:59.116347    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a69fb0a4528e19768a22778d4e4850a90a2ec0e6fddf7c237d086220cc3d8e7"} err="failed to get container status \"6a69fb0a4528e19768a22778d4e4850a90a2ec0e6fddf7c237d086220cc3d8e7\": rpc error: code = NotFound desc = could not find container \"6a69fb0a4528e19768a22778d4e4850a90a2ec0e6fddf7c237d086220cc3d8e7\": container with ID starting with 6a69fb0a4528e19768a22778d4e4850a90a2ec0e6fddf7c237d086220cc3d8e7 not found: ID does not exist"
Sep 15 12:16:59.117915 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:59.117891    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-1a5ccf40-gw-openshift-default-768d8f646d-w85gz"]
Sep 15 12:16:59.127896 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:59.127868    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-1a5ccf40-gw-openshift-default-768d8f646d-w85gz"]
Sep 15 12:16:59.218905 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:59.218834    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3998f670-77755dc994-72bzl"]
Sep 15 12:16:59.219201 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:59.219189    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fd3be91-8806-4522-b50e-97594dc9447b" containerName="istio-proxy"
Sep 15 12:16:59.219247 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:59.219202    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fd3be91-8806-4522-b50e-97594dc9447b" containerName="istio-proxy"
Sep 15 12:16:59.219247 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:59.219211    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f25f50e-c69d-41a3-aa4f-0cd0b3615e16" containerName="payload-processing"
Sep 15 12:16:59.219247 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:59.219217    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f25f50e-c69d-41a3-aa4f-0cd0b3615e16" containerName="payload-processing"
Sep 15 12:16:59.219247 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:59.219226    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fd3be91-8806-4522-b50e-97594dc9447b" containerName="istio-proxy"
Sep 15 12:16:59.219247 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:59.219232    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fd3be91-8806-4522-b50e-97594dc9447b" containerName="istio-proxy"
Sep 15 12:16:59.219430 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:59.219248    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bea5d09-cd4c-4aea-9dfe-cd42a070bfbe" containerName="payload-pre-processing"
Sep 15 12:16:59.219430 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:59.219254    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bea5d09-cd4c-4aea-9dfe-cd42a070bfbe" containerName="payload-pre-processing"
Sep 15 12:16:59.219430 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:59.219325    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bea5d09-cd4c-4aea-9dfe-cd42a070bfbe" containerName="payload-pre-processing"
Sep 15 12:16:59.219430 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:59.219334    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f25f50e-c69d-41a3-aa4f-0cd0b3615e16" containerName="payload-processing"
Sep 15 12:16:59.219430 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:59.219343    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fd3be91-8806-4522-b50e-97594dc9447b" containerName="istio-proxy"
Sep 15 12:16:59.223932 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:59.223909    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3998f670-77755dc994-72bzl"
Sep 15 12:16:59.226610 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:59.226591    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-3998f670-dockercfg-j2ngd\""
Sep 15 12:16:59.226901 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:59.226885    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-3998f670\""
Sep 15 12:16:59.236100 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:59.236076    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3998f670-7bd777cff9-hqjsv"]
Sep 15 12:16:59.236537 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:59.236523    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fd3be91-8806-4522-b50e-97594dc9447b" containerName="istio-proxy"
Sep 15 12:16:59.239565 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:59.239541    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3998f670-7bd777cff9-hqjsv"
Sep 15 12:16:59.241255 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:59.241233    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3998f670-77755dc994-72bzl"]
Sep 15 12:16:59.253107 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:59.252976    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3998f670-7bd777cff9-hqjsv"]
Sep 15 12:16:59.313676 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:59.313643    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14271cdb-fcab-4c96-a148-560660fc9ad4-plugins-config-volume\") pod \"payload-processing-e2e-ait-3998f670-7bd777cff9-hqjsv\" (UID: \"14271cdb-fcab-4c96-a148-560660fc9ad4\") " pod="openshift-ingress/payload-processing-e2e-ait-3998f670-7bd777cff9-hqjsv"
Sep 15 12:16:59.313872 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:59.313688    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mksgv\" (UniqueName: \"kubernetes.io/projected/14271cdb-fcab-4c96-a148-560660fc9ad4-kube-api-access-mksgv\") pod \"payload-processing-e2e-ait-3998f670-7bd777cff9-hqjsv\" (UID: \"14271cdb-fcab-4c96-a148-560660fc9ad4\") " pod="openshift-ingress/payload-processing-e2e-ait-3998f670-7bd777cff9-hqjsv"
Sep 15 12:16:59.313872 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:59.313764    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fedb577-ed82-4376-8b8d-2b0fe330c66d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-3998f670-77755dc994-72bzl\" (UID: \"6fedb577-ed82-4376-8b8d-2b0fe330c66d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3998f670-77755dc994-72bzl"
Sep 15 12:16:59.313872 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:59.313797    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srt8n\" (UniqueName: \"kubernetes.io/projected/6fedb577-ed82-4376-8b8d-2b0fe330c66d-kube-api-access-srt8n\") pod \"payload-pre-processing-e2e-ait-3998f670-77755dc994-72bzl\" (UID: \"6fedb577-ed82-4376-8b8d-2b0fe330c66d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3998f670-77755dc994-72bzl"
Sep 15 12:16:59.414687 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:59.414648    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14271cdb-fcab-4c96-a148-560660fc9ad4-plugins-config-volume\") pod \"payload-processing-e2e-ait-3998f670-7bd777cff9-hqjsv\" (UID: \"14271cdb-fcab-4c96-a148-560660fc9ad4\") " pod="openshift-ingress/payload-processing-e2e-ait-3998f670-7bd777cff9-hqjsv"
Sep 15 12:16:59.414687 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:59.414689    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mksgv\" (UniqueName: \"kubernetes.io/projected/14271cdb-fcab-4c96-a148-560660fc9ad4-kube-api-access-mksgv\") pod \"payload-processing-e2e-ait-3998f670-7bd777cff9-hqjsv\" (UID: \"14271cdb-fcab-4c96-a148-560660fc9ad4\") " pod="openshift-ingress/payload-processing-e2e-ait-3998f670-7bd777cff9-hqjsv"
Sep 15 12:16:59.414981 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:59.414719    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fedb577-ed82-4376-8b8d-2b0fe330c66d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-3998f670-77755dc994-72bzl\" (UID: \"6fedb577-ed82-4376-8b8d-2b0fe330c66d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3998f670-77755dc994-72bzl"
Sep 15 12:16:59.414981 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:59.414734    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srt8n\" (UniqueName: \"kubernetes.io/projected/6fedb577-ed82-4376-8b8d-2b0fe330c66d-kube-api-access-srt8n\") pod \"payload-pre-processing-e2e-ait-3998f670-77755dc994-72bzl\" (UID: \"6fedb577-ed82-4376-8b8d-2b0fe330c66d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3998f670-77755dc994-72bzl"
Sep 15 12:16:59.415556 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:59.415530    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fedb577-ed82-4376-8b8d-2b0fe330c66d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-3998f670-77755dc994-72bzl\" (UID: \"6fedb577-ed82-4376-8b8d-2b0fe330c66d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3998f670-77755dc994-72bzl"
Sep 15 12:16:59.415651 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:59.415538    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14271cdb-fcab-4c96-a148-560660fc9ad4-plugins-config-volume\") pod \"payload-processing-e2e-ait-3998f670-7bd777cff9-hqjsv\" (UID: \"14271cdb-fcab-4c96-a148-560660fc9ad4\") " pod="openshift-ingress/payload-processing-e2e-ait-3998f670-7bd777cff9-hqjsv"
Sep 15 12:16:59.422980 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:59.422959    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-srt8n\" (UniqueName: \"kubernetes.io/projected/6fedb577-ed82-4376-8b8d-2b0fe330c66d-kube-api-access-srt8n\") pod \"payload-pre-processing-e2e-ait-3998f670-77755dc994-72bzl\" (UID: \"6fedb577-ed82-4376-8b8d-2b0fe330c66d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3998f670-77755dc994-72bzl"
Sep 15 12:16:59.423221 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:59.423204    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mksgv\" (UniqueName: \"kubernetes.io/projected/14271cdb-fcab-4c96-a148-560660fc9ad4-kube-api-access-mksgv\") pod \"payload-processing-e2e-ait-3998f670-7bd777cff9-hqjsv\" (UID: \"14271cdb-fcab-4c96-a148-560660fc9ad4\") " pod="openshift-ingress/payload-processing-e2e-ait-3998f670-7bd777cff9-hqjsv"
Sep 15 12:16:59.534686 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:59.534648    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3998f670-77755dc994-72bzl"
Sep 15 12:16:59.551016 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:59.550983    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3998f670-7bd777cff9-hqjsv"
Sep 15 12:16:59.810432 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:59.810403    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3998f670-77755dc994-72bzl"]
Sep 15 12:16:59.812613 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:16:59.812583    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fedb577_ed82_4376_8b8d_2b0fe330c66d.slice/crio-b463ff5c58010bae6dc05531eaafd3cf4e0c7896bf46ab63af7b0ba573c5c8d8 WatchSource:0}: Error finding container b463ff5c58010bae6dc05531eaafd3cf4e0c7896bf46ab63af7b0ba573c5c8d8: Status 404 returned error can't find the container with id b463ff5c58010bae6dc05531eaafd3cf4e0c7896bf46ab63af7b0ba573c5c8d8
Sep 15 12:16:59.818806 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:59.818779    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6fd3be91-8806-4522-b50e-97594dc9447b" path="/var/lib/kubelet/pods/6fd3be91-8806-4522-b50e-97594dc9447b/volumes"
Sep 15 12:16:59.839499 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:16:59.839476    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3998f670-7bd777cff9-hqjsv"]
Sep 15 12:16:59.848562 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:16:59.848535    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14271cdb_fcab_4c96_a148_560660fc9ad4.slice/crio-097fcddc5470dd451327bb1ecc252bf2287e5dee2caaa6b1703eead4ed95e946 WatchSource:0}: Error finding container 097fcddc5470dd451327bb1ecc252bf2287e5dee2caaa6b1703eead4ed95e946: Status 404 returned error can't find the container with id 097fcddc5470dd451327bb1ecc252bf2287e5dee2caaa6b1703eead4ed95e946
Sep 15 12:17:00.109656 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:00.109551    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3998f670-7bd777cff9-hqjsv" event={"ID":"14271cdb-fcab-4c96-a148-560660fc9ad4","Type":"ContainerStarted","Data":"4d659628f9fdc05e1c4c15c97690e5cdad56e199413b267a4a7922b8d68acae9"}
Sep 15 12:17:00.109897 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:00.109711    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-3998f670-7bd777cff9-hqjsv"
Sep 15 12:17:00.109897 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:00.109739    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3998f670-7bd777cff9-hqjsv" event={"ID":"14271cdb-fcab-4c96-a148-560660fc9ad4","Type":"ContainerStarted","Data":"097fcddc5470dd451327bb1ecc252bf2287e5dee2caaa6b1703eead4ed95e946"}
Sep 15 12:17:00.111554 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:00.111521    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3998f670-77755dc994-72bzl" event={"ID":"6fedb577-ed82-4376-8b8d-2b0fe330c66d","Type":"ContainerStarted","Data":"87521cba7a8c6127417ee5f1ef0e623e59d7c09f5786e7c91cd42b3ae9392183"}
Sep 15 12:17:00.111704 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:00.111581    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3998f670-77755dc994-72bzl" event={"ID":"6fedb577-ed82-4376-8b8d-2b0fe330c66d","Type":"ContainerStarted","Data":"b463ff5c58010bae6dc05531eaafd3cf4e0c7896bf46ab63af7b0ba573c5c8d8"}
Sep 15 12:17:00.111704 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:00.111627    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-3998f670-77755dc994-72bzl"
Sep 15 12:17:00.125544 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:00.125488    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-3998f670-7bd777cff9-hqjsv" podStartSLOduration=1.125472652 podStartE2EDuration="1.125472652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:16:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:17:00.124757586 +0000 UTC m=+876.915680097" watchObservedRunningTime="2026-09-15 12:17:00.125472652 +0000 UTC m=+876.916395162"
Sep 15 12:17:00.142784 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:00.142729    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-3998f670-77755dc994-72bzl" podStartSLOduration=1.142711306 podStartE2EDuration="1.142711306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:16:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:17:00.140358115 +0000 UTC m=+876.931280675" watchObservedRunningTime="2026-09-15 12:17:00.142711306 +0000 UTC m=+876.933633816"
Sep 15 12:17:04.168855 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:04.168818    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5gkxs"]
Sep 15 12:17:07.698379 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:07.698341    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-0e10f0/autores-model-0e10f0-kserve-768dd655ff-mgfxr"]
Sep 15 12:17:07.702102 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:07.702084    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-0e10f0/autores-model-0e10f0-kserve-768dd655ff-mgfxr"
Sep 15 12:17:07.705849 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:07.705817    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-0e10f0\"/\"autores-model-0e10f0-kserve-self-signed-certs\""
Sep 15 12:17:07.706011 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:07.705857    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-0e10f0\"/\"kube-root-ca.crt\""
Sep 15 12:17:07.706011 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:07.705888    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-0e10f0\"/\"default-dockercfg-xz54c\""
Sep 15 12:17:07.706011 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:07.705912    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-0e10f0\"/\"openshift-service-ca.crt\""
Sep 15 12:17:07.710593 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:07.710571    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-0e10f0/autores-model-0e10f0-kserve-768dd655ff-mgfxr"]
Sep 15 12:17:07.792186 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:07.792152    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7aaba80-442c-43a4-98a9-9726a96e35c1-model-cache\") pod \"autores-model-0e10f0-kserve-768dd655ff-mgfxr\" (UID: \"d7aaba80-442c-43a4-98a9-9726a96e35c1\") " pod="ai-tenant-e2e-autores-0e10f0/autores-model-0e10f0-kserve-768dd655ff-mgfxr"
Sep 15 12:17:07.792186 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:07.792190    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7aaba80-442c-43a4-98a9-9726a96e35c1-tls-certs\") pod \"autores-model-0e10f0-kserve-768dd655ff-mgfxr\" (UID: \"d7aaba80-442c-43a4-98a9-9726a96e35c1\") " pod="ai-tenant-e2e-autores-0e10f0/autores-model-0e10f0-kserve-768dd655ff-mgfxr"
Sep 15 12:17:07.792428 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:07.792215    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7aaba80-442c-43a4-98a9-9726a96e35c1-dshm\") pod \"autores-model-0e10f0-kserve-768dd655ff-mgfxr\" (UID: \"d7aaba80-442c-43a4-98a9-9726a96e35c1\") " pod="ai-tenant-e2e-autores-0e10f0/autores-model-0e10f0-kserve-768dd655ff-mgfxr"
Sep 15 12:17:07.792428 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:07.792303    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7aaba80-442c-43a4-98a9-9726a96e35c1-tmp-dir\") pod \"autores-model-0e10f0-kserve-768dd655ff-mgfxr\" (UID: \"d7aaba80-442c-43a4-98a9-9726a96e35c1\") " pod="ai-tenant-e2e-autores-0e10f0/autores-model-0e10f0-kserve-768dd655ff-mgfxr"
Sep 15 12:17:07.792428 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:07.792372    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7aaba80-442c-43a4-98a9-9726a96e35c1-home\") pod \"autores-model-0e10f0-kserve-768dd655ff-mgfxr\" (UID: \"d7aaba80-442c-43a4-98a9-9726a96e35c1\") " pod="ai-tenant-e2e-autores-0e10f0/autores-model-0e10f0-kserve-768dd655ff-mgfxr"
Sep 15 12:17:07.792428 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:07.792402    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4d9zf\" (UniqueName: \"kubernetes.io/projected/d7aaba80-442c-43a4-98a9-9726a96e35c1-kube-api-access-4d9zf\") pod \"autores-model-0e10f0-kserve-768dd655ff-mgfxr\" (UID: \"d7aaba80-442c-43a4-98a9-9726a96e35c1\") " pod="ai-tenant-e2e-autores-0e10f0/autores-model-0e10f0-kserve-768dd655ff-mgfxr"
Sep 15 12:17:07.893715 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:07.893663    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7aaba80-442c-43a4-98a9-9726a96e35c1-model-cache\") pod \"autores-model-0e10f0-kserve-768dd655ff-mgfxr\" (UID: \"d7aaba80-442c-43a4-98a9-9726a96e35c1\") " pod="ai-tenant-e2e-autores-0e10f0/autores-model-0e10f0-kserve-768dd655ff-mgfxr"
Sep 15 12:17:07.893715 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:07.893710    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7aaba80-442c-43a4-98a9-9726a96e35c1-tls-certs\") pod \"autores-model-0e10f0-kserve-768dd655ff-mgfxr\" (UID: \"d7aaba80-442c-43a4-98a9-9726a96e35c1\") " pod="ai-tenant-e2e-autores-0e10f0/autores-model-0e10f0-kserve-768dd655ff-mgfxr"
Sep 15 12:17:07.893963 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:07.893735    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7aaba80-442c-43a4-98a9-9726a96e35c1-dshm\") pod \"autores-model-0e10f0-kserve-768dd655ff-mgfxr\" (UID: \"d7aaba80-442c-43a4-98a9-9726a96e35c1\") " pod="ai-tenant-e2e-autores-0e10f0/autores-model-0e10f0-kserve-768dd655ff-mgfxr"
Sep 15 12:17:07.893963 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:07.893752    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7aaba80-442c-43a4-98a9-9726a96e35c1-tmp-dir\") pod \"autores-model-0e10f0-kserve-768dd655ff-mgfxr\" (UID: \"d7aaba80-442c-43a4-98a9-9726a96e35c1\") " pod="ai-tenant-e2e-autores-0e10f0/autores-model-0e10f0-kserve-768dd655ff-mgfxr"
Sep 15 12:17:07.893963 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:07.893788    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7aaba80-442c-43a4-98a9-9726a96e35c1-home\") pod \"autores-model-0e10f0-kserve-768dd655ff-mgfxr\" (UID: \"d7aaba80-442c-43a4-98a9-9726a96e35c1\") " pod="ai-tenant-e2e-autores-0e10f0/autores-model-0e10f0-kserve-768dd655ff-mgfxr"
Sep 15 12:17:07.893963 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:07.893805    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d9zf\" (UniqueName: \"kubernetes.io/projected/d7aaba80-442c-43a4-98a9-9726a96e35c1-kube-api-access-4d9zf\") pod \"autores-model-0e10f0-kserve-768dd655ff-mgfxr\" (UID: \"d7aaba80-442c-43a4-98a9-9726a96e35c1\") " pod="ai-tenant-e2e-autores-0e10f0/autores-model-0e10f0-kserve-768dd655ff-mgfxr"
Sep 15 12:17:07.894131 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:07.894096    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7aaba80-442c-43a4-98a9-9726a96e35c1-model-cache\") pod \"autores-model-0e10f0-kserve-768dd655ff-mgfxr\" (UID: \"d7aaba80-442c-43a4-98a9-9726a96e35c1\") " pod="ai-tenant-e2e-autores-0e10f0/autores-model-0e10f0-kserve-768dd655ff-mgfxr"
Sep 15 12:17:07.894193 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:07.894176    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7aaba80-442c-43a4-98a9-9726a96e35c1-tmp-dir\") pod \"autores-model-0e10f0-kserve-768dd655ff-mgfxr\" (UID: \"d7aaba80-442c-43a4-98a9-9726a96e35c1\") " pod="ai-tenant-e2e-autores-0e10f0/autores-model-0e10f0-kserve-768dd655ff-mgfxr"
Sep 15 12:17:07.894248 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:07.894216    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7aaba80-442c-43a4-98a9-9726a96e35c1-home\") pod \"autores-model-0e10f0-kserve-768dd655ff-mgfxr\" (UID: \"d7aaba80-442c-43a4-98a9-9726a96e35c1\") " pod="ai-tenant-e2e-autores-0e10f0/autores-model-0e10f0-kserve-768dd655ff-mgfxr"
Sep 15 12:17:07.896044 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:07.896020    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7aaba80-442c-43a4-98a9-9726a96e35c1-dshm\") pod \"autores-model-0e10f0-kserve-768dd655ff-mgfxr\" (UID: \"d7aaba80-442c-43a4-98a9-9726a96e35c1\") " pod="ai-tenant-e2e-autores-0e10f0/autores-model-0e10f0-kserve-768dd655ff-mgfxr"
Sep 15 12:17:07.896275 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:07.896244    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7aaba80-442c-43a4-98a9-9726a96e35c1-tls-certs\") pod \"autores-model-0e10f0-kserve-768dd655ff-mgfxr\" (UID: \"d7aaba80-442c-43a4-98a9-9726a96e35c1\") " pod="ai-tenant-e2e-autores-0e10f0/autores-model-0e10f0-kserve-768dd655ff-mgfxr"
Sep 15 12:17:07.901556 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:07.901533    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d9zf\" (UniqueName: \"kubernetes.io/projected/d7aaba80-442c-43a4-98a9-9726a96e35c1-kube-api-access-4d9zf\") pod \"autores-model-0e10f0-kserve-768dd655ff-mgfxr\" (UID: \"d7aaba80-442c-43a4-98a9-9726a96e35c1\") " pod="ai-tenant-e2e-autores-0e10f0/autores-model-0e10f0-kserve-768dd655ff-mgfxr"
Sep 15 12:17:08.012957 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:08.012861    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-0e10f0/autores-model-0e10f0-kserve-768dd655ff-mgfxr"
Sep 15 12:17:08.288909 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:08.288866    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-0e10f0/autores-model-0e10f0-kserve-768dd655ff-mgfxr"]
Sep 15 12:17:08.290027 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:17:08.290002    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd7aaba80_442c_43a4_98a9_9726a96e35c1.slice/crio-a5958ebd8afa6f396795f66a2858a65a5b4581dc711034afafba28fac432c9d3 WatchSource:0}: Error finding container a5958ebd8afa6f396795f66a2858a65a5b4581dc711034afafba28fac432c9d3: Status 404 returned error can't find the container with id a5958ebd8afa6f396795f66a2858a65a5b4581dc711034afafba28fac432c9d3
Sep 15 12:17:09.164003 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:09.163967    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-0e10f0/autores-model-0e10f0-kserve-768dd655ff-mgfxr" event={"ID":"d7aaba80-442c-43a4-98a9-9726a96e35c1","Type":"ContainerStarted","Data":"e3e7eb5330e7b013d95df35c14ef5cacc57840d25bd2f96c99e7d96969640576"}
Sep 15 12:17:09.164424 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:09.164018    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-0e10f0/autores-model-0e10f0-kserve-768dd655ff-mgfxr" event={"ID":"d7aaba80-442c-43a4-98a9-9726a96e35c1","Type":"ContainerStarted","Data":"a5958ebd8afa6f396795f66a2858a65a5b4581dc711034afafba28fac432c9d3"}
Sep 15 12:17:09.187939 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:09.187879    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-0e10f0/autores-model-0e10f0-kserve-768dd655ff-mgfxr" podStartSLOduration=2.187861765 podStartE2EDuration="2.187861765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:17:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:17:09.185818218 +0000 UTC m=+885.976740716" watchObservedRunningTime="2026-09-15 12:17:09.187861765 +0000 UTC m=+885.978784275"
Sep 15 12:17:11.121079 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:11.121048    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-3998f670-7bd777cff9-hqjsv"
Sep 15 12:17:11.121477 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:11.121126    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-3998f670-77755dc994-72bzl"
Sep 15 12:17:18.013091 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:18.012997    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-0e10f0/autores-model-0e10f0-kserve-768dd655ff-mgfxr"
Sep 15 12:17:18.013091 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:18.013059    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-0e10f0/autores-model-0e10f0-kserve-768dd655ff-mgfxr"
Sep 15 12:17:18.026507 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:18.026482    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-0e10f0/autores-model-0e10f0-kserve-768dd655ff-mgfxr"
Sep 15 12:17:18.226865 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:18.226834    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-0e10f0/autores-model-0e10f0-kserve-768dd655ff-mgfxr"
Sep 15 12:17:22.416007 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:22.415969    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3998f670-cz6p9"]
Sep 15 12:17:22.419571 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:22.419553    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3998f670-cz6p9"
Sep 15 12:17:22.422194 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:22.422168    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 12:17:22.422552 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:22.422535    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tbvkt\""
Sep 15 12:17:22.427781 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:22.427755    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3998f670-cz6p9"]
Sep 15 12:17:22.534634 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:22.534602    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec44f139-4b80-4844-bfca-3bef562b0419-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3998f670-cz6p9\" (UID: \"ec44f139-4b80-4844-bfca-3bef562b0419\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3998f670-cz6p9"
Sep 15 12:17:22.635520 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:22.635481    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec44f139-4b80-4844-bfca-3bef562b0419-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3998f670-cz6p9\" (UID: \"ec44f139-4b80-4844-bfca-3bef562b0419\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3998f670-cz6p9"
Sep 15 12:17:22.636127 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:22.636103    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec44f139-4b80-4844-bfca-3bef562b0419-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3998f670-cz6p9\" (UID: \"ec44f139-4b80-4844-bfca-3bef562b0419\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3998f670-cz6p9"
Sep 15 12:17:22.731579 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:22.731490    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3998f670-cz6p9"
Sep 15 12:17:23.008179 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:23.008145    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3998f670-cz6p9"]
Sep 15 12:17:23.009610 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:17:23.009581    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec44f139_4b80_4844_bfca_3bef562b0419.slice/crio-439b1c0e0fc6c477e6c99f099eb51dcd777e372c287a0c54274c0213465977b9 WatchSource:0}: Error finding container 439b1c0e0fc6c477e6c99f099eb51dcd777e372c287a0c54274c0213465977b9: Status 404 returned error can't find the container with id 439b1c0e0fc6c477e6c99f099eb51dcd777e372c287a0c54274c0213465977b9
Sep 15 12:17:23.244122 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:23.244086    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3998f670-cz6p9" event={"ID":"ec44f139-4b80-4844-bfca-3bef562b0419","Type":"ContainerStarted","Data":"439b1c0e0fc6c477e6c99f099eb51dcd777e372c287a0c54274c0213465977b9"}
Sep 15 12:17:24.251251 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:24.251212    2555 generic.go:356] "Generic (PLEG): container finished" podID="ec44f139-4b80-4844-bfca-3bef562b0419" containerID="cd5ab9bd0eea0fadf42514092c095488dc54c17cad87bd59eed5f267e575fb6d" exitCode=0
Sep 15 12:17:24.251680 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:24.251297    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3998f670-cz6p9" event={"ID":"ec44f139-4b80-4844-bfca-3bef562b0419","Type":"ContainerDied","Data":"cd5ab9bd0eea0fadf42514092c095488dc54c17cad87bd59eed5f267e575fb6d"}
Sep 15 12:17:24.275652 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:24.275612    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5gkxs"]
Sep 15 12:17:25.403082 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:25.403056    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3998f670-cz6p9"
Sep 15 12:17:25.562600 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:25.562518    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec44f139-4b80-4844-bfca-3bef562b0419-maas-api-service-ca\") pod \"ec44f139-4b80-4844-bfca-3bef562b0419\" (UID: \"ec44f139-4b80-4844-bfca-3bef562b0419\") "
Sep 15 12:17:25.562879 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:25.562856    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec44f139-4b80-4844-bfca-3bef562b0419-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ec44f139-4b80-4844-bfca-3bef562b0419" (UID: "ec44f139-4b80-4844-bfca-3bef562b0419"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:17:25.664105 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:25.664065    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec44f139-4b80-4844-bfca-3bef562b0419-maas-api-service-ca\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:17:26.263254 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:26.263223    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3998f670-cz6p9"
Sep 15 12:17:26.263454 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:26.263253    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3998f670-cz6p9" event={"ID":"ec44f139-4b80-4844-bfca-3bef562b0419","Type":"ContainerDied","Data":"439b1c0e0fc6c477e6c99f099eb51dcd777e372c287a0c54274c0213465977b9"}
Sep 15 12:17:26.263454 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:26.263320    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="439b1c0e0fc6c477e6c99f099eb51dcd777e372c287a0c54274c0213465977b9"
Sep 15 12:17:26.915412 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:26.915379    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-4a9453/test-model-4a9453-kserve-657c74dd4f-qbbxr"]
Sep 15 12:17:26.915809 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:26.915796    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec44f139-4b80-4844-bfca-3bef562b0419" containerName="revoke-keys"
Sep 15 12:17:26.915858 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:26.915810    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec44f139-4b80-4844-bfca-3bef562b0419" containerName="revoke-keys"
Sep 15 12:17:26.915896 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:26.915885    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec44f139-4b80-4844-bfca-3bef562b0419" containerName="revoke-keys"
Sep 15 12:17:26.919065 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:26.919048    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4a9453/test-model-4a9453-kserve-657c74dd4f-qbbxr"
Sep 15 12:17:26.923988 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:26.923962    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-4a9453\"/\"openshift-service-ca.crt\""
Sep 15 12:17:26.924232 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:26.924203    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-4a9453\"/\"test-model-4a9453-kserve-self-signed-certs\""
Sep 15 12:17:26.924409 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:26.924296    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-4a9453\"/\"kube-root-ca.crt\""
Sep 15 12:17:26.924409 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:26.924345    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-4a9453\"/\"default-dockercfg-wdjbl\""
Sep 15 12:17:26.929770 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:26.929743    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-4a9453/test-model-4a9453-kserve-657c74dd4f-qbbxr"]
Sep 15 12:17:27.077693 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:27.077662    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78cf1c6e-98e8-4296-97ed-00140d47133d-model-cache\") pod \"test-model-4a9453-kserve-657c74dd4f-qbbxr\" (UID: \"78cf1c6e-98e8-4296-97ed-00140d47133d\") " pod="ai-tenant-e2e-inf-a-4a9453/test-model-4a9453-kserve-657c74dd4f-qbbxr"
Sep 15 12:17:27.077693 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:27.077698    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78cf1c6e-98e8-4296-97ed-00140d47133d-tls-certs\") pod \"test-model-4a9453-kserve-657c74dd4f-qbbxr\" (UID: \"78cf1c6e-98e8-4296-97ed-00140d47133d\") " pod="ai-tenant-e2e-inf-a-4a9453/test-model-4a9453-kserve-657c74dd4f-qbbxr"
Sep 15 12:17:27.077906 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:27.077774    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78cf1c6e-98e8-4296-97ed-00140d47133d-home\") pod \"test-model-4a9453-kserve-657c74dd4f-qbbxr\" (UID: \"78cf1c6e-98e8-4296-97ed-00140d47133d\") " pod="ai-tenant-e2e-inf-a-4a9453/test-model-4a9453-kserve-657c74dd4f-qbbxr"
Sep 15 12:17:27.077906 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:27.077849    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78cf1c6e-98e8-4296-97ed-00140d47133d-tmp-dir\") pod \"test-model-4a9453-kserve-657c74dd4f-qbbxr\" (UID: \"78cf1c6e-98e8-4296-97ed-00140d47133d\") " pod="ai-tenant-e2e-inf-a-4a9453/test-model-4a9453-kserve-657c74dd4f-qbbxr"
Sep 15 12:17:27.077906 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:27.077875    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcxgq\" (UniqueName: \"kubernetes.io/projected/78cf1c6e-98e8-4296-97ed-00140d47133d-kube-api-access-vcxgq\") pod \"test-model-4a9453-kserve-657c74dd4f-qbbxr\" (UID: \"78cf1c6e-98e8-4296-97ed-00140d47133d\") " pod="ai-tenant-e2e-inf-a-4a9453/test-model-4a9453-kserve-657c74dd4f-qbbxr"
Sep 15 12:17:27.078008 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:27.077930    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78cf1c6e-98e8-4296-97ed-00140d47133d-dshm\") pod \"test-model-4a9453-kserve-657c74dd4f-qbbxr\" (UID: \"78cf1c6e-98e8-4296-97ed-00140d47133d\") " pod="ai-tenant-e2e-inf-a-4a9453/test-model-4a9453-kserve-657c74dd4f-qbbxr"
Sep 15 12:17:27.178795 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:27.178706    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78cf1c6e-98e8-4296-97ed-00140d47133d-dshm\") pod \"test-model-4a9453-kserve-657c74dd4f-qbbxr\" (UID: \"78cf1c6e-98e8-4296-97ed-00140d47133d\") " pod="ai-tenant-e2e-inf-a-4a9453/test-model-4a9453-kserve-657c74dd4f-qbbxr"
Sep 15 12:17:27.178795 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:27.178758    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78cf1c6e-98e8-4296-97ed-00140d47133d-model-cache\") pod \"test-model-4a9453-kserve-657c74dd4f-qbbxr\" (UID: \"78cf1c6e-98e8-4296-97ed-00140d47133d\") " pod="ai-tenant-e2e-inf-a-4a9453/test-model-4a9453-kserve-657c74dd4f-qbbxr"
Sep 15 12:17:27.178795 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:27.178787    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78cf1c6e-98e8-4296-97ed-00140d47133d-tls-certs\") pod \"test-model-4a9453-kserve-657c74dd4f-qbbxr\" (UID: \"78cf1c6e-98e8-4296-97ed-00140d47133d\") " pod="ai-tenant-e2e-inf-a-4a9453/test-model-4a9453-kserve-657c74dd4f-qbbxr"
Sep 15 12:17:27.179092 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:27.178944    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78cf1c6e-98e8-4296-97ed-00140d47133d-home\") pod \"test-model-4a9453-kserve-657c74dd4f-qbbxr\" (UID: \"78cf1c6e-98e8-4296-97ed-00140d47133d\") " pod="ai-tenant-e2e-inf-a-4a9453/test-model-4a9453-kserve-657c74dd4f-qbbxr"
Sep 15 12:17:27.179092 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:27.179003    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78cf1c6e-98e8-4296-97ed-00140d47133d-tmp-dir\") pod \"test-model-4a9453-kserve-657c74dd4f-qbbxr\" (UID: \"78cf1c6e-98e8-4296-97ed-00140d47133d\") " pod="ai-tenant-e2e-inf-a-4a9453/test-model-4a9453-kserve-657c74dd4f-qbbxr"
Sep 15 12:17:27.179092 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:27.179029    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcxgq\" (UniqueName: \"kubernetes.io/projected/78cf1c6e-98e8-4296-97ed-00140d47133d-kube-api-access-vcxgq\") pod \"test-model-4a9453-kserve-657c74dd4f-qbbxr\" (UID: \"78cf1c6e-98e8-4296-97ed-00140d47133d\") " pod="ai-tenant-e2e-inf-a-4a9453/test-model-4a9453-kserve-657c74dd4f-qbbxr"
Sep 15 12:17:27.179246 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:27.179192    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78cf1c6e-98e8-4296-97ed-00140d47133d-model-cache\") pod \"test-model-4a9453-kserve-657c74dd4f-qbbxr\" (UID: \"78cf1c6e-98e8-4296-97ed-00140d47133d\") " pod="ai-tenant-e2e-inf-a-4a9453/test-model-4a9453-kserve-657c74dd4f-qbbxr"
Sep 15 12:17:27.179246 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:27.179238    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78cf1c6e-98e8-4296-97ed-00140d47133d-home\") pod \"test-model-4a9453-kserve-657c74dd4f-qbbxr\" (UID: \"78cf1c6e-98e8-4296-97ed-00140d47133d\") " pod="ai-tenant-e2e-inf-a-4a9453/test-model-4a9453-kserve-657c74dd4f-qbbxr"
Sep 15 12:17:27.179402 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:27.179379    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78cf1c6e-98e8-4296-97ed-00140d47133d-tmp-dir\") pod \"test-model-4a9453-kserve-657c74dd4f-qbbxr\" (UID: \"78cf1c6e-98e8-4296-97ed-00140d47133d\") " pod="ai-tenant-e2e-inf-a-4a9453/test-model-4a9453-kserve-657c74dd4f-qbbxr"
Sep 15 12:17:27.180979 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:27.180957    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78cf1c6e-98e8-4296-97ed-00140d47133d-dshm\") pod \"test-model-4a9453-kserve-657c74dd4f-qbbxr\" (UID: \"78cf1c6e-98e8-4296-97ed-00140d47133d\") " pod="ai-tenant-e2e-inf-a-4a9453/test-model-4a9453-kserve-657c74dd4f-qbbxr"
Sep 15 12:17:27.181300 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:27.181257    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78cf1c6e-98e8-4296-97ed-00140d47133d-tls-certs\") pod \"test-model-4a9453-kserve-657c74dd4f-qbbxr\" (UID: \"78cf1c6e-98e8-4296-97ed-00140d47133d\") " pod="ai-tenant-e2e-inf-a-4a9453/test-model-4a9453-kserve-657c74dd4f-qbbxr"
Sep 15 12:17:27.187205 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:27.187179    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcxgq\" (UniqueName: \"kubernetes.io/projected/78cf1c6e-98e8-4296-97ed-00140d47133d-kube-api-access-vcxgq\") pod \"test-model-4a9453-kserve-657c74dd4f-qbbxr\" (UID: \"78cf1c6e-98e8-4296-97ed-00140d47133d\") " pod="ai-tenant-e2e-inf-a-4a9453/test-model-4a9453-kserve-657c74dd4f-qbbxr"
Sep 15 12:17:27.230697 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:27.230659    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4a9453/test-model-4a9453-kserve-657c74dd4f-qbbxr"
Sep 15 12:17:27.507798 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:27.507769    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-4a9453/test-model-4a9453-kserve-657c74dd4f-qbbxr"]
Sep 15 12:17:27.508603 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:17:27.508574    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod78cf1c6e_98e8_4296_97ed_00140d47133d.slice/crio-52d853e7e42f6f2d761e361b0594a75cefe691fd01beb2a4957ea381fa8d0258 WatchSource:0}: Error finding container 52d853e7e42f6f2d761e361b0594a75cefe691fd01beb2a4957ea381fa8d0258: Status 404 returned error can't find the container with id 52d853e7e42f6f2d761e361b0594a75cefe691fd01beb2a4957ea381fa8d0258
Sep 15 12:17:28.276362 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:28.276320    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4a9453/test-model-4a9453-kserve-657c74dd4f-qbbxr" event={"ID":"78cf1c6e-98e8-4296-97ed-00140d47133d","Type":"ContainerStarted","Data":"7291105c3578a3f6ce82e5ece08104b032f79bb1ee59ff94171d9729f8453cf8"}
Sep 15 12:17:28.276362 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:28.276368    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4a9453/test-model-4a9453-kserve-657c74dd4f-qbbxr" event={"ID":"78cf1c6e-98e8-4296-97ed-00140d47133d","Type":"ContainerStarted","Data":"52d853e7e42f6f2d761e361b0594a75cefe691fd01beb2a4957ea381fa8d0258"}
Sep 15 12:17:28.293563 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:28.293509    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-4a9453/test-model-4a9453-kserve-657c74dd4f-qbbxr" podStartSLOduration=2.293491724 podStartE2EDuration="2.293491724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:17:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:17:28.292361036 +0000 UTC m=+905.083283551" watchObservedRunningTime="2026-09-15 12:17:28.293491724 +0000 UTC m=+905.084414235"
Sep 15 12:17:28.423109 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:28.423075    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3998f670-7bd777cff9-hqjsv"]
Sep 15 12:17:28.423378 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:28.423350    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-3998f670-7bd777cff9-hqjsv" podUID="14271cdb-fcab-4c96-a148-560660fc9ad4" containerName="payload-processing" containerID="cri-o://4d659628f9fdc05e1c4c15c97690e5cdad56e199413b267a4a7922b8d68acae9" gracePeriod=30
Sep 15 12:17:28.435925 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:28.435896    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3998f670-77755dc994-72bzl"]
Sep 15 12:17:28.436285 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:28.436225    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-3998f670-77755dc994-72bzl" podUID="6fedb577-ed82-4376-8b8d-2b0fe330c66d" containerName="payload-pre-processing" containerID="cri-o://87521cba7a8c6127417ee5f1ef0e623e59d7c09f5786e7c91cd42b3ae9392183" gracePeriod=30
Sep 15 12:17:28.621322 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:28.621288    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3998f670-cz6p9"]
Sep 15 12:17:28.623655 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:28.623634    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3998f670-cz6p9"]
Sep 15 12:17:28.709794 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:28.709762    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3998f670-7bd777cff9-hqjsv"
Sep 15 12:17:28.712584 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:28.712564    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3998f670-77755dc994-72bzl"
Sep 15 12:17:28.793841 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:28.793811    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14271cdb-fcab-4c96-a148-560660fc9ad4-plugins-config-volume\") pod \"14271cdb-fcab-4c96-a148-560660fc9ad4\" (UID: \"14271cdb-fcab-4c96-a148-560660fc9ad4\") "
Sep 15 12:17:28.794016 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:28.793852    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-srt8n\" (UniqueName: \"kubernetes.io/projected/6fedb577-ed82-4376-8b8d-2b0fe330c66d-kube-api-access-srt8n\") pod \"6fedb577-ed82-4376-8b8d-2b0fe330c66d\" (UID: \"6fedb577-ed82-4376-8b8d-2b0fe330c66d\") "
Sep 15 12:17:28.794016 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:28.793921    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mksgv\" (UniqueName: \"kubernetes.io/projected/14271cdb-fcab-4c96-a148-560660fc9ad4-kube-api-access-mksgv\") pod \"14271cdb-fcab-4c96-a148-560660fc9ad4\" (UID: \"14271cdb-fcab-4c96-a148-560660fc9ad4\") "
Sep 15 12:17:28.794016 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:28.793946    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fedb577-ed82-4376-8b8d-2b0fe330c66d-plugins-config-volume\") pod \"6fedb577-ed82-4376-8b8d-2b0fe330c66d\" (UID: \"6fedb577-ed82-4376-8b8d-2b0fe330c66d\") "
Sep 15 12:17:28.794354 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:28.794327    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14271cdb-fcab-4c96-a148-560660fc9ad4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "14271cdb-fcab-4c96-a148-560660fc9ad4" (UID: "14271cdb-fcab-4c96-a148-560660fc9ad4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:17:28.794443 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:28.794365    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fedb577-ed82-4376-8b8d-2b0fe330c66d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6fedb577-ed82-4376-8b8d-2b0fe330c66d" (UID: "6fedb577-ed82-4376-8b8d-2b0fe330c66d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:17:28.795999 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:28.795982    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fedb577-ed82-4376-8b8d-2b0fe330c66d-kube-api-access-srt8n" (OuterVolumeSpecName: "kube-api-access-srt8n") pod "6fedb577-ed82-4376-8b8d-2b0fe330c66d" (UID: "6fedb577-ed82-4376-8b8d-2b0fe330c66d"). InnerVolumeSpecName "kube-api-access-srt8n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:17:28.796077 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:28.796058    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14271cdb-fcab-4c96-a148-560660fc9ad4-kube-api-access-mksgv" (OuterVolumeSpecName: "kube-api-access-mksgv") pod "14271cdb-fcab-4c96-a148-560660fc9ad4" (UID: "14271cdb-fcab-4c96-a148-560660fc9ad4"). InnerVolumeSpecName "kube-api-access-mksgv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:17:28.895528 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:28.895486    2555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14271cdb-fcab-4c96-a148-560660fc9ad4-plugins-config-volume\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:17:28.895528 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:28.895522    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srt8n\" (UniqueName: \"kubernetes.io/projected/6fedb577-ed82-4376-8b8d-2b0fe330c66d-kube-api-access-srt8n\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:17:28.895528 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:28.895531    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mksgv\" (UniqueName: \"kubernetes.io/projected/14271cdb-fcab-4c96-a148-560660fc9ad4-kube-api-access-mksgv\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:17:28.895528 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:28.895539    2555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fedb577-ed82-4376-8b8d-2b0fe330c66d-plugins-config-volume\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:17:29.283135 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:29.283095    2555 generic.go:356] "Generic (PLEG): container finished" podID="14271cdb-fcab-4c96-a148-560660fc9ad4" containerID="4d659628f9fdc05e1c4c15c97690e5cdad56e199413b267a4a7922b8d68acae9" exitCode=0
Sep 15 12:17:29.283577 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:29.283142    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3998f670-7bd777cff9-hqjsv"
Sep 15 12:17:29.283577 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:29.283172    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3998f670-7bd777cff9-hqjsv" event={"ID":"14271cdb-fcab-4c96-a148-560660fc9ad4","Type":"ContainerDied","Data":"4d659628f9fdc05e1c4c15c97690e5cdad56e199413b267a4a7922b8d68acae9"}
Sep 15 12:17:29.283577 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:29.283238    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3998f670-7bd777cff9-hqjsv" event={"ID":"14271cdb-fcab-4c96-a148-560660fc9ad4","Type":"ContainerDied","Data":"097fcddc5470dd451327bb1ecc252bf2287e5dee2caaa6b1703eead4ed95e946"}
Sep 15 12:17:29.283577 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:29.283288    2555 scope.go:117] "RemoveContainer" containerID="4d659628f9fdc05e1c4c15c97690e5cdad56e199413b267a4a7922b8d68acae9"
Sep 15 12:17:29.284619 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:29.284596    2555 generic.go:356] "Generic (PLEG): container finished" podID="6fedb577-ed82-4376-8b8d-2b0fe330c66d" containerID="87521cba7a8c6127417ee5f1ef0e623e59d7c09f5786e7c91cd42b3ae9392183" exitCode=0
Sep 15 12:17:29.284739 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:29.284633    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3998f670-77755dc994-72bzl"
Sep 15 12:17:29.284739 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:29.284668    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3998f670-77755dc994-72bzl" event={"ID":"6fedb577-ed82-4376-8b8d-2b0fe330c66d","Type":"ContainerDied","Data":"87521cba7a8c6127417ee5f1ef0e623e59d7c09f5786e7c91cd42b3ae9392183"}
Sep 15 12:17:29.284739 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:29.284719    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3998f670-77755dc994-72bzl" event={"ID":"6fedb577-ed82-4376-8b8d-2b0fe330c66d","Type":"ContainerDied","Data":"b463ff5c58010bae6dc05531eaafd3cf4e0c7896bf46ab63af7b0ba573c5c8d8"}
Sep 15 12:17:29.292966 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:29.292950    2555 scope.go:117] "RemoveContainer" containerID="4d659628f9fdc05e1c4c15c97690e5cdad56e199413b267a4a7922b8d68acae9"
Sep 15 12:17:29.293239 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:17:29.293223    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d659628f9fdc05e1c4c15c97690e5cdad56e199413b267a4a7922b8d68acae9\": container with ID starting with 4d659628f9fdc05e1c4c15c97690e5cdad56e199413b267a4a7922b8d68acae9 not found: ID does not exist" containerID="4d659628f9fdc05e1c4c15c97690e5cdad56e199413b267a4a7922b8d68acae9"
Sep 15 12:17:29.293329 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:29.293245    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d659628f9fdc05e1c4c15c97690e5cdad56e199413b267a4a7922b8d68acae9"} err="failed to get container status \"4d659628f9fdc05e1c4c15c97690e5cdad56e199413b267a4a7922b8d68acae9\": rpc error: code = NotFound desc = could not find container \"4d659628f9fdc05e1c4c15c97690e5cdad56e199413b267a4a7922b8d68acae9\": container with ID starting with 4d659628f9fdc05e1c4c15c97690e5cdad56e199413b267a4a7922b8d68acae9 not found: ID does not exist"
Sep 15 12:17:29.293329 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:29.293283    2555 scope.go:117] "RemoveContainer" containerID="87521cba7a8c6127417ee5f1ef0e623e59d7c09f5786e7c91cd42b3ae9392183"
Sep 15 12:17:29.301739 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:29.301722    2555 scope.go:117] "RemoveContainer" containerID="87521cba7a8c6127417ee5f1ef0e623e59d7c09f5786e7c91cd42b3ae9392183"
Sep 15 12:17:29.302001 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:17:29.301987    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87521cba7a8c6127417ee5f1ef0e623e59d7c09f5786e7c91cd42b3ae9392183\": container with ID starting with 87521cba7a8c6127417ee5f1ef0e623e59d7c09f5786e7c91cd42b3ae9392183 not found: ID does not exist" containerID="87521cba7a8c6127417ee5f1ef0e623e59d7c09f5786e7c91cd42b3ae9392183"
Sep 15 12:17:29.302050 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:29.302009    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87521cba7a8c6127417ee5f1ef0e623e59d7c09f5786e7c91cd42b3ae9392183"} err="failed to get container status \"87521cba7a8c6127417ee5f1ef0e623e59d7c09f5786e7c91cd42b3ae9392183\": rpc error: code = NotFound desc = could not find container \"87521cba7a8c6127417ee5f1ef0e623e59d7c09f5786e7c91cd42b3ae9392183\": container with ID starting with 87521cba7a8c6127417ee5f1ef0e623e59d7c09f5786e7c91cd42b3ae9392183 not found: ID does not exist"
Sep 15 12:17:29.311063 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:29.311037    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3998f670-77755dc994-72bzl"]
Sep 15 12:17:29.322403 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:29.322377    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3998f670-77755dc994-72bzl"]
Sep 15 12:17:29.336917 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:29.336893    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3998f670-7bd777cff9-hqjsv"]
Sep 15 12:17:29.341250 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:29.341225    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3998f670-7bd777cff9-hqjsv"]
Sep 15 12:17:29.817114 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:29.817076    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14271cdb-fcab-4c96-a148-560660fc9ad4" path="/var/lib/kubelet/pods/14271cdb-fcab-4c96-a148-560660fc9ad4/volumes"
Sep 15 12:17:29.817595 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:29.817578    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6fedb577-ed82-4376-8b8d-2b0fe330c66d" path="/var/lib/kubelet/pods/6fedb577-ed82-4376-8b8d-2b0fe330c66d/volumes"
Sep 15 12:17:29.817876 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:29.817864    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec44f139-4b80-4844-bfca-3bef562b0419" path="/var/lib/kubelet/pods/ec44f139-4b80-4844-bfca-3bef562b0419/volumes"
Sep 15 12:17:30.506201 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:30.506171    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-66020bb6-openshift-default-5868596bc6-hbrkq"]
Sep 15 12:17:30.506669 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:30.506651    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fedb577-ed82-4376-8b8d-2b0fe330c66d" containerName="payload-pre-processing"
Sep 15 12:17:30.506713 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:30.506672    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fedb577-ed82-4376-8b8d-2b0fe330c66d" containerName="payload-pre-processing"
Sep 15 12:17:30.506713 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:30.506692    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14271cdb-fcab-4c96-a148-560660fc9ad4" containerName="payload-processing"
Sep 15 12:17:30.506713 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:30.506701    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="14271cdb-fcab-4c96-a148-560660fc9ad4" containerName="payload-processing"
Sep 15 12:17:30.506812 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:30.506770    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="14271cdb-fcab-4c96-a148-560660fc9ad4" containerName="payload-processing"
Sep 15 12:17:30.506812 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:30.506780    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fedb577-ed82-4376-8b8d-2b0fe330c66d" containerName="payload-pre-processing"
Sep 15 12:17:30.511152 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:30.511133    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-66020bb6-openshift-default-5868596bc6-hbrkq"
Sep 15 12:17:30.514127 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:30.514109    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-66020bb6-openshift-default-dockercfg-bxspv\""
Sep 15 12:17:30.524116 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:30.524093    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-66020bb6-openshift-default-5868596bc6-hbrkq"]
Sep 15 12:17:30.609631 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:30.609590    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0db6fd2c-bba0-469c-a162-d1245e1c666f-workload-socket\") pod \"e2e-derive-66020bb6-openshift-default-5868596bc6-hbrkq\" (UID: \"0db6fd2c-bba0-469c-a162-d1245e1c666f\") " pod="openshift-ingress/e2e-derive-66020bb6-openshift-default-5868596bc6-hbrkq"
Sep 15 12:17:30.609785 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:30.609651    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0db6fd2c-bba0-469c-a162-d1245e1c666f-istiod-ca-cert\") pod \"e2e-derive-66020bb6-openshift-default-5868596bc6-hbrkq\" (UID: \"0db6fd2c-bba0-469c-a162-d1245e1c666f\") " pod="openshift-ingress/e2e-derive-66020bb6-openshift-default-5868596bc6-hbrkq"
Sep 15 12:17:30.609785 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:30.609704    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0db6fd2c-bba0-469c-a162-d1245e1c666f-workload-certs\") pod \"e2e-derive-66020bb6-openshift-default-5868596bc6-hbrkq\" (UID: \"0db6fd2c-bba0-469c-a162-d1245e1c666f\") " pod="openshift-ingress/e2e-derive-66020bb6-openshift-default-5868596bc6-hbrkq"
Sep 15 12:17:30.609785 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:30.609732    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0db6fd2c-bba0-469c-a162-d1245e1c666f-istio-envoy\") pod \"e2e-derive-66020bb6-openshift-default-5868596bc6-hbrkq\" (UID: \"0db6fd2c-bba0-469c-a162-d1245e1c666f\") " pod="openshift-ingress/e2e-derive-66020bb6-openshift-default-5868596bc6-hbrkq"
Sep 15 12:17:30.609913 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:30.609784    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0db6fd2c-bba0-469c-a162-d1245e1c666f-istio-podinfo\") pod \"e2e-derive-66020bb6-openshift-default-5868596bc6-hbrkq\" (UID: \"0db6fd2c-bba0-469c-a162-d1245e1c666f\") " pod="openshift-ingress/e2e-derive-66020bb6-openshift-default-5868596bc6-hbrkq"
Sep 15 12:17:30.609913 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:30.609812    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwqdd\" (UniqueName: \"kubernetes.io/projected/0db6fd2c-bba0-469c-a162-d1245e1c666f-kube-api-access-jwqdd\") pod \"e2e-derive-66020bb6-openshift-default-5868596bc6-hbrkq\" (UID: \"0db6fd2c-bba0-469c-a162-d1245e1c666f\") " pod="openshift-ingress/e2e-derive-66020bb6-openshift-default-5868596bc6-hbrkq"
Sep 15 12:17:30.609913 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:30.609872    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0db6fd2c-bba0-469c-a162-d1245e1c666f-istio-token\") pod \"e2e-derive-66020bb6-openshift-default-5868596bc6-hbrkq\" (UID: \"0db6fd2c-bba0-469c-a162-d1245e1c666f\") " pod="openshift-ingress/e2e-derive-66020bb6-openshift-default-5868596bc6-hbrkq"
Sep 15 12:17:30.610006 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:30.609948    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0db6fd2c-bba0-469c-a162-d1245e1c666f-credential-socket\") pod \"e2e-derive-66020bb6-openshift-default-5868596bc6-hbrkq\" (UID: \"0db6fd2c-bba0-469c-a162-d1245e1c666f\") " pod="openshift-ingress/e2e-derive-66020bb6-openshift-default-5868596bc6-hbrkq"
Sep 15 12:17:30.610006 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:30.609977    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0db6fd2c-bba0-469c-a162-d1245e1c666f-istio-data\") pod \"e2e-derive-66020bb6-openshift-default-5868596bc6-hbrkq\" (UID: \"0db6fd2c-bba0-469c-a162-d1245e1c666f\") " pod="openshift-ingress/e2e-derive-66020bb6-openshift-default-5868596bc6-hbrkq"
Sep 15 12:17:30.711321 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:30.711284    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0db6fd2c-bba0-469c-a162-d1245e1c666f-workload-certs\") pod \"e2e-derive-66020bb6-openshift-default-5868596bc6-hbrkq\" (UID: \"0db6fd2c-bba0-469c-a162-d1245e1c666f\") " pod="openshift-ingress/e2e-derive-66020bb6-openshift-default-5868596bc6-hbrkq"
Sep 15 12:17:30.711512 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:30.711331    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0db6fd2c-bba0-469c-a162-d1245e1c666f-istio-envoy\") pod \"e2e-derive-66020bb6-openshift-default-5868596bc6-hbrkq\" (UID: \"0db6fd2c-bba0-469c-a162-d1245e1c666f\") " pod="openshift-ingress/e2e-derive-66020bb6-openshift-default-5868596bc6-hbrkq"
Sep 15 12:17:30.711512 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:30.711371    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0db6fd2c-bba0-469c-a162-d1245e1c666f-istio-podinfo\") pod \"e2e-derive-66020bb6-openshift-default-5868596bc6-hbrkq\" (UID: \"0db6fd2c-bba0-469c-a162-d1245e1c666f\") " pod="openshift-ingress/e2e-derive-66020bb6-openshift-default-5868596bc6-hbrkq"
Sep 15 12:17:30.711512 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:30.711400    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwqdd\" (UniqueName: \"kubernetes.io/projected/0db6fd2c-bba0-469c-a162-d1245e1c666f-kube-api-access-jwqdd\") pod \"e2e-derive-66020bb6-openshift-default-5868596bc6-hbrkq\" (UID: \"0db6fd2c-bba0-469c-a162-d1245e1c666f\") " pod="openshift-ingress/e2e-derive-66020bb6-openshift-default-5868596bc6-hbrkq"
Sep 15 12:17:30.711512 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:30.711431    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0db6fd2c-bba0-469c-a162-d1245e1c666f-istio-token\") pod \"e2e-derive-66020bb6-openshift-default-5868596bc6-hbrkq\" (UID: \"0db6fd2c-bba0-469c-a162-d1245e1c666f\") " pod="openshift-ingress/e2e-derive-66020bb6-openshift-default-5868596bc6-hbrkq"
Sep 15 12:17:30.711512 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:30.711486    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0db6fd2c-bba0-469c-a162-d1245e1c666f-credential-socket\") pod \"e2e-derive-66020bb6-openshift-default-5868596bc6-hbrkq\" (UID: \"0db6fd2c-bba0-469c-a162-d1245e1c666f\") " pod="openshift-ingress/e2e-derive-66020bb6-openshift-default-5868596bc6-hbrkq"
Sep 15 12:17:30.711512 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:30.711502    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0db6fd2c-bba0-469c-a162-d1245e1c666f-istio-data\") pod \"e2e-derive-66020bb6-openshift-default-5868596bc6-hbrkq\" (UID: \"0db6fd2c-bba0-469c-a162-d1245e1c666f\") " pod="openshift-ingress/e2e-derive-66020bb6-openshift-default-5868596bc6-hbrkq"
Sep 15 12:17:30.711864 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:30.711540    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0db6fd2c-bba0-469c-a162-d1245e1c666f-workload-socket\") pod \"e2e-derive-66020bb6-openshift-default-5868596bc6-hbrkq\" (UID: \"0db6fd2c-bba0-469c-a162-d1245e1c666f\") " pod="openshift-ingress/e2e-derive-66020bb6-openshift-default-5868596bc6-hbrkq"
Sep 15 12:17:30.711864 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:30.711597    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0db6fd2c-bba0-469c-a162-d1245e1c666f-istiod-ca-cert\") pod \"e2e-derive-66020bb6-openshift-default-5868596bc6-hbrkq\" (UID: \"0db6fd2c-bba0-469c-a162-d1245e1c666f\") " pod="openshift-ingress/e2e-derive-66020bb6-openshift-default-5868596bc6-hbrkq"
Sep 15 12:17:30.711864 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:30.711721    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0db6fd2c-bba0-469c-a162-d1245e1c666f-workload-certs\") pod \"e2e-derive-66020bb6-openshift-default-5868596bc6-hbrkq\" (UID: \"0db6fd2c-bba0-469c-a162-d1245e1c666f\") " pod="openshift-ingress/e2e-derive-66020bb6-openshift-default-5868596bc6-hbrkq"
Sep 15 12:17:30.711864 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:30.711831    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0db6fd2c-bba0-469c-a162-d1245e1c666f-istio-data\") pod \"e2e-derive-66020bb6-openshift-default-5868596bc6-hbrkq\" (UID: \"0db6fd2c-bba0-469c-a162-d1245e1c666f\") " pod="openshift-ingress/e2e-derive-66020bb6-openshift-default-5868596bc6-hbrkq"
Sep 15 12:17:30.712006 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:30.711880    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0db6fd2c-bba0-469c-a162-d1245e1c666f-credential-socket\") pod \"e2e-derive-66020bb6-openshift-default-5868596bc6-hbrkq\" (UID: \"0db6fd2c-bba0-469c-a162-d1245e1c666f\") " pod="openshift-ingress/e2e-derive-66020bb6-openshift-default-5868596bc6-hbrkq"
Sep 15 12:17:30.712006 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:30.711917    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0db6fd2c-bba0-469c-a162-d1245e1c666f-workload-socket\") pod \"e2e-derive-66020bb6-openshift-default-5868596bc6-hbrkq\" (UID: \"0db6fd2c-bba0-469c-a162-d1245e1c666f\") " pod="openshift-ingress/e2e-derive-66020bb6-openshift-default-5868596bc6-hbrkq"
Sep 15 12:17:30.712232 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:30.712211    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0db6fd2c-bba0-469c-a162-d1245e1c666f-istiod-ca-cert\") pod \"e2e-derive-66020bb6-openshift-default-5868596bc6-hbrkq\" (UID: \"0db6fd2c-bba0-469c-a162-d1245e1c666f\") " pod="openshift-ingress/e2e-derive-66020bb6-openshift-default-5868596bc6-hbrkq"
Sep 15 12:17:30.713817 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:30.713801    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0db6fd2c-bba0-469c-a162-d1245e1c666f-istio-envoy\") pod \"e2e-derive-66020bb6-openshift-default-5868596bc6-hbrkq\" (UID: \"0db6fd2c-bba0-469c-a162-d1245e1c666f\") " pod="openshift-ingress/e2e-derive-66020bb6-openshift-default-5868596bc6-hbrkq"
Sep 15 12:17:30.714154 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:30.714137    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0db6fd2c-bba0-469c-a162-d1245e1c666f-istio-podinfo\") pod \"e2e-derive-66020bb6-openshift-default-5868596bc6-hbrkq\" (UID: \"0db6fd2c-bba0-469c-a162-d1245e1c666f\") " pod="openshift-ingress/e2e-derive-66020bb6-openshift-default-5868596bc6-hbrkq"
Sep 15 12:17:30.723144 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:30.723119    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwqdd\" (UniqueName: \"kubernetes.io/projected/0db6fd2c-bba0-469c-a162-d1245e1c666f-kube-api-access-jwqdd\") pod \"e2e-derive-66020bb6-openshift-default-5868596bc6-hbrkq\" (UID: \"0db6fd2c-bba0-469c-a162-d1245e1c666f\") " pod="openshift-ingress/e2e-derive-66020bb6-openshift-default-5868596bc6-hbrkq"
Sep 15 12:17:30.725331 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:30.725307    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0db6fd2c-bba0-469c-a162-d1245e1c666f-istio-token\") pod \"e2e-derive-66020bb6-openshift-default-5868596bc6-hbrkq\" (UID: \"0db6fd2c-bba0-469c-a162-d1245e1c666f\") " pod="openshift-ingress/e2e-derive-66020bb6-openshift-default-5868596bc6-hbrkq"
Sep 15 12:17:30.821067 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:30.820970    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-66020bb6-openshift-default-5868596bc6-hbrkq"
Sep 15 12:17:31.102486 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:31.102454    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-66020bb6-openshift-default-5868596bc6-hbrkq"]
Sep 15 12:17:31.104081 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:17:31.104050    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0db6fd2c_bba0_469c_a162_d1245e1c666f.slice/crio-7924eff083aaddb39a294d265989001c9d364bcb291753f1acb4679fa015417f WatchSource:0}: Error finding container 7924eff083aaddb39a294d265989001c9d364bcb291753f1acb4679fa015417f: Status 404 returned error can't find the container with id 7924eff083aaddb39a294d265989001c9d364bcb291753f1acb4679fa015417f
Sep 15 12:17:31.106495 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:31.106463    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:17:31.106576 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:31.106531    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:17:31.106576 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:31.106557    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:17:31.299334 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:31.299299    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-66020bb6-openshift-default-5868596bc6-hbrkq" event={"ID":"0db6fd2c-bba0-469c-a162-d1245e1c666f","Type":"ContainerStarted","Data":"7301a14b5a92cb92452b1ee230c34f9d152258ceaba3c72fe03ce3a4ac2e3fd4"}
Sep 15 12:17:31.299537 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:31.299359    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-66020bb6-openshift-default-5868596bc6-hbrkq" event={"ID":"0db6fd2c-bba0-469c-a162-d1245e1c666f","Type":"ContainerStarted","Data":"7924eff083aaddb39a294d265989001c9d364bcb291753f1acb4679fa015417f"}
Sep 15 12:17:31.324357 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:31.324296    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-66020bb6-openshift-default-5868596bc6-hbrkq" podStartSLOduration=1.324278038 podStartE2EDuration="1.324278038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:17:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:17:31.322855623 +0000 UTC m=+908.113778158" watchObservedRunningTime="2026-09-15 12:17:31.324278038 +0000 UTC m=+908.115200547"
Sep 15 12:17:31.822116 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:31.822084    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-66020bb6-openshift-default-5868596bc6-hbrkq"
Sep 15 12:17:32.827045 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:32.827013    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-66020bb6-openshift-default-5868596bc6-hbrkq"
Sep 15 12:17:33.310851 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:33.310805    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-66020bb6-openshift-default-5868596bc6-hbrkq"
Sep 15 12:17:33.311761 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:33.311740    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-66020bb6-openshift-default-5868596bc6-hbrkq"
Sep 15 12:17:35.089719 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:35.089679    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-66020bb6-66dfccbc46-6zchx"]
Sep 15 12:17:35.094162 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:35.094129    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-66020bb6-66dfccbc46-6zchx"
Sep 15 12:17:35.097138 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:35.097110    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-66020bb6-dockercfg-nltf2\""
Sep 15 12:17:35.097294 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:35.097216    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-66020bb6\""
Sep 15 12:17:35.109363 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:35.109330    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-66020bb6-66dfccbc46-6zchx"]
Sep 15 12:17:35.256906 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:35.256871    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmsjm\" (UniqueName: \"kubernetes.io/projected/769bcc30-93b1-40ef-972d-94c9cdf5df74-kube-api-access-lmsjm\") pod \"payload-pre-processing-e2e-derive-66020bb6-66dfccbc46-6zchx\" (UID: \"769bcc30-93b1-40ef-972d-94c9cdf5df74\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-66020bb6-66dfccbc46-6zchx"
Sep 15 12:17:35.257073 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:35.256921    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/769bcc30-93b1-40ef-972d-94c9cdf5df74-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-66020bb6-66dfccbc46-6zchx\" (UID: \"769bcc30-93b1-40ef-972d-94c9cdf5df74\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-66020bb6-66dfccbc46-6zchx"
Sep 15 12:17:35.358166 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:35.358082    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmsjm\" (UniqueName: \"kubernetes.io/projected/769bcc30-93b1-40ef-972d-94c9cdf5df74-kube-api-access-lmsjm\") pod \"payload-pre-processing-e2e-derive-66020bb6-66dfccbc46-6zchx\" (UID: \"769bcc30-93b1-40ef-972d-94c9cdf5df74\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-66020bb6-66dfccbc46-6zchx"
Sep 15 12:17:35.358166 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:35.358128    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/769bcc30-93b1-40ef-972d-94c9cdf5df74-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-66020bb6-66dfccbc46-6zchx\" (UID: \"769bcc30-93b1-40ef-972d-94c9cdf5df74\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-66020bb6-66dfccbc46-6zchx"
Sep 15 12:17:35.358910 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:35.358890    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/769bcc30-93b1-40ef-972d-94c9cdf5df74-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-66020bb6-66dfccbc46-6zchx\" (UID: \"769bcc30-93b1-40ef-972d-94c9cdf5df74\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-66020bb6-66dfccbc46-6zchx"
Sep 15 12:17:35.367576 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:35.367547    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmsjm\" (UniqueName: \"kubernetes.io/projected/769bcc30-93b1-40ef-972d-94c9cdf5df74-kube-api-access-lmsjm\") pod \"payload-pre-processing-e2e-derive-66020bb6-66dfccbc46-6zchx\" (UID: \"769bcc30-93b1-40ef-972d-94c9cdf5df74\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-66020bb6-66dfccbc46-6zchx"
Sep 15 12:17:35.406430 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:35.406395    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-66020bb6-66dfccbc46-6zchx"
Sep 15 12:17:35.684204 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:35.684177    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-66020bb6-66dfccbc46-6zchx"]
Sep 15 12:17:35.686998 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:17:35.686971    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod769bcc30_93b1_40ef_972d_94c9cdf5df74.slice/crio-c35fb7f462e8d4722ecbd695e3870563e3e06d842bf42144f22b22ba1c4a2fc8 WatchSource:0}: Error finding container c35fb7f462e8d4722ecbd695e3870563e3e06d842bf42144f22b22ba1c4a2fc8: Status 404 returned error can't find the container with id c35fb7f462e8d4722ecbd695e3870563e3e06d842bf42144f22b22ba1c4a2fc8
Sep 15 12:17:36.333326 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:36.333292    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-66020bb6-66dfccbc46-6zchx" event={"ID":"769bcc30-93b1-40ef-972d-94c9cdf5df74","Type":"ContainerStarted","Data":"1af51167f987da69905c050506048cca21ed8a163db7bed41acd495053787ab2"}
Sep 15 12:17:36.333718 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:36.333348    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-66020bb6-66dfccbc46-6zchx" event={"ID":"769bcc30-93b1-40ef-972d-94c9cdf5df74","Type":"ContainerStarted","Data":"c35fb7f462e8d4722ecbd695e3870563e3e06d842bf42144f22b22ba1c4a2fc8"}
Sep 15 12:17:36.333718 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:36.333451    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-66020bb6-66dfccbc46-6zchx"
Sep 15 12:17:36.355101 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:36.355042    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-66020bb6-66dfccbc46-6zchx" podStartSLOduration=1.355024729 podStartE2EDuration="1.355024729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:17:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:17:36.352155109 +0000 UTC m=+913.143077683" watchObservedRunningTime="2026-09-15 12:17:36.355024729 +0000 UTC m=+913.145947239"
Sep 15 12:17:37.230821 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:37.230785    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-4a9453/test-model-4a9453-kserve-657c74dd4f-qbbxr"
Sep 15 12:17:37.231020 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:37.230841    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-4a9453/test-model-4a9453-kserve-657c74dd4f-qbbxr"
Sep 15 12:17:37.243555 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:37.243524    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-4a9453/test-model-4a9453-kserve-657c74dd4f-qbbxr"
Sep 15 12:17:37.350509 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:37.350474    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-4a9453/test-model-4a9453-kserve-657c74dd4f-qbbxr"
Sep 15 12:17:43.284792 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:43.284743    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0e10f0-bgrr8"]
Sep 15 12:17:43.288613 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:43.288595    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0e10f0-bgrr8"
Sep 15 12:17:43.291373 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:43.291349    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 12:17:43.291487 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:43.291386    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tbvkt\""
Sep 15 12:17:43.296947 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:43.296917    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0e10f0-bgrr8"]
Sep 15 12:17:43.340631 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:43.340595    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c20ec42-95af-4655-ad91-900dc3c63860-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-0e10f0-bgrr8\" (UID: \"6c20ec42-95af-4655-ad91-900dc3c63860\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0e10f0-bgrr8"
Sep 15 12:17:43.441755 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:43.441712    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c20ec42-95af-4655-ad91-900dc3c63860-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-0e10f0-bgrr8\" (UID: \"6c20ec42-95af-4655-ad91-900dc3c63860\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0e10f0-bgrr8"
Sep 15 12:17:43.442386 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:43.442366    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c20ec42-95af-4655-ad91-900dc3c63860-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-0e10f0-bgrr8\" (UID: \"6c20ec42-95af-4655-ad91-900dc3c63860\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0e10f0-bgrr8"
Sep 15 12:17:43.600608 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:43.600526    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0e10f0-bgrr8"
Sep 15 12:17:43.880531 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:43.880502    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0e10f0-bgrr8"]
Sep 15 12:17:43.881597 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:17:43.881571    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c20ec42_95af_4655_ad91_900dc3c63860.slice/crio-11535a95fd918fee17f80cb1ac6b8786fe0a5b8dad4acf0f2871063b9312e338 WatchSource:0}: Error finding container 11535a95fd918fee17f80cb1ac6b8786fe0a5b8dad4acf0f2871063b9312e338: Status 404 returned error can't find the container with id 11535a95fd918fee17f80cb1ac6b8786fe0a5b8dad4acf0f2871063b9312e338
Sep 15 12:17:44.380467 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:44.380427    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0e10f0-bgrr8" event={"ID":"6c20ec42-95af-4655-ad91-900dc3c63860","Type":"ContainerStarted","Data":"1b3a7f13e100104113ea1e090b5ec973c9422c65d8a8026e6ee250f11db2a294"}
Sep 15 12:17:44.380866 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:44.380489    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0e10f0-bgrr8" event={"ID":"6c20ec42-95af-4655-ad91-900dc3c63860","Type":"ContainerStarted","Data":"11535a95fd918fee17f80cb1ac6b8786fe0a5b8dad4acf0f2871063b9312e338"}
Sep 15 12:17:44.400256 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:44.400204    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0e10f0-bgrr8" podStartSLOduration=1.40018862 podStartE2EDuration="1.40018862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:17:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:17:44.39812201 +0000 UTC m=+921.189044547" watchObservedRunningTime="2026-09-15 12:17:44.40018862 +0000 UTC m=+921.191111128"
Sep 15 12:17:45.387906 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:45.387868    2555 generic.go:356] "Generic (PLEG): container finished" podID="6c20ec42-95af-4655-ad91-900dc3c63860" containerID="1b3a7f13e100104113ea1e090b5ec973c9422c65d8a8026e6ee250f11db2a294" exitCode=0
Sep 15 12:17:45.388515 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:45.387930    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0e10f0-bgrr8" event={"ID":"6c20ec42-95af-4655-ad91-900dc3c63860","Type":"ContainerDied","Data":"1b3a7f13e100104113ea1e090b5ec973c9422c65d8a8026e6ee250f11db2a294"}
Sep 15 12:17:46.526344 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:46.526318    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0e10f0-bgrr8"
Sep 15 12:17:46.575018 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:46.574978    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c20ec42-95af-4655-ad91-900dc3c63860-maas-api-service-ca\") pod \"6c20ec42-95af-4655-ad91-900dc3c63860\" (UID: \"6c20ec42-95af-4655-ad91-900dc3c63860\") "
Sep 15 12:17:46.575404 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:46.575382    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c20ec42-95af-4655-ad91-900dc3c63860-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6c20ec42-95af-4655-ad91-900dc3c63860" (UID: "6c20ec42-95af-4655-ad91-900dc3c63860"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:17:46.676177 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:46.676094    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c20ec42-95af-4655-ad91-900dc3c63860-maas-api-service-ca\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:17:47.323414 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:47.323379    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-66020bb6-z5hcj"]
Sep 15 12:17:47.323762 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:47.323750    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c20ec42-95af-4655-ad91-900dc3c63860" containerName="revoke-keys"
Sep 15 12:17:47.323817 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:47.323764    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c20ec42-95af-4655-ad91-900dc3c63860" containerName="revoke-keys"
Sep 15 12:17:47.323851 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:47.323834    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c20ec42-95af-4655-ad91-900dc3c63860" containerName="revoke-keys"
Sep 15 12:17:47.326887 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:47.326866    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-66020bb6-z5hcj"
Sep 15 12:17:47.339756 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:47.339726    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-66020bb6-z5hcj"]
Sep 15 12:17:47.341048 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:47.341025    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-66020bb6-66dfccbc46-6zchx"
Sep 15 12:17:47.383828 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:47.383789    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/629b89b3-47fa-491b-b604-d43dc810f667-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-66020bb6-z5hcj\" (UID: \"629b89b3-47fa-491b-b604-d43dc810f667\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-66020bb6-z5hcj"
Sep 15 12:17:47.403144 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:47.403118    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0e10f0-bgrr8"
Sep 15 12:17:47.403291 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:47.403156    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0e10f0-bgrr8" event={"ID":"6c20ec42-95af-4655-ad91-900dc3c63860","Type":"ContainerDied","Data":"11535a95fd918fee17f80cb1ac6b8786fe0a5b8dad4acf0f2871063b9312e338"}
Sep 15 12:17:47.403291 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:47.403220    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="11535a95fd918fee17f80cb1ac6b8786fe0a5b8dad4acf0f2871063b9312e338"
Sep 15 12:17:47.485437 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:47.485401    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/629b89b3-47fa-491b-b604-d43dc810f667-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-66020bb6-z5hcj\" (UID: \"629b89b3-47fa-491b-b604-d43dc810f667\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-66020bb6-z5hcj"
Sep 15 12:17:47.486046 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:47.486026    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/629b89b3-47fa-491b-b604-d43dc810f667-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-66020bb6-z5hcj\" (UID: \"629b89b3-47fa-491b-b604-d43dc810f667\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-66020bb6-z5hcj"
Sep 15 12:17:47.637743 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:47.637705    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-66020bb6-z5hcj"
Sep 15 12:17:47.904373 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:47.904285    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5gkxs"]
Sep 15 12:17:47.916971 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:47.916946    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-66020bb6-z5hcj"]
Sep 15 12:17:47.919030 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:17:47.918998    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod629b89b3_47fa_491b_b604_d43dc810f667.slice/crio-ad7e276d7c0d6753edf04491f3180b4acbb5927f038dffed06b19833cfd8226c WatchSource:0}: Error finding container ad7e276d7c0d6753edf04491f3180b4acbb5927f038dffed06b19833cfd8226c: Status 404 returned error can't find the container with id ad7e276d7c0d6753edf04491f3180b4acbb5927f038dffed06b19833cfd8226c
Sep 15 12:17:48.412136 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:48.412099    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-66020bb6-z5hcj" event={"ID":"629b89b3-47fa-491b-b604-d43dc810f667","Type":"ContainerStarted","Data":"df2c9b99621377ebca9ee46cddf8ae474025c168d5523aff4811cd7e58462c5e"}
Sep 15 12:17:48.412343 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:48.412154    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-66020bb6-z5hcj" event={"ID":"629b89b3-47fa-491b-b604-d43dc810f667","Type":"ContainerStarted","Data":"ad7e276d7c0d6753edf04491f3180b4acbb5927f038dffed06b19833cfd8226c"}
Sep 15 12:17:48.442341 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:48.442206    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-66020bb6-z5hcj" podStartSLOduration=1.442184972 podStartE2EDuration="1.442184972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:17:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:17:48.440044428 +0000 UTC m=+925.230966960" watchObservedRunningTime="2026-09-15 12:17:48.442184972 +0000 UTC m=+925.233107483"
Sep 15 12:17:49.419683 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:49.419646    2555 generic.go:356] "Generic (PLEG): container finished" podID="629b89b3-47fa-491b-b604-d43dc810f667" containerID="df2c9b99621377ebca9ee46cddf8ae474025c168d5523aff4811cd7e58462c5e" exitCode=0
Sep 15 12:17:49.420078 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:49.419692    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-66020bb6-z5hcj" event={"ID":"629b89b3-47fa-491b-b604-d43dc810f667","Type":"ContainerDied","Data":"df2c9b99621377ebca9ee46cddf8ae474025c168d5523aff4811cd7e58462c5e"}
Sep 15 12:17:50.599023 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:50.598995    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-66020bb6-z5hcj"
Sep 15 12:17:50.714217 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:50.714124    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/629b89b3-47fa-491b-b604-d43dc810f667-maas-api-service-ca\") pod \"629b89b3-47fa-491b-b604-d43dc810f667\" (UID: \"629b89b3-47fa-491b-b604-d43dc810f667\") "
Sep 15 12:17:50.714564 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:50.714542    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/629b89b3-47fa-491b-b604-d43dc810f667-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "629b89b3-47fa-491b-b604-d43dc810f667" (UID: "629b89b3-47fa-491b-b604-d43dc810f667"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:17:50.815387 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:50.815346    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/629b89b3-47fa-491b-b604-d43dc810f667-maas-api-service-ca\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:17:51.431633 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:51.431602    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-66020bb6-z5hcj"
Sep 15 12:17:51.431802 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:51.431629    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-66020bb6-z5hcj" event={"ID":"629b89b3-47fa-491b-b604-d43dc810f667","Type":"ContainerDied","Data":"ad7e276d7c0d6753edf04491f3180b4acbb5927f038dffed06b19833cfd8226c"}
Sep 15 12:17:51.431802 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:51.431689    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad7e276d7c0d6753edf04491f3180b4acbb5927f038dffed06b19833cfd8226c"
Sep 15 12:17:54.185589 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:54.185551    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-0e10f0-859b7f7d9d-hxr92"]
Sep 15 12:17:54.186110 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:54.186020    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-0e10f0-859b7f7d9d-hxr92" podUID="398ac422-ed36-46db-bcdd-cf23226c2110" containerName="payload-pre-processing" containerID="cri-o://d34b9c4567f44b838df1bdb54d25a5f991714ac16f50bde089164939ec190548" gracePeriod=30
Sep 15 12:17:54.366873 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:54.366835    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0e10f0-bgrr8"]
Sep 15 12:17:54.376870 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:54.376794    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0e10f0-bgrr8"]
Sep 15 12:17:54.478831 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:54.478797    2555 generic.go:356] "Generic (PLEG): container finished" podID="398ac422-ed36-46db-bcdd-cf23226c2110" containerID="d34b9c4567f44b838df1bdb54d25a5f991714ac16f50bde089164939ec190548" exitCode=0
Sep 15 12:17:54.478979 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:54.478875    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-0e10f0-859b7f7d9d-hxr92" event={"ID":"398ac422-ed36-46db-bcdd-cf23226c2110","Type":"ContainerDied","Data":"d34b9c4567f44b838df1bdb54d25a5f991714ac16f50bde089164939ec190548"}
Sep 15 12:17:54.494953 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:54.494932    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-0e10f0-859b7f7d9d-hxr92"
Sep 15 12:17:54.570286 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:54.564427    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4hdcn\" (UniqueName: \"kubernetes.io/projected/398ac422-ed36-46db-bcdd-cf23226c2110-kube-api-access-4hdcn\") pod \"398ac422-ed36-46db-bcdd-cf23226c2110\" (UID: \"398ac422-ed36-46db-bcdd-cf23226c2110\") "
Sep 15 12:17:54.570286 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:54.564603    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/398ac422-ed36-46db-bcdd-cf23226c2110-plugins-config-volume\") pod \"398ac422-ed36-46db-bcdd-cf23226c2110\" (UID: \"398ac422-ed36-46db-bcdd-cf23226c2110\") "
Sep 15 12:17:54.574572 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:54.574529    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/398ac422-ed36-46db-bcdd-cf23226c2110-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "398ac422-ed36-46db-bcdd-cf23226c2110" (UID: "398ac422-ed36-46db-bcdd-cf23226c2110"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:17:54.574791 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:54.574757    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/398ac422-ed36-46db-bcdd-cf23226c2110-kube-api-access-4hdcn" (OuterVolumeSpecName: "kube-api-access-4hdcn") pod "398ac422-ed36-46db-bcdd-cf23226c2110" (UID: "398ac422-ed36-46db-bcdd-cf23226c2110"). InnerVolumeSpecName "kube-api-access-4hdcn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:17:54.666596 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:54.666521    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hdcn\" (UniqueName: \"kubernetes.io/projected/398ac422-ed36-46db-bcdd-cf23226c2110-kube-api-access-4hdcn\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:17:54.666596 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:54.666568    2555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/398ac422-ed36-46db-bcdd-cf23226c2110-plugins-config-volume\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:17:55.495142 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:55.495093    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-0e10f0-859b7f7d9d-hxr92" event={"ID":"398ac422-ed36-46db-bcdd-cf23226c2110","Type":"ContainerDied","Data":"a301251fe4ea01cfc8a18a5693214aa82b47c9c57558df7b1562a2d66a003643"}
Sep 15 12:17:55.495142 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:55.495127    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-0e10f0-859b7f7d9d-hxr92"
Sep 15 12:17:55.495733 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:55.495208    2555 scope.go:117] "RemoveContainer" containerID="d34b9c4567f44b838df1bdb54d25a5f991714ac16f50bde089164939ec190548"
Sep 15 12:17:55.521721 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:55.521677    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-0e10f0-859b7f7d9d-hxr92"]
Sep 15 12:17:55.529230 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:55.529192    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-0e10f0-859b7f7d9d-hxr92"]
Sep 15 12:17:55.817854 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:55.817771    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="398ac422-ed36-46db-bcdd-cf23226c2110" path="/var/lib/kubelet/pods/398ac422-ed36-46db-bcdd-cf23226c2110/volumes"
Sep 15 12:17:55.818142 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:55.818129    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c20ec42-95af-4655-ad91-900dc3c63860" path="/var/lib/kubelet/pods/6c20ec42-95af-4655-ad91-900dc3c63860/volumes"
Sep 15 12:17:58.514681 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:58.514649    2555 generic.go:356] "Generic (PLEG): container finished" podID="0db6fd2c-bba0-469c-a162-d1245e1c666f" containerID="7301a14b5a92cb92452b1ee230c34f9d152258ceaba3c72fe03ce3a4ac2e3fd4" exitCode=0
Sep 15 12:17:58.515110 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:58.514694    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-66020bb6-openshift-default-5868596bc6-hbrkq" event={"ID":"0db6fd2c-bba0-469c-a162-d1245e1c666f","Type":"ContainerDied","Data":"7301a14b5a92cb92452b1ee230c34f9d152258ceaba3c72fe03ce3a4ac2e3fd4"}
Sep 15 12:17:58.515110 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:58.515061    2555 scope.go:117] "RemoveContainer" containerID="7301a14b5a92cb92452b1ee230c34f9d152258ceaba3c72fe03ce3a4ac2e3fd4"
Sep 15 12:17:58.516189 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:58.516163    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:17:58.516242 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:58.516231    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:17:58.516291 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:58.516277    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:17:58.939081 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:58.938948    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-66020bb6-66dfccbc46-6zchx"]
Sep 15 12:17:58.939311 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:58.939209    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-66020bb6-66dfccbc46-6zchx" podUID="769bcc30-93b1-40ef-972d-94c9cdf5df74" containerName="payload-pre-processing" containerID="cri-o://1af51167f987da69905c050506048cca21ed8a163db7bed41acd495053787ab2" gracePeriod=30
Sep 15 12:17:59.146672 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:59.146605    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-66020bb6-z5hcj"]
Sep 15 12:17:59.152284 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:59.150225    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-66020bb6-z5hcj"]
Sep 15 12:17:59.239311 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:59.239281    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-66020bb6-66dfccbc46-6zchx"
Sep 15 12:17:59.284177 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:59.284144    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-66020bb6-openshift-default-5868596bc6-hbrkq"]
Sep 15 12:17:59.317912 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:59.317876    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lmsjm\" (UniqueName: \"kubernetes.io/projected/769bcc30-93b1-40ef-972d-94c9cdf5df74-kube-api-access-lmsjm\") pod \"769bcc30-93b1-40ef-972d-94c9cdf5df74\" (UID: \"769bcc30-93b1-40ef-972d-94c9cdf5df74\") "
Sep 15 12:17:59.318105 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:59.317941    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/769bcc30-93b1-40ef-972d-94c9cdf5df74-plugins-config-volume\") pod \"769bcc30-93b1-40ef-972d-94c9cdf5df74\" (UID: \"769bcc30-93b1-40ef-972d-94c9cdf5df74\") "
Sep 15 12:17:59.318632 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:59.318605    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/769bcc30-93b1-40ef-972d-94c9cdf5df74-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "769bcc30-93b1-40ef-972d-94c9cdf5df74" (UID: "769bcc30-93b1-40ef-972d-94c9cdf5df74"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:17:59.320685 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:59.320646    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/769bcc30-93b1-40ef-972d-94c9cdf5df74-kube-api-access-lmsjm" (OuterVolumeSpecName: "kube-api-access-lmsjm") pod "769bcc30-93b1-40ef-972d-94c9cdf5df74" (UID: "769bcc30-93b1-40ef-972d-94c9cdf5df74"). InnerVolumeSpecName "kube-api-access-lmsjm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:17:59.419103 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:59.419067    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmsjm\" (UniqueName: \"kubernetes.io/projected/769bcc30-93b1-40ef-972d-94c9cdf5df74-kube-api-access-lmsjm\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:17:59.419103 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:59.419096    2555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/769bcc30-93b1-40ef-972d-94c9cdf5df74-plugins-config-volume\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:17:59.529106 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:59.529011    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-66020bb6-openshift-default-5868596bc6-hbrkq" event={"ID":"0db6fd2c-bba0-469c-a162-d1245e1c666f","Type":"ContainerStarted","Data":"836c601d16e19e1d10ef967c8042826755360f85506a36c777a9f0372bbdabf9"}
Sep 15 12:17:59.529585 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:59.529193    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-66020bb6-openshift-default-5868596bc6-hbrkq" podUID="0db6fd2c-bba0-469c-a162-d1245e1c666f" containerName="istio-proxy" containerID="cri-o://836c601d16e19e1d10ef967c8042826755360f85506a36c777a9f0372bbdabf9" gracePeriod=30
Sep 15 12:17:59.534309 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:59.534255    2555 generic.go:356] "Generic (PLEG): container finished" podID="769bcc30-93b1-40ef-972d-94c9cdf5df74" containerID="1af51167f987da69905c050506048cca21ed8a163db7bed41acd495053787ab2" exitCode=0
Sep 15 12:17:59.534460 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:59.534361    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-66020bb6-66dfccbc46-6zchx" event={"ID":"769bcc30-93b1-40ef-972d-94c9cdf5df74","Type":"ContainerDied","Data":"1af51167f987da69905c050506048cca21ed8a163db7bed41acd495053787ab2"}
Sep 15 12:17:59.534528 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:59.534451    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-66020bb6-66dfccbc46-6zchx"
Sep 15 12:17:59.534528 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:59.534470    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-66020bb6-66dfccbc46-6zchx" event={"ID":"769bcc30-93b1-40ef-972d-94c9cdf5df74","Type":"ContainerDied","Data":"c35fb7f462e8d4722ecbd695e3870563e3e06d842bf42144f22b22ba1c4a2fc8"}
Sep 15 12:17:59.534528 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:59.534514    2555 scope.go:117] "RemoveContainer" containerID="1af51167f987da69905c050506048cca21ed8a163db7bed41acd495053787ab2"
Sep 15 12:17:59.545105 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:59.545090    2555 scope.go:117] "RemoveContainer" containerID="1af51167f987da69905c050506048cca21ed8a163db7bed41acd495053787ab2"
Sep 15 12:17:59.545393 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:17:59.545374    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1af51167f987da69905c050506048cca21ed8a163db7bed41acd495053787ab2\": container with ID starting with 1af51167f987da69905c050506048cca21ed8a163db7bed41acd495053787ab2 not found: ID does not exist" containerID="1af51167f987da69905c050506048cca21ed8a163db7bed41acd495053787ab2"
Sep 15 12:17:59.545472 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:59.545402    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1af51167f987da69905c050506048cca21ed8a163db7bed41acd495053787ab2"} err="failed to get container status \"1af51167f987da69905c050506048cca21ed8a163db7bed41acd495053787ab2\": rpc error: code = NotFound desc = could not find container \"1af51167f987da69905c050506048cca21ed8a163db7bed41acd495053787ab2\": container with ID starting with 1af51167f987da69905c050506048cca21ed8a163db7bed41acd495053787ab2 not found: ID does not exist"
Sep 15 12:17:59.586456 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:59.586424    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-66020bb6-66dfccbc46-6zchx"]
Sep 15 12:17:59.592044 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:59.592015    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-66020bb6-66dfccbc46-6zchx"]
Sep 15 12:17:59.727094 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:59.727058    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-0e10f0/autores-model-0e10f0-kserve-768dd655ff-mgfxr"]
Sep 15 12:17:59.819317 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:59.819214    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="629b89b3-47fa-491b-b604-d43dc810f667" path="/var/lib/kubelet/pods/629b89b3-47fa-491b-b604-d43dc810f667/volumes"
Sep 15 12:17:59.819757 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:17:59.819740    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="769bcc30-93b1-40ef-972d-94c9cdf5df74" path="/var/lib/kubelet/pods/769bcc30-93b1-40ef-972d-94c9cdf5df74/volumes"
Sep 15 12:18:03.310943 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:03.310901    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-66020bb6-openshift-default-5868596bc6-hbrkq"
Sep 15 12:18:07.065129 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:07.065084    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5gkxs"]
Sep 15 12:18:09.981722 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:09.981698    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-66020bb6-openshift-default-5868596bc6-hbrkq"
Sep 15 12:18:10.139647 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:10.139616    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0db6fd2c-bba0-469c-a162-d1245e1c666f-istio-data\") pod \"0db6fd2c-bba0-469c-a162-d1245e1c666f\" (UID: \"0db6fd2c-bba0-469c-a162-d1245e1c666f\") "
Sep 15 12:18:10.139842 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:10.139669    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0db6fd2c-bba0-469c-a162-d1245e1c666f-istiod-ca-cert\") pod \"0db6fd2c-bba0-469c-a162-d1245e1c666f\" (UID: \"0db6fd2c-bba0-469c-a162-d1245e1c666f\") "
Sep 15 12:18:10.139842 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:10.139696    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0db6fd2c-bba0-469c-a162-d1245e1c666f-workload-certs\") pod \"0db6fd2c-bba0-469c-a162-d1245e1c666f\" (UID: \"0db6fd2c-bba0-469c-a162-d1245e1c666f\") "
Sep 15 12:18:10.139842 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:10.139722    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0db6fd2c-bba0-469c-a162-d1245e1c666f-workload-socket\") pod \"0db6fd2c-bba0-469c-a162-d1245e1c666f\" (UID: \"0db6fd2c-bba0-469c-a162-d1245e1c666f\") "
Sep 15 12:18:10.139842 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:10.139808    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0db6fd2c-bba0-469c-a162-d1245e1c666f-istio-podinfo\") pod \"0db6fd2c-bba0-469c-a162-d1245e1c666f\" (UID: \"0db6fd2c-bba0-469c-a162-d1245e1c666f\") "
Sep 15 12:18:10.139842 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:10.139833    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0db6fd2c-bba0-469c-a162-d1245e1c666f-credential-socket\") pod \"0db6fd2c-bba0-469c-a162-d1245e1c666f\" (UID: \"0db6fd2c-bba0-469c-a162-d1245e1c666f\") "
Sep 15 12:18:10.140129 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:10.139859    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0db6fd2c-bba0-469c-a162-d1245e1c666f-istio-token\") pod \"0db6fd2c-bba0-469c-a162-d1245e1c666f\" (UID: \"0db6fd2c-bba0-469c-a162-d1245e1c666f\") "
Sep 15 12:18:10.140129 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:10.139936    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0db6fd2c-bba0-469c-a162-d1245e1c666f-istio-envoy\") pod \"0db6fd2c-bba0-469c-a162-d1245e1c666f\" (UID: \"0db6fd2c-bba0-469c-a162-d1245e1c666f\") "
Sep 15 12:18:10.140129 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:10.139945    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0db6fd2c-bba0-469c-a162-d1245e1c666f-istio-data" (OuterVolumeSpecName: "istio-data") pod "0db6fd2c-bba0-469c-a162-d1245e1c666f" (UID: "0db6fd2c-bba0-469c-a162-d1245e1c666f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:18:10.140129 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:10.139986    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jwqdd\" (UniqueName: \"kubernetes.io/projected/0db6fd2c-bba0-469c-a162-d1245e1c666f-kube-api-access-jwqdd\") pod \"0db6fd2c-bba0-469c-a162-d1245e1c666f\" (UID: \"0db6fd2c-bba0-469c-a162-d1245e1c666f\") "
Sep 15 12:18:10.140364 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:10.140126    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0db6fd2c-bba0-469c-a162-d1245e1c666f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0db6fd2c-bba0-469c-a162-d1245e1c666f" (UID: "0db6fd2c-bba0-469c-a162-d1245e1c666f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:18:10.140364 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:10.140198    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0db6fd2c-bba0-469c-a162-d1245e1c666f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0db6fd2c-bba0-469c-a162-d1245e1c666f" (UID: "0db6fd2c-bba0-469c-a162-d1245e1c666f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:18:10.140585 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:10.140542    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0db6fd2c-bba0-469c-a162-d1245e1c666f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0db6fd2c-bba0-469c-a162-d1245e1c666f" (UID: "0db6fd2c-bba0-469c-a162-d1245e1c666f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:18:10.140792 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:10.140758    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0db6fd2c-bba0-469c-a162-d1245e1c666f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0db6fd2c-bba0-469c-a162-d1245e1c666f" (UID: "0db6fd2c-bba0-469c-a162-d1245e1c666f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:18:10.148697 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:10.141106    2555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0db6fd2c-bba0-469c-a162-d1245e1c666f-istio-data\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:18:10.148697 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:10.141151    2555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0db6fd2c-bba0-469c-a162-d1245e1c666f-istiod-ca-cert\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:18:10.148697 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:10.141175    2555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0db6fd2c-bba0-469c-a162-d1245e1c666f-workload-socket\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:18:10.148697 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:10.144532    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0db6fd2c-bba0-469c-a162-d1245e1c666f-istio-token" (OuterVolumeSpecName: "istio-token") pod "0db6fd2c-bba0-469c-a162-d1245e1c666f" (UID: "0db6fd2c-bba0-469c-a162-d1245e1c666f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:18:10.148697 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:10.144715    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0db6fd2c-bba0-469c-a162-d1245e1c666f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0db6fd2c-bba0-469c-a162-d1245e1c666f" (UID: "0db6fd2c-bba0-469c-a162-d1245e1c666f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 12:18:10.149070 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:10.148720    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0db6fd2c-bba0-469c-a162-d1245e1c666f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0db6fd2c-bba0-469c-a162-d1245e1c666f" (UID: "0db6fd2c-bba0-469c-a162-d1245e1c666f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:18:10.149070 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:10.148886    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0db6fd2c-bba0-469c-a162-d1245e1c666f-kube-api-access-jwqdd" (OuterVolumeSpecName: "kube-api-access-jwqdd") pod "0db6fd2c-bba0-469c-a162-d1245e1c666f" (UID: "0db6fd2c-bba0-469c-a162-d1245e1c666f"). InnerVolumeSpecName "kube-api-access-jwqdd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:18:10.242563 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:10.242517    2555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0db6fd2c-bba0-469c-a162-d1245e1c666f-workload-certs\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:18:10.242563 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:10.242553    2555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0db6fd2c-bba0-469c-a162-d1245e1c666f-istio-podinfo\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:18:10.242563 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:10.242569    2555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0db6fd2c-bba0-469c-a162-d1245e1c666f-credential-socket\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:18:10.242810 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:10.242584    2555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0db6fd2c-bba0-469c-a162-d1245e1c666f-istio-token\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:18:10.242810 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:10.242592    2555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0db6fd2c-bba0-469c-a162-d1245e1c666f-istio-envoy\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:18:10.242810 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:10.242600    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwqdd\" (UniqueName: \"kubernetes.io/projected/0db6fd2c-bba0-469c-a162-d1245e1c666f-kube-api-access-jwqdd\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:18:10.610215 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:10.610126    2555 generic.go:356] "Generic (PLEG): container finished" podID="0db6fd2c-bba0-469c-a162-d1245e1c666f" containerID="836c601d16e19e1d10ef967c8042826755360f85506a36c777a9f0372bbdabf9" exitCode=0
Sep 15 12:18:10.610215 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:10.610186    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-66020bb6-openshift-default-5868596bc6-hbrkq"
Sep 15 12:18:10.610452 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:10.610213    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-66020bb6-openshift-default-5868596bc6-hbrkq" event={"ID":"0db6fd2c-bba0-469c-a162-d1245e1c666f","Type":"ContainerDied","Data":"836c601d16e19e1d10ef967c8042826755360f85506a36c777a9f0372bbdabf9"}
Sep 15 12:18:10.610452 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:10.610328    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-66020bb6-openshift-default-5868596bc6-hbrkq" event={"ID":"0db6fd2c-bba0-469c-a162-d1245e1c666f","Type":"ContainerDied","Data":"7924eff083aaddb39a294d265989001c9d364bcb291753f1acb4679fa015417f"}
Sep 15 12:18:10.610452 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:10.610357    2555 scope.go:117] "RemoveContainer" containerID="836c601d16e19e1d10ef967c8042826755360f85506a36c777a9f0372bbdabf9"
Sep 15 12:18:10.620389 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:10.620368    2555 scope.go:117] "RemoveContainer" containerID="7301a14b5a92cb92452b1ee230c34f9d152258ceaba3c72fe03ce3a4ac2e3fd4"
Sep 15 12:18:10.629183 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:10.629157    2555 scope.go:117] "RemoveContainer" containerID="836c601d16e19e1d10ef967c8042826755360f85506a36c777a9f0372bbdabf9"
Sep 15 12:18:10.629561 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:18:10.629538    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"836c601d16e19e1d10ef967c8042826755360f85506a36c777a9f0372bbdabf9\": container with ID starting with 836c601d16e19e1d10ef967c8042826755360f85506a36c777a9f0372bbdabf9 not found: ID does not exist" containerID="836c601d16e19e1d10ef967c8042826755360f85506a36c777a9f0372bbdabf9"
Sep 15 12:18:10.629663 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:10.629574    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"836c601d16e19e1d10ef967c8042826755360f85506a36c777a9f0372bbdabf9"} err="failed to get container status \"836c601d16e19e1d10ef967c8042826755360f85506a36c777a9f0372bbdabf9\": rpc error: code = NotFound desc = could not find container \"836c601d16e19e1d10ef967c8042826755360f85506a36c777a9f0372bbdabf9\": container with ID starting with 836c601d16e19e1d10ef967c8042826755360f85506a36c777a9f0372bbdabf9 not found: ID does not exist"
Sep 15 12:18:10.629663 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:10.629598    2555 scope.go:117] "RemoveContainer" containerID="7301a14b5a92cb92452b1ee230c34f9d152258ceaba3c72fe03ce3a4ac2e3fd4"
Sep 15 12:18:10.629926 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:18:10.629907    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7301a14b5a92cb92452b1ee230c34f9d152258ceaba3c72fe03ce3a4ac2e3fd4\": container with ID starting with 7301a14b5a92cb92452b1ee230c34f9d152258ceaba3c72fe03ce3a4ac2e3fd4 not found: ID does not exist" containerID="7301a14b5a92cb92452b1ee230c34f9d152258ceaba3c72fe03ce3a4ac2e3fd4"
Sep 15 12:18:10.629981 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:10.629932    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7301a14b5a92cb92452b1ee230c34f9d152258ceaba3c72fe03ce3a4ac2e3fd4"} err="failed to get container status \"7301a14b5a92cb92452b1ee230c34f9d152258ceaba3c72fe03ce3a4ac2e3fd4\": rpc error: code = NotFound desc = could not find container \"7301a14b5a92cb92452b1ee230c34f9d152258ceaba3c72fe03ce3a4ac2e3fd4\": container with ID starting with 7301a14b5a92cb92452b1ee230c34f9d152258ceaba3c72fe03ce3a4ac2e3fd4 not found: ID does not exist"
Sep 15 12:18:10.632342 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:10.632320    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-66020bb6-openshift-default-5868596bc6-hbrkq"]
Sep 15 12:18:10.635762 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:10.635739    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-66020bb6-openshift-default-5868596bc6-hbrkq"]
Sep 15 12:18:11.818096 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:11.818060    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0db6fd2c-bba0-469c-a162-d1245e1c666f" path="/var/lib/kubelet/pods/0db6fd2c-bba0-469c-a162-d1245e1c666f/volumes"
Sep 15 12:18:14.746243 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:14.746191    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-0e10f0/autores-model-0e10f0-kserve-768dd655ff-mgfxr" podUID="d7aaba80-442c-43a4-98a9-9726a96e35c1" containerName="main" containerID="cri-o://e3e7eb5330e7b013d95df35c14ef5cacc57840d25bd2f96c99e7d96969640576" gracePeriod=45
Sep 15 12:18:15.019225 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:15.019193    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-0e10f0/autores-model-0e10f0-kserve-768dd655ff-mgfxr"
Sep 15 12:18:15.079687 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:15.079656    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7aaba80-442c-43a4-98a9-9726a96e35c1-tmp-dir\") pod \"d7aaba80-442c-43a4-98a9-9726a96e35c1\" (UID: \"d7aaba80-442c-43a4-98a9-9726a96e35c1\") "
Sep 15 12:18:15.079853 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:15.079707    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7aaba80-442c-43a4-98a9-9726a96e35c1-tls-certs\") pod \"d7aaba80-442c-43a4-98a9-9726a96e35c1\" (UID: \"d7aaba80-442c-43a4-98a9-9726a96e35c1\") "
Sep 15 12:18:15.079853 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:15.079766    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4d9zf\" (UniqueName: \"kubernetes.io/projected/d7aaba80-442c-43a4-98a9-9726a96e35c1-kube-api-access-4d9zf\") pod \"d7aaba80-442c-43a4-98a9-9726a96e35c1\" (UID: \"d7aaba80-442c-43a4-98a9-9726a96e35c1\") "
Sep 15 12:18:15.079853 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:15.079808    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7aaba80-442c-43a4-98a9-9726a96e35c1-home\") pod \"d7aaba80-442c-43a4-98a9-9726a96e35c1\" (UID: \"d7aaba80-442c-43a4-98a9-9726a96e35c1\") "
Sep 15 12:18:15.079853 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:15.079832    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7aaba80-442c-43a4-98a9-9726a96e35c1-dshm\") pod \"d7aaba80-442c-43a4-98a9-9726a96e35c1\" (UID: \"d7aaba80-442c-43a4-98a9-9726a96e35c1\") "
Sep 15 12:18:15.080085 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:15.079923    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7aaba80-442c-43a4-98a9-9726a96e35c1-model-cache\") pod \"d7aaba80-442c-43a4-98a9-9726a96e35c1\" (UID: \"d7aaba80-442c-43a4-98a9-9726a96e35c1\") "
Sep 15 12:18:15.080085 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:15.080053    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7aaba80-442c-43a4-98a9-9726a96e35c1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d7aaba80-442c-43a4-98a9-9726a96e35c1" (UID: "d7aaba80-442c-43a4-98a9-9726a96e35c1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:18:15.080192 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:15.080162    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7aaba80-442c-43a4-98a9-9726a96e35c1-home" (OuterVolumeSpecName: "home") pod "d7aaba80-442c-43a4-98a9-9726a96e35c1" (UID: "d7aaba80-442c-43a4-98a9-9726a96e35c1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:18:15.080301 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:15.080281    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7aaba80-442c-43a4-98a9-9726a96e35c1-home\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:18:15.080372 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:15.080313    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7aaba80-442c-43a4-98a9-9726a96e35c1-model-cache" (OuterVolumeSpecName: "model-cache") pod "d7aaba80-442c-43a4-98a9-9726a96e35c1" (UID: "d7aaba80-442c-43a4-98a9-9726a96e35c1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:18:15.080372 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:15.080350    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7aaba80-442c-43a4-98a9-9726a96e35c1-tmp-dir\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:18:15.082197 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:15.082171    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7aaba80-442c-43a4-98a9-9726a96e35c1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d7aaba80-442c-43a4-98a9-9726a96e35c1" (UID: "d7aaba80-442c-43a4-98a9-9726a96e35c1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:18:15.082459 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:15.082435    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7aaba80-442c-43a4-98a9-9726a96e35c1-kube-api-access-4d9zf" (OuterVolumeSpecName: "kube-api-access-4d9zf") pod "d7aaba80-442c-43a4-98a9-9726a96e35c1" (UID: "d7aaba80-442c-43a4-98a9-9726a96e35c1"). InnerVolumeSpecName "kube-api-access-4d9zf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:18:15.082541 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:15.082462    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7aaba80-442c-43a4-98a9-9726a96e35c1-dshm" (OuterVolumeSpecName: "dshm") pod "d7aaba80-442c-43a4-98a9-9726a96e35c1" (UID: "d7aaba80-442c-43a4-98a9-9726a96e35c1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:18:15.180893 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:15.180848    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7aaba80-442c-43a4-98a9-9726a96e35c1-model-cache\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:18:15.180893 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:15.180890    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7aaba80-442c-43a4-98a9-9726a96e35c1-tls-certs\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:18:15.180893 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:15.180903    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4d9zf\" (UniqueName: \"kubernetes.io/projected/d7aaba80-442c-43a4-98a9-9726a96e35c1-kube-api-access-4d9zf\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:18:15.181147 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:15.180916    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7aaba80-442c-43a4-98a9-9726a96e35c1-dshm\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:18:15.640398 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:15.640365    2555 generic.go:356] "Generic (PLEG): container finished" podID="d7aaba80-442c-43a4-98a9-9726a96e35c1" containerID="e3e7eb5330e7b013d95df35c14ef5cacc57840d25bd2f96c99e7d96969640576" exitCode=0
Sep 15 12:18:15.640591 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:15.640418    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-0e10f0/autores-model-0e10f0-kserve-768dd655ff-mgfxr" event={"ID":"d7aaba80-442c-43a4-98a9-9726a96e35c1","Type":"ContainerDied","Data":"e3e7eb5330e7b013d95df35c14ef5cacc57840d25bd2f96c99e7d96969640576"}
Sep 15 12:18:15.640591 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:15.640452    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-0e10f0/autores-model-0e10f0-kserve-768dd655ff-mgfxr" event={"ID":"d7aaba80-442c-43a4-98a9-9726a96e35c1","Type":"ContainerDied","Data":"a5958ebd8afa6f396795f66a2858a65a5b4581dc711034afafba28fac432c9d3"}
Sep 15 12:18:15.640591 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:15.640469    2555 scope.go:117] "RemoveContainer" containerID="e3e7eb5330e7b013d95df35c14ef5cacc57840d25bd2f96c99e7d96969640576"
Sep 15 12:18:15.640591 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:15.640423    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-0e10f0/autores-model-0e10f0-kserve-768dd655ff-mgfxr"
Sep 15 12:18:15.650157 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:15.650133    2555 scope.go:117] "RemoveContainer" containerID="e3e7eb5330e7b013d95df35c14ef5cacc57840d25bd2f96c99e7d96969640576"
Sep 15 12:18:15.650463 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:18:15.650443    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3e7eb5330e7b013d95df35c14ef5cacc57840d25bd2f96c99e7d96969640576\": container with ID starting with e3e7eb5330e7b013d95df35c14ef5cacc57840d25bd2f96c99e7d96969640576 not found: ID does not exist" containerID="e3e7eb5330e7b013d95df35c14ef5cacc57840d25bd2f96c99e7d96969640576"
Sep 15 12:18:15.650512 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:15.650471    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3e7eb5330e7b013d95df35c14ef5cacc57840d25bd2f96c99e7d96969640576"} err="failed to get container status \"e3e7eb5330e7b013d95df35c14ef5cacc57840d25bd2f96c99e7d96969640576\": rpc error: code = NotFound desc = could not find container \"e3e7eb5330e7b013d95df35c14ef5cacc57840d25bd2f96c99e7d96969640576\": container with ID starting with e3e7eb5330e7b013d95df35c14ef5cacc57840d25bd2f96c99e7d96969640576 not found: ID does not exist"
Sep 15 12:18:15.665771 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:15.665731    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-0e10f0/autores-model-0e10f0-kserve-768dd655ff-mgfxr"]
Sep 15 12:18:15.669771 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:15.669743    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-0e10f0/autores-model-0e10f0-kserve-768dd655ff-mgfxr"]
Sep 15 12:18:15.817670 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:15.817634    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7aaba80-442c-43a4-98a9-9726a96e35c1" path="/var/lib/kubelet/pods/d7aaba80-442c-43a4-98a9-9726a96e35c1/volumes"
Sep 15 12:18:33.837206 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:33.837166    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-680229/premium-w0-680229-kserve-67cfbb7659-xt66w"]
Sep 15 12:18:33.837837 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:33.837714    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7aaba80-442c-43a4-98a9-9726a96e35c1" containerName="main"
Sep 15 12:18:33.837837 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:33.837737    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7aaba80-442c-43a4-98a9-9726a96e35c1" containerName="main"
Sep 15 12:18:33.837837 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:33.837754    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="769bcc30-93b1-40ef-972d-94c9cdf5df74" containerName="payload-pre-processing"
Sep 15 12:18:33.837837 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:33.837763    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="769bcc30-93b1-40ef-972d-94c9cdf5df74" containerName="payload-pre-processing"
Sep 15 12:18:33.837837 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:33.837777    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="398ac422-ed36-46db-bcdd-cf23226c2110" containerName="payload-pre-processing"
Sep 15 12:18:33.837837 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:33.837785    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="398ac422-ed36-46db-bcdd-cf23226c2110" containerName="payload-pre-processing"
Sep 15 12:18:33.837837 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:33.837800    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0db6fd2c-bba0-469c-a162-d1245e1c666f" containerName="istio-proxy"
Sep 15 12:18:33.837837 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:33.837809    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="0db6fd2c-bba0-469c-a162-d1245e1c666f" containerName="istio-proxy"
Sep 15 12:18:33.837837 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:33.837820    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0db6fd2c-bba0-469c-a162-d1245e1c666f" containerName="istio-proxy"
Sep 15 12:18:33.837837 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:33.837828    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="0db6fd2c-bba0-469c-a162-d1245e1c666f" containerName="istio-proxy"
Sep 15 12:18:33.837837 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:33.837840    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="629b89b3-47fa-491b-b604-d43dc810f667" containerName="revoke-keys"
Sep 15 12:18:33.838417 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:33.837848    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="629b89b3-47fa-491b-b604-d43dc810f667" containerName="revoke-keys"
Sep 15 12:18:33.838417 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:33.837959    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="769bcc30-93b1-40ef-972d-94c9cdf5df74" containerName="payload-pre-processing"
Sep 15 12:18:33.838417 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:33.837972    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="0db6fd2c-bba0-469c-a162-d1245e1c666f" containerName="istio-proxy"
Sep 15 12:18:33.838417 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:33.837982    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="0db6fd2c-bba0-469c-a162-d1245e1c666f" containerName="istio-proxy"
Sep 15 12:18:33.838417 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:33.837991    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7aaba80-442c-43a4-98a9-9726a96e35c1" containerName="main"
Sep 15 12:18:33.838417 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:33.838001    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="629b89b3-47fa-491b-b604-d43dc810f667" containerName="revoke-keys"
Sep 15 12:18:33.838417 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:33.838012    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="398ac422-ed36-46db-bcdd-cf23226c2110" containerName="payload-pre-processing"
Sep 15 12:18:33.842699 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:33.842676    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-680229/premium-w0-680229-kserve-67cfbb7659-xt66w"
Sep 15 12:18:33.846766 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:33.846741    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-680229\"/\"premium-w0-680229-kserve-self-signed-certs\""
Sep 15 12:18:33.857668 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:33.857638    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-680229/premium-w0-680229-kserve-67cfbb7659-xt66w"]
Sep 15 12:18:33.954755 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:33.954711    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4759dbe9-eeff-4ef1-aefc-59d9fa2ee19f-tls-certs\") pod \"premium-w0-680229-kserve-67cfbb7659-xt66w\" (UID: \"4759dbe9-eeff-4ef1-aefc-59d9fa2ee19f\") " pod="e2e-models-e2e-worker-w0-680229/premium-w0-680229-kserve-67cfbb7659-xt66w"
Sep 15 12:18:33.954755 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:33.954748    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4966\" (UniqueName: \"kubernetes.io/projected/4759dbe9-eeff-4ef1-aefc-59d9fa2ee19f-kube-api-access-p4966\") pod \"premium-w0-680229-kserve-67cfbb7659-xt66w\" (UID: \"4759dbe9-eeff-4ef1-aefc-59d9fa2ee19f\") " pod="e2e-models-e2e-worker-w0-680229/premium-w0-680229-kserve-67cfbb7659-xt66w"
Sep 15 12:18:33.954991 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:33.954774    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4759dbe9-eeff-4ef1-aefc-59d9fa2ee19f-home\") pod \"premium-w0-680229-kserve-67cfbb7659-xt66w\" (UID: \"4759dbe9-eeff-4ef1-aefc-59d9fa2ee19f\") " pod="e2e-models-e2e-worker-w0-680229/premium-w0-680229-kserve-67cfbb7659-xt66w"
Sep 15 12:18:33.954991 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:33.954920    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4759dbe9-eeff-4ef1-aefc-59d9fa2ee19f-dshm\") pod \"premium-w0-680229-kserve-67cfbb7659-xt66w\" (UID: \"4759dbe9-eeff-4ef1-aefc-59d9fa2ee19f\") " pod="e2e-models-e2e-worker-w0-680229/premium-w0-680229-kserve-67cfbb7659-xt66w"
Sep 15 12:18:33.954991 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:33.954988    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4759dbe9-eeff-4ef1-aefc-59d9fa2ee19f-tmp-dir\") pod \"premium-w0-680229-kserve-67cfbb7659-xt66w\" (UID: \"4759dbe9-eeff-4ef1-aefc-59d9fa2ee19f\") " pod="e2e-models-e2e-worker-w0-680229/premium-w0-680229-kserve-67cfbb7659-xt66w"
Sep 15 12:18:33.955106 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:33.955013    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4759dbe9-eeff-4ef1-aefc-59d9fa2ee19f-model-cache\") pod \"premium-w0-680229-kserve-67cfbb7659-xt66w\" (UID: \"4759dbe9-eeff-4ef1-aefc-59d9fa2ee19f\") " pod="e2e-models-e2e-worker-w0-680229/premium-w0-680229-kserve-67cfbb7659-xt66w"
Sep 15 12:18:34.055548 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:34.055514    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4759dbe9-eeff-4ef1-aefc-59d9fa2ee19f-dshm\") pod \"premium-w0-680229-kserve-67cfbb7659-xt66w\" (UID: \"4759dbe9-eeff-4ef1-aefc-59d9fa2ee19f\") " pod="e2e-models-e2e-worker-w0-680229/premium-w0-680229-kserve-67cfbb7659-xt66w"
Sep 15 12:18:34.055710 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:34.055578    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4759dbe9-eeff-4ef1-aefc-59d9fa2ee19f-tmp-dir\") pod \"premium-w0-680229-kserve-67cfbb7659-xt66w\" (UID: \"4759dbe9-eeff-4ef1-aefc-59d9fa2ee19f\") " pod="e2e-models-e2e-worker-w0-680229/premium-w0-680229-kserve-67cfbb7659-xt66w"
Sep 15 12:18:34.055710 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:34.055604    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4759dbe9-eeff-4ef1-aefc-59d9fa2ee19f-model-cache\") pod \"premium-w0-680229-kserve-67cfbb7659-xt66w\" (UID: \"4759dbe9-eeff-4ef1-aefc-59d9fa2ee19f\") " pod="e2e-models-e2e-worker-w0-680229/premium-w0-680229-kserve-67cfbb7659-xt66w"
Sep 15 12:18:34.055710 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:34.055640    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4759dbe9-eeff-4ef1-aefc-59d9fa2ee19f-tls-certs\") pod \"premium-w0-680229-kserve-67cfbb7659-xt66w\" (UID: \"4759dbe9-eeff-4ef1-aefc-59d9fa2ee19f\") " pod="e2e-models-e2e-worker-w0-680229/premium-w0-680229-kserve-67cfbb7659-xt66w"
Sep 15 12:18:34.055710 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:34.055661    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4966\" (UniqueName: \"kubernetes.io/projected/4759dbe9-eeff-4ef1-aefc-59d9fa2ee19f-kube-api-access-p4966\") pod \"premium-w0-680229-kserve-67cfbb7659-xt66w\" (UID: \"4759dbe9-eeff-4ef1-aefc-59d9fa2ee19f\") " pod="e2e-models-e2e-worker-w0-680229/premium-w0-680229-kserve-67cfbb7659-xt66w"
Sep 15 12:18:34.055710 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:34.055694    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4759dbe9-eeff-4ef1-aefc-59d9fa2ee19f-home\") pod \"premium-w0-680229-kserve-67cfbb7659-xt66w\" (UID: \"4759dbe9-eeff-4ef1-aefc-59d9fa2ee19f\") " pod="e2e-models-e2e-worker-w0-680229/premium-w0-680229-kserve-67cfbb7659-xt66w"
Sep 15 12:18:34.056095 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:34.056047    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4759dbe9-eeff-4ef1-aefc-59d9fa2ee19f-tmp-dir\") pod \"premium-w0-680229-kserve-67cfbb7659-xt66w\" (UID: \"4759dbe9-eeff-4ef1-aefc-59d9fa2ee19f\") " pod="e2e-models-e2e-worker-w0-680229/premium-w0-680229-kserve-67cfbb7659-xt66w"
Sep 15 12:18:34.056095 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:34.056069    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4759dbe9-eeff-4ef1-aefc-59d9fa2ee19f-model-cache\") pod \"premium-w0-680229-kserve-67cfbb7659-xt66w\" (UID: \"4759dbe9-eeff-4ef1-aefc-59d9fa2ee19f\") " pod="e2e-models-e2e-worker-w0-680229/premium-w0-680229-kserve-67cfbb7659-xt66w"
Sep 15 12:18:34.056347 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:34.056326    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4759dbe9-eeff-4ef1-aefc-59d9fa2ee19f-home\") pod \"premium-w0-680229-kserve-67cfbb7659-xt66w\" (UID: \"4759dbe9-eeff-4ef1-aefc-59d9fa2ee19f\") " pod="e2e-models-e2e-worker-w0-680229/premium-w0-680229-kserve-67cfbb7659-xt66w"
Sep 15 12:18:34.058150 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:34.058126    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4759dbe9-eeff-4ef1-aefc-59d9fa2ee19f-dshm\") pod \"premium-w0-680229-kserve-67cfbb7659-xt66w\" (UID: \"4759dbe9-eeff-4ef1-aefc-59d9fa2ee19f\") " pod="e2e-models-e2e-worker-w0-680229/premium-w0-680229-kserve-67cfbb7659-xt66w"
Sep 15 12:18:34.058353 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:34.058330    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4759dbe9-eeff-4ef1-aefc-59d9fa2ee19f-tls-certs\") pod \"premium-w0-680229-kserve-67cfbb7659-xt66w\" (UID: \"4759dbe9-eeff-4ef1-aefc-59d9fa2ee19f\") " pod="e2e-models-e2e-worker-w0-680229/premium-w0-680229-kserve-67cfbb7659-xt66w"
Sep 15 12:18:34.064441 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:34.064410    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4966\" (UniqueName: \"kubernetes.io/projected/4759dbe9-eeff-4ef1-aefc-59d9fa2ee19f-kube-api-access-p4966\") pod \"premium-w0-680229-kserve-67cfbb7659-xt66w\" (UID: \"4759dbe9-eeff-4ef1-aefc-59d9fa2ee19f\") " pod="e2e-models-e2e-worker-w0-680229/premium-w0-680229-kserve-67cfbb7659-xt66w"
Sep 15 12:18:34.152414 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:34.152368    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-680229/premium-w0-680229-kserve-67cfbb7659-xt66w"
Sep 15 12:18:34.430985 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:34.430954    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-680229/premium-w0-680229-kserve-67cfbb7659-xt66w"]
Sep 15 12:18:34.433197 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:18:34.433162    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4759dbe9_eeff_4ef1_aefc_59d9fa2ee19f.slice/crio-3b493510c8651e4a4db327726c571b83efab68a3609373d3a9388590a5bef7c4 WatchSource:0}: Error finding container 3b493510c8651e4a4db327726c571b83efab68a3609373d3a9388590a5bef7c4: Status 404 returned error can't find the container with id 3b493510c8651e4a4db327726c571b83efab68a3609373d3a9388590a5bef7c4
Sep 15 12:18:34.753889 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:34.753795    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-680229/premium-w0-680229-kserve-67cfbb7659-xt66w" event={"ID":"4759dbe9-eeff-4ef1-aefc-59d9fa2ee19f","Type":"ContainerStarted","Data":"4b485a3aba5d2d4cb91d8db2c65d7af0346e540c6a534563e049a071f3d6f75a"}
Sep 15 12:18:34.753889 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:34.753846    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-680229/premium-w0-680229-kserve-67cfbb7659-xt66w" event={"ID":"4759dbe9-eeff-4ef1-aefc-59d9fa2ee19f","Type":"ContainerStarted","Data":"3b493510c8651e4a4db327726c571b83efab68a3609373d3a9388590a5bef7c4"}
Sep 15 12:18:34.775598 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:34.775548    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-680229/premium-w0-680229-kserve-67cfbb7659-xt66w" podStartSLOduration=1.77553173 podStartE2EDuration="1.77553173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:18:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:18:34.774533525 +0000 UTC m=+971.565456055" watchObservedRunningTime="2026-09-15 12:18:34.77553173 +0000 UTC m=+971.566454222"
Sep 15 12:18:40.737648 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:40.737604    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5gkxs"]
Sep 15 12:18:44.152934 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:44.152890    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-680229/premium-w0-680229-kserve-67cfbb7659-xt66w"
Sep 15 12:18:44.153427 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:44.152946    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-680229/premium-w0-680229-kserve-67cfbb7659-xt66w"
Sep 15 12:18:44.166378 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:44.166343    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-680229/premium-w0-680229-kserve-67cfbb7659-xt66w"
Sep 15 12:18:44.819051 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:44.819014    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-680229/premium-w0-680229-kserve-67cfbb7659-xt66w"
Sep 15 12:18:45.038567 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:45.038530    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-9ba48f/premium-w4-9ba48f-kserve-5746585544-8rnhs"]
Sep 15 12:18:45.042227 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:45.042202    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9ba48f/premium-w4-9ba48f-kserve-5746585544-8rnhs"
Sep 15 12:18:45.045399 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:45.045372    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-9ba48f\"/\"premium-w4-9ba48f-kserve-self-signed-certs\""
Sep 15 12:18:45.067625 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:45.067587    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-9ba48f/premium-w4-9ba48f-kserve-5746585544-8rnhs"]
Sep 15 12:18:45.167296 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:45.167217    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e54c8cbe-3dad-4a5e-95d3-feeec2887d82-tmp-dir\") pod \"premium-w4-9ba48f-kserve-5746585544-8rnhs\" (UID: \"e54c8cbe-3dad-4a5e-95d3-feeec2887d82\") " pod="e2e-models-e2e-worker-w4-9ba48f/premium-w4-9ba48f-kserve-5746585544-8rnhs"
Sep 15 12:18:45.167296 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:45.167293    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e54c8cbe-3dad-4a5e-95d3-feeec2887d82-tls-certs\") pod \"premium-w4-9ba48f-kserve-5746585544-8rnhs\" (UID: \"e54c8cbe-3dad-4a5e-95d3-feeec2887d82\") " pod="e2e-models-e2e-worker-w4-9ba48f/premium-w4-9ba48f-kserve-5746585544-8rnhs"
Sep 15 12:18:45.167746 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:45.167422    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e54c8cbe-3dad-4a5e-95d3-feeec2887d82-model-cache\") pod \"premium-w4-9ba48f-kserve-5746585544-8rnhs\" (UID: \"e54c8cbe-3dad-4a5e-95d3-feeec2887d82\") " pod="e2e-models-e2e-worker-w4-9ba48f/premium-w4-9ba48f-kserve-5746585544-8rnhs"
Sep 15 12:18:45.167746 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:45.167444    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e54c8cbe-3dad-4a5e-95d3-feeec2887d82-home\") pod \"premium-w4-9ba48f-kserve-5746585544-8rnhs\" (UID: \"e54c8cbe-3dad-4a5e-95d3-feeec2887d82\") " pod="e2e-models-e2e-worker-w4-9ba48f/premium-w4-9ba48f-kserve-5746585544-8rnhs"
Sep 15 12:18:45.167746 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:45.167468    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e54c8cbe-3dad-4a5e-95d3-feeec2887d82-dshm\") pod \"premium-w4-9ba48f-kserve-5746585544-8rnhs\" (UID: \"e54c8cbe-3dad-4a5e-95d3-feeec2887d82\") " pod="e2e-models-e2e-worker-w4-9ba48f/premium-w4-9ba48f-kserve-5746585544-8rnhs"
Sep 15 12:18:45.167746 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:45.167482    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz64v\" (UniqueName: \"kubernetes.io/projected/e54c8cbe-3dad-4a5e-95d3-feeec2887d82-kube-api-access-lz64v\") pod \"premium-w4-9ba48f-kserve-5746585544-8rnhs\" (UID: \"e54c8cbe-3dad-4a5e-95d3-feeec2887d82\") " pod="e2e-models-e2e-worker-w4-9ba48f/premium-w4-9ba48f-kserve-5746585544-8rnhs"
Sep 15 12:18:45.268277 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:45.268227    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e54c8cbe-3dad-4a5e-95d3-feeec2887d82-model-cache\") pod \"premium-w4-9ba48f-kserve-5746585544-8rnhs\" (UID: \"e54c8cbe-3dad-4a5e-95d3-feeec2887d82\") " pod="e2e-models-e2e-worker-w4-9ba48f/premium-w4-9ba48f-kserve-5746585544-8rnhs"
Sep 15 12:18:45.268459 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:45.268330    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e54c8cbe-3dad-4a5e-95d3-feeec2887d82-home\") pod \"premium-w4-9ba48f-kserve-5746585544-8rnhs\" (UID: \"e54c8cbe-3dad-4a5e-95d3-feeec2887d82\") " pod="e2e-models-e2e-worker-w4-9ba48f/premium-w4-9ba48f-kserve-5746585544-8rnhs"
Sep 15 12:18:45.268459 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:45.268358    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e54c8cbe-3dad-4a5e-95d3-feeec2887d82-dshm\") pod \"premium-w4-9ba48f-kserve-5746585544-8rnhs\" (UID: \"e54c8cbe-3dad-4a5e-95d3-feeec2887d82\") " pod="e2e-models-e2e-worker-w4-9ba48f/premium-w4-9ba48f-kserve-5746585544-8rnhs"
Sep 15 12:18:45.268459 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:45.268374    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz64v\" (UniqueName: \"kubernetes.io/projected/e54c8cbe-3dad-4a5e-95d3-feeec2887d82-kube-api-access-lz64v\") pod \"premium-w4-9ba48f-kserve-5746585544-8rnhs\" (UID: \"e54c8cbe-3dad-4a5e-95d3-feeec2887d82\") " pod="e2e-models-e2e-worker-w4-9ba48f/premium-w4-9ba48f-kserve-5746585544-8rnhs"
Sep 15 12:18:45.268459 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:45.268413    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e54c8cbe-3dad-4a5e-95d3-feeec2887d82-tmp-dir\") pod \"premium-w4-9ba48f-kserve-5746585544-8rnhs\" (UID: \"e54c8cbe-3dad-4a5e-95d3-feeec2887d82\") " pod="e2e-models-e2e-worker-w4-9ba48f/premium-w4-9ba48f-kserve-5746585544-8rnhs"
Sep 15 12:18:45.268459 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:45.268441    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e54c8cbe-3dad-4a5e-95d3-feeec2887d82-tls-certs\") pod \"premium-w4-9ba48f-kserve-5746585544-8rnhs\" (UID: \"e54c8cbe-3dad-4a5e-95d3-feeec2887d82\") " pod="e2e-models-e2e-worker-w4-9ba48f/premium-w4-9ba48f-kserve-5746585544-8rnhs"
Sep 15 12:18:45.268744 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:45.268721    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e54c8cbe-3dad-4a5e-95d3-feeec2887d82-model-cache\") pod \"premium-w4-9ba48f-kserve-5746585544-8rnhs\" (UID: \"e54c8cbe-3dad-4a5e-95d3-feeec2887d82\") " pod="e2e-models-e2e-worker-w4-9ba48f/premium-w4-9ba48f-kserve-5746585544-8rnhs"
Sep 15 12:18:45.268787 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:45.268754    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e54c8cbe-3dad-4a5e-95d3-feeec2887d82-home\") pod \"premium-w4-9ba48f-kserve-5746585544-8rnhs\" (UID: \"e54c8cbe-3dad-4a5e-95d3-feeec2887d82\") " pod="e2e-models-e2e-worker-w4-9ba48f/premium-w4-9ba48f-kserve-5746585544-8rnhs"
Sep 15 12:18:45.268846 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:45.268826    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e54c8cbe-3dad-4a5e-95d3-feeec2887d82-tmp-dir\") pod \"premium-w4-9ba48f-kserve-5746585544-8rnhs\" (UID: \"e54c8cbe-3dad-4a5e-95d3-feeec2887d82\") " pod="e2e-models-e2e-worker-w4-9ba48f/premium-w4-9ba48f-kserve-5746585544-8rnhs"
Sep 15 12:18:45.270702 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:45.270680    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e54c8cbe-3dad-4a5e-95d3-feeec2887d82-dshm\") pod \"premium-w4-9ba48f-kserve-5746585544-8rnhs\" (UID: \"e54c8cbe-3dad-4a5e-95d3-feeec2887d82\") " pod="e2e-models-e2e-worker-w4-9ba48f/premium-w4-9ba48f-kserve-5746585544-8rnhs"
Sep 15 12:18:45.270869 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:45.270852    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e54c8cbe-3dad-4a5e-95d3-feeec2887d82-tls-certs\") pod \"premium-w4-9ba48f-kserve-5746585544-8rnhs\" (UID: \"e54c8cbe-3dad-4a5e-95d3-feeec2887d82\") " pod="e2e-models-e2e-worker-w4-9ba48f/premium-w4-9ba48f-kserve-5746585544-8rnhs"
Sep 15 12:18:45.277922 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:45.277892    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz64v\" (UniqueName: \"kubernetes.io/projected/e54c8cbe-3dad-4a5e-95d3-feeec2887d82-kube-api-access-lz64v\") pod \"premium-w4-9ba48f-kserve-5746585544-8rnhs\" (UID: \"e54c8cbe-3dad-4a5e-95d3-feeec2887d82\") " pod="e2e-models-e2e-worker-w4-9ba48f/premium-w4-9ba48f-kserve-5746585544-8rnhs"
Sep 15 12:18:45.352006 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:45.351960    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9ba48f/premium-w4-9ba48f-kserve-5746585544-8rnhs"
Sep 15 12:18:45.634211 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:45.634081    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-9ba48f/premium-w4-9ba48f-kserve-5746585544-8rnhs"]
Sep 15 12:18:45.637090 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:18:45.637058    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode54c8cbe_3dad_4a5e_95d3_feeec2887d82.slice/crio-a9fcbca4856c2486126e9d64317bbdfcc228083f2e4c039390063bc39b208f16 WatchSource:0}: Error finding container a9fcbca4856c2486126e9d64317bbdfcc228083f2e4c039390063bc39b208f16: Status 404 returned error can't find the container with id a9fcbca4856c2486126e9d64317bbdfcc228083f2e4c039390063bc39b208f16
Sep 15 12:18:45.818224 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:45.818191    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9ba48f/premium-w4-9ba48f-kserve-5746585544-8rnhs" event={"ID":"e54c8cbe-3dad-4a5e-95d3-feeec2887d82","Type":"ContainerStarted","Data":"9d20018093f0649e88ee94be1fc03948ccf0ef9f0ee73cdf95d92cbf429b4b1f"}
Sep 15 12:18:45.818224 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:45.818233    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9ba48f/premium-w4-9ba48f-kserve-5746585544-8rnhs" event={"ID":"e54c8cbe-3dad-4a5e-95d3-feeec2887d82","Type":"ContainerStarted","Data":"a9fcbca4856c2486126e9d64317bbdfcc228083f2e4c039390063bc39b208f16"}
Sep 15 12:18:45.849281 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:45.849209    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-9ba48f/premium-w4-9ba48f-kserve-5746585544-8rnhs" podStartSLOduration=0.849186074 podStartE2EDuration="849.186074ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:18:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:18:45.846246893 +0000 UTC m=+982.637169424" watchObservedRunningTime="2026-09-15 12:18:45.849186074 +0000 UTC m=+982.640108588"
Sep 15 12:18:47.675333 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:47.675298    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-0e10f0-openshift-default-5d9bcff6b6-86lrx"]
Sep 15 12:18:47.675727 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:47.675519    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-0e10f0-openshift-default-5d9bcff6b6-86lrx" podUID="bc1353ca-8864-4cc7-93af-1a53f57fc7ab" containerName="istio-proxy" containerID="cri-o://d321d233d3b3fd56cb7150af58f6ab96ff51c70b3df9b0086deff32420868855" gracePeriod=30
Sep 15 12:18:52.314282 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:52.314224    2555 patch_prober.go:28] interesting pod/e2e-autores-0e10f0-openshift-default-5d9bcff6b6-86lrx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 15 12:18:52.314651 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:52.314317    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-0e10f0-openshift-default-5d9bcff6b6-86lrx" podUID="bc1353ca-8864-4cc7-93af-1a53f57fc7ab" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 12:18:52.857410 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:52.857374    2555 generic.go:356] "Generic (PLEG): container finished" podID="bc1353ca-8864-4cc7-93af-1a53f57fc7ab" containerID="d321d233d3b3fd56cb7150af58f6ab96ff51c70b3df9b0086deff32420868855" exitCode=0
Sep 15 12:18:52.857568 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:52.857456    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-0e10f0-openshift-default-5d9bcff6b6-86lrx" event={"ID":"bc1353ca-8864-4cc7-93af-1a53f57fc7ab","Type":"ContainerDied","Data":"d321d233d3b3fd56cb7150af58f6ab96ff51c70b3df9b0086deff32420868855"}
Sep 15 12:18:52.857568 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:52.857559    2555 scope.go:117] "RemoveContainer" containerID="f331574f5a752a05d4b75cb1d7b2e8e134a2f87fb2aa62f396cff6d6a628f27f"
Sep 15 12:18:52.928764 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:52.928738    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-0e10f0-openshift-default-5d9bcff6b6-86lrx"
Sep 15 12:18:53.041947 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:53.041860    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc1353ca-8864-4cc7-93af-1a53f57fc7ab-istio-envoy\") pod \"bc1353ca-8864-4cc7-93af-1a53f57fc7ab\" (UID: \"bc1353ca-8864-4cc7-93af-1a53f57fc7ab\") "
Sep 15 12:18:53.041947 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:53.041947    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc1353ca-8864-4cc7-93af-1a53f57fc7ab-istio-data\") pod \"bc1353ca-8864-4cc7-93af-1a53f57fc7ab\" (UID: \"bc1353ca-8864-4cc7-93af-1a53f57fc7ab\") "
Sep 15 12:18:53.042189 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:53.041972    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hb5l4\" (UniqueName: \"kubernetes.io/projected/bc1353ca-8864-4cc7-93af-1a53f57fc7ab-kube-api-access-hb5l4\") pod \"bc1353ca-8864-4cc7-93af-1a53f57fc7ab\" (UID: \"bc1353ca-8864-4cc7-93af-1a53f57fc7ab\") "
Sep 15 12:18:53.042189 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:53.042006    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc1353ca-8864-4cc7-93af-1a53f57fc7ab-istio-token\") pod \"bc1353ca-8864-4cc7-93af-1a53f57fc7ab\" (UID: \"bc1353ca-8864-4cc7-93af-1a53f57fc7ab\") "
Sep 15 12:18:53.042189 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:53.042045    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc1353ca-8864-4cc7-93af-1a53f57fc7ab-workload-socket\") pod \"bc1353ca-8864-4cc7-93af-1a53f57fc7ab\" (UID: \"bc1353ca-8864-4cc7-93af-1a53f57fc7ab\") "
Sep 15 12:18:53.042189 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:53.042064    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc1353ca-8864-4cc7-93af-1a53f57fc7ab-istiod-ca-cert\") pod \"bc1353ca-8864-4cc7-93af-1a53f57fc7ab\" (UID: \"bc1353ca-8864-4cc7-93af-1a53f57fc7ab\") "
Sep 15 12:18:53.042189 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:53.042092    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc1353ca-8864-4cc7-93af-1a53f57fc7ab-istio-podinfo\") pod \"bc1353ca-8864-4cc7-93af-1a53f57fc7ab\" (UID: \"bc1353ca-8864-4cc7-93af-1a53f57fc7ab\") "
Sep 15 12:18:53.042189 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:53.042184    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc1353ca-8864-4cc7-93af-1a53f57fc7ab-credential-socket\") pod \"bc1353ca-8864-4cc7-93af-1a53f57fc7ab\" (UID: \"bc1353ca-8864-4cc7-93af-1a53f57fc7ab\") "
Sep 15 12:18:53.042564 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:53.042209    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc1353ca-8864-4cc7-93af-1a53f57fc7ab-workload-certs\") pod \"bc1353ca-8864-4cc7-93af-1a53f57fc7ab\" (UID: \"bc1353ca-8864-4cc7-93af-1a53f57fc7ab\") "
Sep 15 12:18:53.042564 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:53.042300    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc1353ca-8864-4cc7-93af-1a53f57fc7ab-istio-data" (OuterVolumeSpecName: "istio-data") pod "bc1353ca-8864-4cc7-93af-1a53f57fc7ab" (UID: "bc1353ca-8864-4cc7-93af-1a53f57fc7ab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:18:53.042564 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:53.042528    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc1353ca-8864-4cc7-93af-1a53f57fc7ab-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bc1353ca-8864-4cc7-93af-1a53f57fc7ab" (UID: "bc1353ca-8864-4cc7-93af-1a53f57fc7ab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:18:53.042564 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:53.042543    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc1353ca-8864-4cc7-93af-1a53f57fc7ab-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bc1353ca-8864-4cc7-93af-1a53f57fc7ab" (UID: "bc1353ca-8864-4cc7-93af-1a53f57fc7ab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:18:53.042778 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:53.042633    2555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc1353ca-8864-4cc7-93af-1a53f57fc7ab-istio-data\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:18:53.042778 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:53.042650    2555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc1353ca-8864-4cc7-93af-1a53f57fc7ab-workload-socket\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:18:53.042778 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:53.042664    2555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc1353ca-8864-4cc7-93af-1a53f57fc7ab-credential-socket\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:18:53.042778 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:53.042741    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc1353ca-8864-4cc7-93af-1a53f57fc7ab-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bc1353ca-8864-4cc7-93af-1a53f57fc7ab" (UID: "bc1353ca-8864-4cc7-93af-1a53f57fc7ab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:18:53.042778 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:53.042759    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc1353ca-8864-4cc7-93af-1a53f57fc7ab-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bc1353ca-8864-4cc7-93af-1a53f57fc7ab" (UID: "bc1353ca-8864-4cc7-93af-1a53f57fc7ab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:18:53.044413 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:53.044386    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc1353ca-8864-4cc7-93af-1a53f57fc7ab-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bc1353ca-8864-4cc7-93af-1a53f57fc7ab" (UID: "bc1353ca-8864-4cc7-93af-1a53f57fc7ab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:18:53.044541 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:53.044433    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc1353ca-8864-4cc7-93af-1a53f57fc7ab-kube-api-access-hb5l4" (OuterVolumeSpecName: "kube-api-access-hb5l4") pod "bc1353ca-8864-4cc7-93af-1a53f57fc7ab" (UID: "bc1353ca-8864-4cc7-93af-1a53f57fc7ab"). InnerVolumeSpecName "kube-api-access-hb5l4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:18:53.044541 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:53.044473    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc1353ca-8864-4cc7-93af-1a53f57fc7ab-istio-token" (OuterVolumeSpecName: "istio-token") pod "bc1353ca-8864-4cc7-93af-1a53f57fc7ab" (UID: "bc1353ca-8864-4cc7-93af-1a53f57fc7ab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:18:53.044541 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:53.044502    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bc1353ca-8864-4cc7-93af-1a53f57fc7ab-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bc1353ca-8864-4cc7-93af-1a53f57fc7ab" (UID: "bc1353ca-8864-4cc7-93af-1a53f57fc7ab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 12:18:53.143671 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:53.143619    2555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc1353ca-8864-4cc7-93af-1a53f57fc7ab-istiod-ca-cert\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:18:53.143671 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:53.143664    2555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc1353ca-8864-4cc7-93af-1a53f57fc7ab-istio-podinfo\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:18:53.143671 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:53.143673    2555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc1353ca-8864-4cc7-93af-1a53f57fc7ab-workload-certs\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:18:53.143671 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:53.143681    2555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc1353ca-8864-4cc7-93af-1a53f57fc7ab-istio-envoy\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:18:53.143671 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:53.143690    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hb5l4\" (UniqueName: \"kubernetes.io/projected/bc1353ca-8864-4cc7-93af-1a53f57fc7ab-kube-api-access-hb5l4\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:18:53.143995 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:53.143699    2555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc1353ca-8864-4cc7-93af-1a53f57fc7ab-istio-token\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:18:53.864599 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:53.864568    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-0e10f0-openshift-default-5d9bcff6b6-86lrx"
Sep 15 12:18:53.865072 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:53.864562    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-0e10f0-openshift-default-5d9bcff6b6-86lrx" event={"ID":"bc1353ca-8864-4cc7-93af-1a53f57fc7ab","Type":"ContainerDied","Data":"757482f65eed99af581adc479bb4858714083398038a930068e59a769cff0917"}
Sep 15 12:18:53.865072 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:53.864731    2555 scope.go:117] "RemoveContainer" containerID="d321d233d3b3fd56cb7150af58f6ab96ff51c70b3df9b0086deff32420868855"
Sep 15 12:18:53.901454 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:53.901414    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-0e10f0-openshift-default-5d9bcff6b6-86lrx"]
Sep 15 12:18:53.916051 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:53.916019    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-0e10f0-openshift-default-5d9bcff6b6-86lrx"]
Sep 15 12:18:55.352933 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:55.352900    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9ba48f/premium-w4-9ba48f-kserve-5746585544-8rnhs"
Sep 15 12:18:55.353357 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:55.352959    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-9ba48f/premium-w4-9ba48f-kserve-5746585544-8rnhs"
Sep 15 12:18:55.365752 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:55.365725    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-9ba48f/premium-w4-9ba48f-kserve-5746585544-8rnhs"
Sep 15 12:18:55.821999 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:55.821963    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc1353ca-8864-4cc7-93af-1a53f57fc7ab" path="/var/lib/kubelet/pods/bc1353ca-8864-4cc7-93af-1a53f57fc7ab/volumes"
Sep 15 12:18:55.888682 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:55.888647    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9ba48f/premium-w4-9ba48f-kserve-5746585544-8rnhs"
Sep 15 12:18:57.576026 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:57.575989    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-294781-bbcfb7766-2h6m9"]
Sep 15 12:18:57.576468 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:57.576394    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc1353ca-8864-4cc7-93af-1a53f57fc7ab" containerName="istio-proxy"
Sep 15 12:18:57.576468 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:57.576408    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc1353ca-8864-4cc7-93af-1a53f57fc7ab" containerName="istio-proxy"
Sep 15 12:18:57.576553 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:57.576480    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc1353ca-8864-4cc7-93af-1a53f57fc7ab" containerName="istio-proxy"
Sep 15 12:18:57.576553 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:57.576489    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc1353ca-8864-4cc7-93af-1a53f57fc7ab" containerName="istio-proxy"
Sep 15 12:18:57.581086 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:57.581064    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-294781-bbcfb7766-2h6m9"
Sep 15 12:18:57.587874 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:57.587558    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5l2hc\" (UniqueName: \"kubernetes.io/projected/ffc20dee-6524-4a5e-a8ff-1142b97b66f7-kube-api-access-5l2hc\") pod \"payload-pre-processing-e2e-explicit-294781-bbcfb7766-2h6m9\" (UID: \"ffc20dee-6524-4a5e-a8ff-1142b97b66f7\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-294781-bbcfb7766-2h6m9"
Sep 15 12:18:57.587874 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:57.587673    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffc20dee-6524-4a5e-a8ff-1142b97b66f7-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-294781-bbcfb7766-2h6m9\" (UID: \"ffc20dee-6524-4a5e-a8ff-1142b97b66f7\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-294781-bbcfb7766-2h6m9"
Sep 15 12:18:57.588240 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:57.588222    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-294781-dockercfg-tczj8\""
Sep 15 12:18:57.588361 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:57.588343    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-294781\""
Sep 15 12:18:57.599799 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:57.599763    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-294781-bbcfb7766-2h6m9"]
Sep 15 12:18:57.648673 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:57.648635    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-294781-6bd9d6cbb9-zl7w5"]
Sep 15 12:18:57.649093 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:57.649076    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc1353ca-8864-4cc7-93af-1a53f57fc7ab" containerName="istio-proxy"
Sep 15 12:18:57.649195 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:57.649096    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc1353ca-8864-4cc7-93af-1a53f57fc7ab" containerName="istio-proxy"
Sep 15 12:18:57.652348 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:57.652304    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-294781-6bd9d6cbb9-zl7w5"
Sep 15 12:18:57.668660 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:57.668632    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-294781-6bd9d6cbb9-zl7w5"]
Sep 15 12:18:57.688537 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:57.688505    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlr2t\" (UniqueName: \"kubernetes.io/projected/513d30bc-2e6e-472b-a4dc-b15722b1bf5e-kube-api-access-hlr2t\") pod \"payload-processing-e2e-explicit-294781-6bd9d6cbb9-zl7w5\" (UID: \"513d30bc-2e6e-472b-a4dc-b15722b1bf5e\") " pod="openshift-ingress/payload-processing-e2e-explicit-294781-6bd9d6cbb9-zl7w5"
Sep 15 12:18:57.688719 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:57.688579    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffc20dee-6524-4a5e-a8ff-1142b97b66f7-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-294781-bbcfb7766-2h6m9\" (UID: \"ffc20dee-6524-4a5e-a8ff-1142b97b66f7\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-294781-bbcfb7766-2h6m9"
Sep 15 12:18:57.688719 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:57.688643    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5l2hc\" (UniqueName: \"kubernetes.io/projected/ffc20dee-6524-4a5e-a8ff-1142b97b66f7-kube-api-access-5l2hc\") pod \"payload-pre-processing-e2e-explicit-294781-bbcfb7766-2h6m9\" (UID: \"ffc20dee-6524-4a5e-a8ff-1142b97b66f7\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-294781-bbcfb7766-2h6m9"
Sep 15 12:18:57.688719 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:57.688671    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/513d30bc-2e6e-472b-a4dc-b15722b1bf5e-plugins-config-volume\") pod \"payload-processing-e2e-explicit-294781-6bd9d6cbb9-zl7w5\" (UID: \"513d30bc-2e6e-472b-a4dc-b15722b1bf5e\") " pod="openshift-ingress/payload-processing-e2e-explicit-294781-6bd9d6cbb9-zl7w5"
Sep 15 12:18:57.689341 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:57.689321    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffc20dee-6524-4a5e-a8ff-1142b97b66f7-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-294781-bbcfb7766-2h6m9\" (UID: \"ffc20dee-6524-4a5e-a8ff-1142b97b66f7\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-294781-bbcfb7766-2h6m9"
Sep 15 12:18:57.701093 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:57.701065    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5l2hc\" (UniqueName: \"kubernetes.io/projected/ffc20dee-6524-4a5e-a8ff-1142b97b66f7-kube-api-access-5l2hc\") pod \"payload-pre-processing-e2e-explicit-294781-bbcfb7766-2h6m9\" (UID: \"ffc20dee-6524-4a5e-a8ff-1142b97b66f7\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-294781-bbcfb7766-2h6m9"
Sep 15 12:18:57.789663 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:57.789620    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/513d30bc-2e6e-472b-a4dc-b15722b1bf5e-plugins-config-volume\") pod \"payload-processing-e2e-explicit-294781-6bd9d6cbb9-zl7w5\" (UID: \"513d30bc-2e6e-472b-a4dc-b15722b1bf5e\") " pod="openshift-ingress/payload-processing-e2e-explicit-294781-6bd9d6cbb9-zl7w5"
Sep 15 12:18:57.789663 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:57.789668    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlr2t\" (UniqueName: \"kubernetes.io/projected/513d30bc-2e6e-472b-a4dc-b15722b1bf5e-kube-api-access-hlr2t\") pod \"payload-processing-e2e-explicit-294781-6bd9d6cbb9-zl7w5\" (UID: \"513d30bc-2e6e-472b-a4dc-b15722b1bf5e\") " pod="openshift-ingress/payload-processing-e2e-explicit-294781-6bd9d6cbb9-zl7w5"
Sep 15 12:18:57.790364 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:57.790344    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/513d30bc-2e6e-472b-a4dc-b15722b1bf5e-plugins-config-volume\") pod \"payload-processing-e2e-explicit-294781-6bd9d6cbb9-zl7w5\" (UID: \"513d30bc-2e6e-472b-a4dc-b15722b1bf5e\") " pod="openshift-ingress/payload-processing-e2e-explicit-294781-6bd9d6cbb9-zl7w5"
Sep 15 12:18:57.801989 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:57.801951    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlr2t\" (UniqueName: \"kubernetes.io/projected/513d30bc-2e6e-472b-a4dc-b15722b1bf5e-kube-api-access-hlr2t\") pod \"payload-processing-e2e-explicit-294781-6bd9d6cbb9-zl7w5\" (UID: \"513d30bc-2e6e-472b-a4dc-b15722b1bf5e\") " pod="openshift-ingress/payload-processing-e2e-explicit-294781-6bd9d6cbb9-zl7w5"
Sep 15 12:18:57.891998 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:57.891967    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-294781-bbcfb7766-2h6m9"
Sep 15 12:18:57.962656 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:57.962624    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-294781-6bd9d6cbb9-zl7w5"
Sep 15 12:18:58.171660 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:58.171629    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-294781-bbcfb7766-2h6m9"]
Sep 15 12:18:58.173687 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:18:58.173658    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffc20dee_6524_4a5e_a8ff_1142b97b66f7.slice/crio-d5642b2fe2fd99b7b541607656d621ad0d24a1296268d0d1c30da925e7ab0708 WatchSource:0}: Error finding container d5642b2fe2fd99b7b541607656d621ad0d24a1296268d0d1c30da925e7ab0708: Status 404 returned error can't find the container with id d5642b2fe2fd99b7b541607656d621ad0d24a1296268d0d1c30da925e7ab0708
Sep 15 12:18:58.248591 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:58.248473    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-294781-6bd9d6cbb9-zl7w5"]
Sep 15 12:18:58.897471 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:58.897436    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-294781-bbcfb7766-2h6m9" event={"ID":"ffc20dee-6524-4a5e-a8ff-1142b97b66f7","Type":"ContainerStarted","Data":"4427d76ff34963611e65f116c9bd1c3177f9e5b041be354c886cd2af66c0e329"}
Sep 15 12:18:58.897948 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:58.897491    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-294781-bbcfb7766-2h6m9" event={"ID":"ffc20dee-6524-4a5e-a8ff-1142b97b66f7","Type":"ContainerStarted","Data":"d5642b2fe2fd99b7b541607656d621ad0d24a1296268d0d1c30da925e7ab0708"}
Sep 15 12:18:58.897948 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:58.897522    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-294781-bbcfb7766-2h6m9"
Sep 15 12:18:58.899235 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:58.899207    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-294781-6bd9d6cbb9-zl7w5" event={"ID":"513d30bc-2e6e-472b-a4dc-b15722b1bf5e","Type":"ContainerStarted","Data":"2f25954715a4495d569e88ae6fd32d2566981b9bcce0e78b0044b21f569ef47c"}
Sep 15 12:18:58.899387 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:58.899256    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-294781-6bd9d6cbb9-zl7w5" event={"ID":"513d30bc-2e6e-472b-a4dc-b15722b1bf5e","Type":"ContainerStarted","Data":"94d1d1d41e2f377a99c712d1af41dff824227ea6f2d5c6b6b9280c65eeeb70a6"}
Sep 15 12:18:58.899387 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:58.899313    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-294781-6bd9d6cbb9-zl7w5"
Sep 15 12:18:58.946048 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:58.945993    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-294781-6bd9d6cbb9-zl7w5" podStartSLOduration=1.945974998 podStartE2EDuration="1.945974998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:18:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:18:58.944506568 +0000 UTC m=+995.735429104" watchObservedRunningTime="2026-09-15 12:18:58.945974998 +0000 UTC m=+995.736897509"
Sep 15 12:18:58.946586 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:18:58.946561    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-294781-bbcfb7766-2h6m9" podStartSLOduration=1.9465496340000001 podStartE2EDuration="1.946549634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:18:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:18:58.920850872 +0000 UTC m=+995.711773393" watchObservedRunningTime="2026-09-15 12:18:58.946549634 +0000 UTC m=+995.737472190"
Sep 15 12:19:04.588018 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:19:04.587973    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5gkxs"]
Sep 15 12:19:09.907476 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:19:09.907444    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-294781-6bd9d6cbb9-zl7w5"
Sep 15 12:19:09.907930 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:19:09.907534    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-294781-bbcfb7766-2h6m9"
Sep 15 12:19:29.538638 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:19:29.538592    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-69d7ba/test-model-69d7ba-kserve-8dc8bcc6c-gft8f"]
Sep 15 12:19:29.544867 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:19:29.544842    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-69d7ba/test-model-69d7ba-kserve-8dc8bcc6c-gft8f"
Sep 15 12:19:29.548446 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:19:29.548422    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-69d7ba\"/\"openshift-service-ca.crt\""
Sep 15 12:19:29.548590 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:19:29.548447    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-69d7ba\"/\"kube-root-ca.crt\""
Sep 15 12:19:29.549495 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:19:29.549481    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-69d7ba\"/\"default-dockercfg-xn476\""
Sep 15 12:19:29.549553 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:19:29.549522    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-69d7ba\"/\"test-model-69d7ba-kserve-self-signed-certs\""
Sep 15 12:19:29.558621 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:19:29.558597    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-69d7ba/test-model-69d7ba-kserve-8dc8bcc6c-gft8f"]
Sep 15 12:19:29.698828 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:19:29.698783    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8jwb\" (UniqueName: \"kubernetes.io/projected/84978e46-13ee-4363-a728-9510efe618bb-kube-api-access-l8jwb\") pod \"test-model-69d7ba-kserve-8dc8bcc6c-gft8f\" (UID: \"84978e46-13ee-4363-a728-9510efe618bb\") " pod="ai-tenant-e2e-inf-b-69d7ba/test-model-69d7ba-kserve-8dc8bcc6c-gft8f"
Sep 15 12:19:29.698828 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:19:29.698832    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84978e46-13ee-4363-a728-9510efe618bb-tmp-dir\") pod \"test-model-69d7ba-kserve-8dc8bcc6c-gft8f\" (UID: \"84978e46-13ee-4363-a728-9510efe618bb\") " pod="ai-tenant-e2e-inf-b-69d7ba/test-model-69d7ba-kserve-8dc8bcc6c-gft8f"
Sep 15 12:19:29.699044 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:19:29.698909    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84978e46-13ee-4363-a728-9510efe618bb-model-cache\") pod \"test-model-69d7ba-kserve-8dc8bcc6c-gft8f\" (UID: \"84978e46-13ee-4363-a728-9510efe618bb\") " pod="ai-tenant-e2e-inf-b-69d7ba/test-model-69d7ba-kserve-8dc8bcc6c-gft8f"
Sep 15 12:19:29.699044 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:19:29.698931    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84978e46-13ee-4363-a728-9510efe618bb-tls-certs\") pod \"test-model-69d7ba-kserve-8dc8bcc6c-gft8f\" (UID: \"84978e46-13ee-4363-a728-9510efe618bb\") " pod="ai-tenant-e2e-inf-b-69d7ba/test-model-69d7ba-kserve-8dc8bcc6c-gft8f"
Sep 15 12:19:29.699044 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:19:29.698994    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84978e46-13ee-4363-a728-9510efe618bb-dshm\") pod \"test-model-69d7ba-kserve-8dc8bcc6c-gft8f\" (UID: \"84978e46-13ee-4363-a728-9510efe618bb\") " pod="ai-tenant-e2e-inf-b-69d7ba/test-model-69d7ba-kserve-8dc8bcc6c-gft8f"
Sep 15 12:19:29.699044 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:19:29.699024    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84978e46-13ee-4363-a728-9510efe618bb-home\") pod \"test-model-69d7ba-kserve-8dc8bcc6c-gft8f\" (UID: \"84978e46-13ee-4363-a728-9510efe618bb\") " pod="ai-tenant-e2e-inf-b-69d7ba/test-model-69d7ba-kserve-8dc8bcc6c-gft8f"
Sep 15 12:19:29.800528 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:19:29.800436    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84978e46-13ee-4363-a728-9510efe618bb-model-cache\") pod \"test-model-69d7ba-kserve-8dc8bcc6c-gft8f\" (UID: \"84978e46-13ee-4363-a728-9510efe618bb\") " pod="ai-tenant-e2e-inf-b-69d7ba/test-model-69d7ba-kserve-8dc8bcc6c-gft8f"
Sep 15 12:19:29.800528 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:19:29.800484    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84978e46-13ee-4363-a728-9510efe618bb-tls-certs\") pod \"test-model-69d7ba-kserve-8dc8bcc6c-gft8f\" (UID: \"84978e46-13ee-4363-a728-9510efe618bb\") " pod="ai-tenant-e2e-inf-b-69d7ba/test-model-69d7ba-kserve-8dc8bcc6c-gft8f"
Sep 15 12:19:29.800528 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:19:29.800524    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84978e46-13ee-4363-a728-9510efe618bb-dshm\") pod \"test-model-69d7ba-kserve-8dc8bcc6c-gft8f\" (UID: \"84978e46-13ee-4363-a728-9510efe618bb\") " pod="ai-tenant-e2e-inf-b-69d7ba/test-model-69d7ba-kserve-8dc8bcc6c-gft8f"
Sep 15 12:19:29.800828 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:19:29.800541    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84978e46-13ee-4363-a728-9510efe618bb-home\") pod \"test-model-69d7ba-kserve-8dc8bcc6c-gft8f\" (UID: \"84978e46-13ee-4363-a728-9510efe618bb\") " pod="ai-tenant-e2e-inf-b-69d7ba/test-model-69d7ba-kserve-8dc8bcc6c-gft8f"
Sep 15 12:19:29.800828 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:19:29.800600    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8jwb\" (UniqueName: \"kubernetes.io/projected/84978e46-13ee-4363-a728-9510efe618bb-kube-api-access-l8jwb\") pod \"test-model-69d7ba-kserve-8dc8bcc6c-gft8f\" (UID: \"84978e46-13ee-4363-a728-9510efe618bb\") " pod="ai-tenant-e2e-inf-b-69d7ba/test-model-69d7ba-kserve-8dc8bcc6c-gft8f"
Sep 15 12:19:29.800828 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:19:29.800628    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84978e46-13ee-4363-a728-9510efe618bb-tmp-dir\") pod \"test-model-69d7ba-kserve-8dc8bcc6c-gft8f\" (UID: \"84978e46-13ee-4363-a728-9510efe618bb\") " pod="ai-tenant-e2e-inf-b-69d7ba/test-model-69d7ba-kserve-8dc8bcc6c-gft8f"
Sep 15 12:19:29.800974 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:19:29.800863    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84978e46-13ee-4363-a728-9510efe618bb-model-cache\") pod \"test-model-69d7ba-kserve-8dc8bcc6c-gft8f\" (UID: \"84978e46-13ee-4363-a728-9510efe618bb\") " pod="ai-tenant-e2e-inf-b-69d7ba/test-model-69d7ba-kserve-8dc8bcc6c-gft8f"
Sep 15 12:19:29.800974 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:19:29.800943    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84978e46-13ee-4363-a728-9510efe618bb-home\") pod \"test-model-69d7ba-kserve-8dc8bcc6c-gft8f\" (UID: \"84978e46-13ee-4363-a728-9510efe618bb\") " pod="ai-tenant-e2e-inf-b-69d7ba/test-model-69d7ba-kserve-8dc8bcc6c-gft8f"
Sep 15 12:19:29.801056 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:19:29.800973    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84978e46-13ee-4363-a728-9510efe618bb-tmp-dir\") pod \"test-model-69d7ba-kserve-8dc8bcc6c-gft8f\" (UID: \"84978e46-13ee-4363-a728-9510efe618bb\") " pod="ai-tenant-e2e-inf-b-69d7ba/test-model-69d7ba-kserve-8dc8bcc6c-gft8f"
Sep 15 12:19:29.802961 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:19:29.802938    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84978e46-13ee-4363-a728-9510efe618bb-dshm\") pod \"test-model-69d7ba-kserve-8dc8bcc6c-gft8f\" (UID: \"84978e46-13ee-4363-a728-9510efe618bb\") " pod="ai-tenant-e2e-inf-b-69d7ba/test-model-69d7ba-kserve-8dc8bcc6c-gft8f"
Sep 15 12:19:29.803512 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:19:29.803494    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84978e46-13ee-4363-a728-9510efe618bb-tls-certs\") pod \"test-model-69d7ba-kserve-8dc8bcc6c-gft8f\" (UID: \"84978e46-13ee-4363-a728-9510efe618bb\") " pod="ai-tenant-e2e-inf-b-69d7ba/test-model-69d7ba-kserve-8dc8bcc6c-gft8f"
Sep 15 12:19:29.812135 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:19:29.812106    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8jwb\" (UniqueName: \"kubernetes.io/projected/84978e46-13ee-4363-a728-9510efe618bb-kube-api-access-l8jwb\") pod \"test-model-69d7ba-kserve-8dc8bcc6c-gft8f\" (UID: \"84978e46-13ee-4363-a728-9510efe618bb\") " pod="ai-tenant-e2e-inf-b-69d7ba/test-model-69d7ba-kserve-8dc8bcc6c-gft8f"
Sep 15 12:19:29.854443 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:19:29.854411    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-69d7ba/test-model-69d7ba-kserve-8dc8bcc6c-gft8f"
Sep 15 12:19:30.139364 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:19:30.139340    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-69d7ba/test-model-69d7ba-kserve-8dc8bcc6c-gft8f"]
Sep 15 12:19:30.141213 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:19:30.141181    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod84978e46_13ee_4363_a728_9510efe618bb.slice/crio-1ec88fdbea5fd462dec34e5a8a3f3aa95a5682f5d118cc4a10a79507c0c1e853 WatchSource:0}: Error finding container 1ec88fdbea5fd462dec34e5a8a3f3aa95a5682f5d118cc4a10a79507c0c1e853: Status 404 returned error can't find the container with id 1ec88fdbea5fd462dec34e5a8a3f3aa95a5682f5d118cc4a10a79507c0c1e853
Sep 15 12:19:31.096839 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:19:31.096796    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-69d7ba/test-model-69d7ba-kserve-8dc8bcc6c-gft8f" event={"ID":"84978e46-13ee-4363-a728-9510efe618bb","Type":"ContainerStarted","Data":"a9579d1eb1f7d5f47264c9e0e1715985afb364f47071a0fb6b65968d5305a1e5"}
Sep 15 12:19:31.097256 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:19:31.096848    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-69d7ba/test-model-69d7ba-kserve-8dc8bcc6c-gft8f" event={"ID":"84978e46-13ee-4363-a728-9510efe618bb","Type":"ContainerStarted","Data":"1ec88fdbea5fd462dec34e5a8a3f3aa95a5682f5d118cc4a10a79507c0c1e853"}
Sep 15 12:19:31.119603 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:19:31.119539    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-69d7ba/test-model-69d7ba-kserve-8dc8bcc6c-gft8f" podStartSLOduration=2.119517048 podStartE2EDuration="2.119517048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:19:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:19:31.117458256 +0000 UTC m=+1027.908380770" watchObservedRunningTime="2026-09-15 12:19:31.119517048 +0000 UTC m=+1027.910439562"
Sep 15 12:19:32.091388 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:19:32.091355    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5gkxs"]
Sep 15 12:19:39.855506 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:19:39.855453    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-69d7ba/test-model-69d7ba-kserve-8dc8bcc6c-gft8f"
Sep 15 12:19:39.855921 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:19:39.855554    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-69d7ba/test-model-69d7ba-kserve-8dc8bcc6c-gft8f"
Sep 15 12:19:39.868537 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:19:39.868504    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-69d7ba/test-model-69d7ba-kserve-8dc8bcc6c-gft8f"
Sep 15 12:19:40.157067 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:19:40.157032    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-69d7ba/test-model-69d7ba-kserve-8dc8bcc6c-gft8f"
Sep 15 12:20:02.993016 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:02.992982    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-294781/explicit-model-294781-kserve-7575b9cb55-4xssj"]
Sep 15 12:20:02.996748 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:02.996727    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-294781/explicit-model-294781-kserve-7575b9cb55-4xssj"
Sep 15 12:20:02.999992 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:02.999970    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-294781\"/\"default-dockercfg-r2q69\""
Sep 15 12:20:03.001249 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:03.001230    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-294781\"/\"openshift-service-ca.crt\""
Sep 15 12:20:03.001469 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:03.001452    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-294781\"/\"explicit-model-294781-kserve-self-signed-certs\""
Sep 15 12:20:03.004504 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:03.004489    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-294781\"/\"kube-root-ca.crt\""
Sep 15 12:20:03.023356 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:03.023324    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-294781/explicit-model-294781-kserve-7575b9cb55-4xssj"]
Sep 15 12:20:03.122817 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:03.122764    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e48a7af-305e-452b-973a-af750c3c102b-home\") pod \"explicit-model-294781-kserve-7575b9cb55-4xssj\" (UID: \"1e48a7af-305e-452b-973a-af750c3c102b\") " pod="ai-tenant-e2e-explicit-294781/explicit-model-294781-kserve-7575b9cb55-4xssj"
Sep 15 12:20:03.123028 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:03.122851    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5c5x\" (UniqueName: \"kubernetes.io/projected/1e48a7af-305e-452b-973a-af750c3c102b-kube-api-access-m5c5x\") pod \"explicit-model-294781-kserve-7575b9cb55-4xssj\" (UID: \"1e48a7af-305e-452b-973a-af750c3c102b\") " pod="ai-tenant-e2e-explicit-294781/explicit-model-294781-kserve-7575b9cb55-4xssj"
Sep 15 12:20:03.123028 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:03.122880    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e48a7af-305e-452b-973a-af750c3c102b-model-cache\") pod \"explicit-model-294781-kserve-7575b9cb55-4xssj\" (UID: \"1e48a7af-305e-452b-973a-af750c3c102b\") " pod="ai-tenant-e2e-explicit-294781/explicit-model-294781-kserve-7575b9cb55-4xssj"
Sep 15 12:20:03.123028 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:03.122901    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e48a7af-305e-452b-973a-af750c3c102b-tmp-dir\") pod \"explicit-model-294781-kserve-7575b9cb55-4xssj\" (UID: \"1e48a7af-305e-452b-973a-af750c3c102b\") " pod="ai-tenant-e2e-explicit-294781/explicit-model-294781-kserve-7575b9cb55-4xssj"
Sep 15 12:20:03.123028 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:03.122952    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e48a7af-305e-452b-973a-af750c3c102b-tls-certs\") pod \"explicit-model-294781-kserve-7575b9cb55-4xssj\" (UID: \"1e48a7af-305e-452b-973a-af750c3c102b\") " pod="ai-tenant-e2e-explicit-294781/explicit-model-294781-kserve-7575b9cb55-4xssj"
Sep 15 12:20:03.123028 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:03.122990    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e48a7af-305e-452b-973a-af750c3c102b-dshm\") pod \"explicit-model-294781-kserve-7575b9cb55-4xssj\" (UID: \"1e48a7af-305e-452b-973a-af750c3c102b\") " pod="ai-tenant-e2e-explicit-294781/explicit-model-294781-kserve-7575b9cb55-4xssj"
Sep 15 12:20:03.224389 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:03.224352    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e48a7af-305e-452b-973a-af750c3c102b-dshm\") pod \"explicit-model-294781-kserve-7575b9cb55-4xssj\" (UID: \"1e48a7af-305e-452b-973a-af750c3c102b\") " pod="ai-tenant-e2e-explicit-294781/explicit-model-294781-kserve-7575b9cb55-4xssj"
Sep 15 12:20:03.224573 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:03.224437    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e48a7af-305e-452b-973a-af750c3c102b-home\") pod \"explicit-model-294781-kserve-7575b9cb55-4xssj\" (UID: \"1e48a7af-305e-452b-973a-af750c3c102b\") " pod="ai-tenant-e2e-explicit-294781/explicit-model-294781-kserve-7575b9cb55-4xssj"
Sep 15 12:20:03.224573 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:03.224473    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5c5x\" (UniqueName: \"kubernetes.io/projected/1e48a7af-305e-452b-973a-af750c3c102b-kube-api-access-m5c5x\") pod \"explicit-model-294781-kserve-7575b9cb55-4xssj\" (UID: \"1e48a7af-305e-452b-973a-af750c3c102b\") " pod="ai-tenant-e2e-explicit-294781/explicit-model-294781-kserve-7575b9cb55-4xssj"
Sep 15 12:20:03.224573 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:03.224492    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e48a7af-305e-452b-973a-af750c3c102b-model-cache\") pod \"explicit-model-294781-kserve-7575b9cb55-4xssj\" (UID: \"1e48a7af-305e-452b-973a-af750c3c102b\") " pod="ai-tenant-e2e-explicit-294781/explicit-model-294781-kserve-7575b9cb55-4xssj"
Sep 15 12:20:03.224573 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:03.224507    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e48a7af-305e-452b-973a-af750c3c102b-tmp-dir\") pod \"explicit-model-294781-kserve-7575b9cb55-4xssj\" (UID: \"1e48a7af-305e-452b-973a-af750c3c102b\") " pod="ai-tenant-e2e-explicit-294781/explicit-model-294781-kserve-7575b9cb55-4xssj"
Sep 15 12:20:03.224573 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:03.224529    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e48a7af-305e-452b-973a-af750c3c102b-tls-certs\") pod \"explicit-model-294781-kserve-7575b9cb55-4xssj\" (UID: \"1e48a7af-305e-452b-973a-af750c3c102b\") " pod="ai-tenant-e2e-explicit-294781/explicit-model-294781-kserve-7575b9cb55-4xssj"
Sep 15 12:20:03.224877 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:03.224854    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e48a7af-305e-452b-973a-af750c3c102b-home\") pod \"explicit-model-294781-kserve-7575b9cb55-4xssj\" (UID: \"1e48a7af-305e-452b-973a-af750c3c102b\") " pod="ai-tenant-e2e-explicit-294781/explicit-model-294781-kserve-7575b9cb55-4xssj"
Sep 15 12:20:03.224947 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:03.224904    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e48a7af-305e-452b-973a-af750c3c102b-model-cache\") pod \"explicit-model-294781-kserve-7575b9cb55-4xssj\" (UID: \"1e48a7af-305e-452b-973a-af750c3c102b\") " pod="ai-tenant-e2e-explicit-294781/explicit-model-294781-kserve-7575b9cb55-4xssj"
Sep 15 12:20:03.225004 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:03.224955    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e48a7af-305e-452b-973a-af750c3c102b-tmp-dir\") pod \"explicit-model-294781-kserve-7575b9cb55-4xssj\" (UID: \"1e48a7af-305e-452b-973a-af750c3c102b\") " pod="ai-tenant-e2e-explicit-294781/explicit-model-294781-kserve-7575b9cb55-4xssj"
Sep 15 12:20:03.226711 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:03.226690    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e48a7af-305e-452b-973a-af750c3c102b-dshm\") pod \"explicit-model-294781-kserve-7575b9cb55-4xssj\" (UID: \"1e48a7af-305e-452b-973a-af750c3c102b\") " pod="ai-tenant-e2e-explicit-294781/explicit-model-294781-kserve-7575b9cb55-4xssj"
Sep 15 12:20:03.226863 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:03.226847    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e48a7af-305e-452b-973a-af750c3c102b-tls-certs\") pod \"explicit-model-294781-kserve-7575b9cb55-4xssj\" (UID: \"1e48a7af-305e-452b-973a-af750c3c102b\") " pod="ai-tenant-e2e-explicit-294781/explicit-model-294781-kserve-7575b9cb55-4xssj"
Sep 15 12:20:03.233610 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:03.233585    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5c5x\" (UniqueName: \"kubernetes.io/projected/1e48a7af-305e-452b-973a-af750c3c102b-kube-api-access-m5c5x\") pod \"explicit-model-294781-kserve-7575b9cb55-4xssj\" (UID: \"1e48a7af-305e-452b-973a-af750c3c102b\") " pod="ai-tenant-e2e-explicit-294781/explicit-model-294781-kserve-7575b9cb55-4xssj"
Sep 15 12:20:03.307091 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:03.306991    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-294781/explicit-model-294781-kserve-7575b9cb55-4xssj"
Sep 15 12:20:03.581093 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:03.581058    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-294781/explicit-model-294781-kserve-7575b9cb55-4xssj"]
Sep 15 12:20:03.581693 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:20:03.581667    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1e48a7af_305e_452b_973a_af750c3c102b.slice/crio-f4ec566d0102159dec1bd1cc20848d4f74ece58a41fd92345d2ed81a352b3e5c WatchSource:0}: Error finding container f4ec566d0102159dec1bd1cc20848d4f74ece58a41fd92345d2ed81a352b3e5c: Status 404 returned error can't find the container with id f4ec566d0102159dec1bd1cc20848d4f74ece58a41fd92345d2ed81a352b3e5c
Sep 15 12:20:04.285779 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:04.285739    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-294781/explicit-model-294781-kserve-7575b9cb55-4xssj" event={"ID":"1e48a7af-305e-452b-973a-af750c3c102b","Type":"ContainerStarted","Data":"6d97c187e02de582dda5cd4e4aca0068a348cd9d5df5da4257a292c2b24ac763"}
Sep 15 12:20:04.285779 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:04.285785    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-294781/explicit-model-294781-kserve-7575b9cb55-4xssj" event={"ID":"1e48a7af-305e-452b-973a-af750c3c102b","Type":"ContainerStarted","Data":"f4ec566d0102159dec1bd1cc20848d4f74ece58a41fd92345d2ed81a352b3e5c"}
Sep 15 12:20:04.305885 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:04.305830    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-294781/explicit-model-294781-kserve-7575b9cb55-4xssj" podStartSLOduration=2.305812583 podStartE2EDuration="2.305812583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:20:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:20:04.303344665 +0000 UTC m=+1061.094267188" watchObservedRunningTime="2026-09-15 12:20:04.305812583 +0000 UTC m=+1061.096735110"
Sep 15 12:20:04.685694 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:04.685660    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5gkxs"]
Sep 15 12:20:13.307812 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:13.307774    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-294781/explicit-model-294781-kserve-7575b9cb55-4xssj"
Sep 15 12:20:13.308394 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:13.307860    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-294781/explicit-model-294781-kserve-7575b9cb55-4xssj"
Sep 15 12:20:13.320670 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:13.320643    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-294781/explicit-model-294781-kserve-7575b9cb55-4xssj"
Sep 15 12:20:13.348567 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:13.348542    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-294781/explicit-model-294781-kserve-7575b9cb55-4xssj"
Sep 15 12:20:13.943851 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:13.943811    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-680229/distinct-w0-680229-kserve-5fcc68849d-nmsc7"]
Sep 15 12:20:13.947718 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:13.947691    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-680229/distinct-w0-680229-kserve-5fcc68849d-nmsc7"
Sep 15 12:20:13.953507 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:13.953479    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-680229\"/\"distinct-w0-680229-kserve-self-signed-certs\""
Sep 15 12:20:13.962257 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:13.962232    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-680229/distinct-w0-680229-kserve-5fcc68849d-nmsc7"]
Sep 15 12:20:14.027107 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:14.027066    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c2b1801-220f-4b66-a232-09aa568820f9-tls-certs\") pod \"distinct-w0-680229-kserve-5fcc68849d-nmsc7\" (UID: \"3c2b1801-220f-4b66-a232-09aa568820f9\") " pod="e2e-models-e2e-worker-w0-680229/distinct-w0-680229-kserve-5fcc68849d-nmsc7"
Sep 15 12:20:14.027107 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:14.027108    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5bkb\" (UniqueName: \"kubernetes.io/projected/3c2b1801-220f-4b66-a232-09aa568820f9-kube-api-access-c5bkb\") pod \"distinct-w0-680229-kserve-5fcc68849d-nmsc7\" (UID: \"3c2b1801-220f-4b66-a232-09aa568820f9\") " pod="e2e-models-e2e-worker-w0-680229/distinct-w0-680229-kserve-5fcc68849d-nmsc7"
Sep 15 12:20:14.027367 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:14.027187    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c2b1801-220f-4b66-a232-09aa568820f9-tmp-dir\") pod \"distinct-w0-680229-kserve-5fcc68849d-nmsc7\" (UID: \"3c2b1801-220f-4b66-a232-09aa568820f9\") " pod="e2e-models-e2e-worker-w0-680229/distinct-w0-680229-kserve-5fcc68849d-nmsc7"
Sep 15 12:20:14.027367 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:14.027286    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c2b1801-220f-4b66-a232-09aa568820f9-dshm\") pod \"distinct-w0-680229-kserve-5fcc68849d-nmsc7\" (UID: \"3c2b1801-220f-4b66-a232-09aa568820f9\") " pod="e2e-models-e2e-worker-w0-680229/distinct-w0-680229-kserve-5fcc68849d-nmsc7"
Sep 15 12:20:14.027367 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:14.027323    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c2b1801-220f-4b66-a232-09aa568820f9-model-cache\") pod \"distinct-w0-680229-kserve-5fcc68849d-nmsc7\" (UID: \"3c2b1801-220f-4b66-a232-09aa568820f9\") " pod="e2e-models-e2e-worker-w0-680229/distinct-w0-680229-kserve-5fcc68849d-nmsc7"
Sep 15 12:20:14.027472 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:14.027392    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c2b1801-220f-4b66-a232-09aa568820f9-home\") pod \"distinct-w0-680229-kserve-5fcc68849d-nmsc7\" (UID: \"3c2b1801-220f-4b66-a232-09aa568820f9\") " pod="e2e-models-e2e-worker-w0-680229/distinct-w0-680229-kserve-5fcc68849d-nmsc7"
Sep 15 12:20:14.128096 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:14.128062    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c2b1801-220f-4b66-a232-09aa568820f9-dshm\") pod \"distinct-w0-680229-kserve-5fcc68849d-nmsc7\" (UID: \"3c2b1801-220f-4b66-a232-09aa568820f9\") " pod="e2e-models-e2e-worker-w0-680229/distinct-w0-680229-kserve-5fcc68849d-nmsc7"
Sep 15 12:20:14.128301 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:14.128112    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c2b1801-220f-4b66-a232-09aa568820f9-model-cache\") pod \"distinct-w0-680229-kserve-5fcc68849d-nmsc7\" (UID: \"3c2b1801-220f-4b66-a232-09aa568820f9\") " pod="e2e-models-e2e-worker-w0-680229/distinct-w0-680229-kserve-5fcc68849d-nmsc7"
Sep 15 12:20:14.128301 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:14.128184    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c2b1801-220f-4b66-a232-09aa568820f9-home\") pod \"distinct-w0-680229-kserve-5fcc68849d-nmsc7\" (UID: \"3c2b1801-220f-4b66-a232-09aa568820f9\") " pod="e2e-models-e2e-worker-w0-680229/distinct-w0-680229-kserve-5fcc68849d-nmsc7"
Sep 15 12:20:14.128301 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:14.128253    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c2b1801-220f-4b66-a232-09aa568820f9-tls-certs\") pod \"distinct-w0-680229-kserve-5fcc68849d-nmsc7\" (UID: \"3c2b1801-220f-4b66-a232-09aa568820f9\") " pod="e2e-models-e2e-worker-w0-680229/distinct-w0-680229-kserve-5fcc68849d-nmsc7"
Sep 15 12:20:14.128481 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:14.128305    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5bkb\" (UniqueName: \"kubernetes.io/projected/3c2b1801-220f-4b66-a232-09aa568820f9-kube-api-access-c5bkb\") pod \"distinct-w0-680229-kserve-5fcc68849d-nmsc7\" (UID: \"3c2b1801-220f-4b66-a232-09aa568820f9\") " pod="e2e-models-e2e-worker-w0-680229/distinct-w0-680229-kserve-5fcc68849d-nmsc7"
Sep 15 12:20:14.128481 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:14.128354    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c2b1801-220f-4b66-a232-09aa568820f9-tmp-dir\") pod \"distinct-w0-680229-kserve-5fcc68849d-nmsc7\" (UID: \"3c2b1801-220f-4b66-a232-09aa568820f9\") " pod="e2e-models-e2e-worker-w0-680229/distinct-w0-680229-kserve-5fcc68849d-nmsc7"
Sep 15 12:20:14.128627 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:14.128600    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c2b1801-220f-4b66-a232-09aa568820f9-model-cache\") pod \"distinct-w0-680229-kserve-5fcc68849d-nmsc7\" (UID: \"3c2b1801-220f-4b66-a232-09aa568820f9\") " pod="e2e-models-e2e-worker-w0-680229/distinct-w0-680229-kserve-5fcc68849d-nmsc7"
Sep 15 12:20:14.128755 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:14.128665    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c2b1801-220f-4b66-a232-09aa568820f9-tmp-dir\") pod \"distinct-w0-680229-kserve-5fcc68849d-nmsc7\" (UID: \"3c2b1801-220f-4b66-a232-09aa568820f9\") " pod="e2e-models-e2e-worker-w0-680229/distinct-w0-680229-kserve-5fcc68849d-nmsc7"
Sep 15 12:20:14.128887 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:14.128867    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c2b1801-220f-4b66-a232-09aa568820f9-home\") pod \"distinct-w0-680229-kserve-5fcc68849d-nmsc7\" (UID: \"3c2b1801-220f-4b66-a232-09aa568820f9\") " pod="e2e-models-e2e-worker-w0-680229/distinct-w0-680229-kserve-5fcc68849d-nmsc7"
Sep 15 12:20:14.130713 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:14.130693    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c2b1801-220f-4b66-a232-09aa568820f9-dshm\") pod \"distinct-w0-680229-kserve-5fcc68849d-nmsc7\" (UID: \"3c2b1801-220f-4b66-a232-09aa568820f9\") " pod="e2e-models-e2e-worker-w0-680229/distinct-w0-680229-kserve-5fcc68849d-nmsc7"
Sep 15 12:20:14.131004 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:14.130980    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c2b1801-220f-4b66-a232-09aa568820f9-tls-certs\") pod \"distinct-w0-680229-kserve-5fcc68849d-nmsc7\" (UID: \"3c2b1801-220f-4b66-a232-09aa568820f9\") " pod="e2e-models-e2e-worker-w0-680229/distinct-w0-680229-kserve-5fcc68849d-nmsc7"
Sep 15 12:20:14.138651 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:14.138615    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5bkb\" (UniqueName: \"kubernetes.io/projected/3c2b1801-220f-4b66-a232-09aa568820f9-kube-api-access-c5bkb\") pod \"distinct-w0-680229-kserve-5fcc68849d-nmsc7\" (UID: \"3c2b1801-220f-4b66-a232-09aa568820f9\") " pod="e2e-models-e2e-worker-w0-680229/distinct-w0-680229-kserve-5fcc68849d-nmsc7"
Sep 15 12:20:14.257856 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:14.257765    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-680229/distinct-w0-680229-kserve-5fcc68849d-nmsc7"
Sep 15 12:20:14.537085 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:14.537053    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-680229/distinct-w0-680229-kserve-5fcc68849d-nmsc7"]
Sep 15 12:20:14.538708 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:20:14.538682    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3c2b1801_220f_4b66_a232_09aa568820f9.slice/crio-dfb06f4cce080c5504355cfe36544d0fcd71807847746d924ac8222c1eb3a5c6 WatchSource:0}: Error finding container dfb06f4cce080c5504355cfe36544d0fcd71807847746d924ac8222c1eb3a5c6: Status 404 returned error can't find the container with id dfb06f4cce080c5504355cfe36544d0fcd71807847746d924ac8222c1eb3a5c6
Sep 15 12:20:15.351790 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:15.351750    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-680229/distinct-w0-680229-kserve-5fcc68849d-nmsc7" event={"ID":"3c2b1801-220f-4b66-a232-09aa568820f9","Type":"ContainerStarted","Data":"3a9b08fc7ecfe6a4ed3b26fc7a8b779c83b0768b2a68b70ab57c1d86033e596e"}
Sep 15 12:20:15.351984 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:15.351804    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-680229/distinct-w0-680229-kserve-5fcc68849d-nmsc7" event={"ID":"3c2b1801-220f-4b66-a232-09aa568820f9","Type":"ContainerStarted","Data":"dfb06f4cce080c5504355cfe36544d0fcd71807847746d924ac8222c1eb3a5c6"}
Sep 15 12:20:15.376013 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:15.375959    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-680229/distinct-w0-680229-kserve-5fcc68849d-nmsc7" podStartSLOduration=2.375941675 podStartE2EDuration="2.375941675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:20:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:20:15.374007684 +0000 UTC m=+1072.164930198" watchObservedRunningTime="2026-09-15 12:20:15.375941675 +0000 UTC m=+1072.166864259"
Sep 15 12:20:16.705135 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:16.705058    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-294781/explicit-model-294781-kserve-7575b9cb55-4xssj"]
Sep 15 12:20:24.258004 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:24.257960    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-680229/distinct-w0-680229-kserve-5fcc68849d-nmsc7"
Sep 15 12:20:24.258467 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:24.258022    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-680229/distinct-w0-680229-kserve-5fcc68849d-nmsc7"
Sep 15 12:20:24.271015 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:24.270985    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-680229/distinct-w0-680229-kserve-5fcc68849d-nmsc7"
Sep 15 12:20:24.416548 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:24.416506    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-680229/distinct-w0-680229-kserve-5fcc68849d-nmsc7"
Sep 15 12:20:27.727589 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:27.727552    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-294781-6bd9d6cbb9-zl7w5"]
Sep 15 12:20:27.728114 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:27.727799    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-294781-6bd9d6cbb9-zl7w5" podUID="513d30bc-2e6e-472b-a4dc-b15722b1bf5e" containerName="payload-processing" containerID="cri-o://2f25954715a4495d569e88ae6fd32d2566981b9bcce0e78b0044b21f569ef47c" gracePeriod=30
Sep 15 12:20:27.740445 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:27.740415    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-294781-bbcfb7766-2h6m9"]
Sep 15 12:20:27.740708 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:27.740662    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-294781-bbcfb7766-2h6m9" podUID="ffc20dee-6524-4a5e-a8ff-1142b97b66f7" containerName="payload-pre-processing" containerID="cri-o://4427d76ff34963611e65f116c9bd1c3177f9e5b041be354c886cd2af66c0e329" gracePeriod=30
Sep 15 12:20:28.025380 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:28.025357    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-294781-6bd9d6cbb9-zl7w5"
Sep 15 12:20:28.028396 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:28.028377    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-294781-bbcfb7766-2h6m9"
Sep 15 12:20:28.070516 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:28.070484    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5l2hc\" (UniqueName: \"kubernetes.io/projected/ffc20dee-6524-4a5e-a8ff-1142b97b66f7-kube-api-access-5l2hc\") pod \"ffc20dee-6524-4a5e-a8ff-1142b97b66f7\" (UID: \"ffc20dee-6524-4a5e-a8ff-1142b97b66f7\") "
Sep 15 12:20:28.070699 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:28.070525    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffc20dee-6524-4a5e-a8ff-1142b97b66f7-plugins-config-volume\") pod \"ffc20dee-6524-4a5e-a8ff-1142b97b66f7\" (UID: \"ffc20dee-6524-4a5e-a8ff-1142b97b66f7\") "
Sep 15 12:20:28.070699 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:28.070553    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/513d30bc-2e6e-472b-a4dc-b15722b1bf5e-plugins-config-volume\") pod \"513d30bc-2e6e-472b-a4dc-b15722b1bf5e\" (UID: \"513d30bc-2e6e-472b-a4dc-b15722b1bf5e\") "
Sep 15 12:20:28.070699 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:28.070670    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hlr2t\" (UniqueName: \"kubernetes.io/projected/513d30bc-2e6e-472b-a4dc-b15722b1bf5e-kube-api-access-hlr2t\") pod \"513d30bc-2e6e-472b-a4dc-b15722b1bf5e\" (UID: \"513d30bc-2e6e-472b-a4dc-b15722b1bf5e\") "
Sep 15 12:20:28.070962 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:28.070933    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffc20dee-6524-4a5e-a8ff-1142b97b66f7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ffc20dee-6524-4a5e-a8ff-1142b97b66f7" (UID: "ffc20dee-6524-4a5e-a8ff-1142b97b66f7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:20:28.070962 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:28.070953    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/513d30bc-2e6e-472b-a4dc-b15722b1bf5e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "513d30bc-2e6e-472b-a4dc-b15722b1bf5e" (UID: "513d30bc-2e6e-472b-a4dc-b15722b1bf5e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:20:28.071092 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:28.071049    2555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffc20dee-6524-4a5e-a8ff-1142b97b66f7-plugins-config-volume\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:20:28.071092 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:28.071065    2555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/513d30bc-2e6e-472b-a4dc-b15722b1bf5e-plugins-config-volume\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:20:28.072765 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:28.072734    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffc20dee-6524-4a5e-a8ff-1142b97b66f7-kube-api-access-5l2hc" (OuterVolumeSpecName: "kube-api-access-5l2hc") pod "ffc20dee-6524-4a5e-a8ff-1142b97b66f7" (UID: "ffc20dee-6524-4a5e-a8ff-1142b97b66f7"). InnerVolumeSpecName "kube-api-access-5l2hc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:20:28.073063 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:28.073047    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/513d30bc-2e6e-472b-a4dc-b15722b1bf5e-kube-api-access-hlr2t" (OuterVolumeSpecName: "kube-api-access-hlr2t") pod "513d30bc-2e6e-472b-a4dc-b15722b1bf5e" (UID: "513d30bc-2e6e-472b-a4dc-b15722b1bf5e"). InnerVolumeSpecName "kube-api-access-hlr2t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:20:28.172342 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:28.172300    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlr2t\" (UniqueName: \"kubernetes.io/projected/513d30bc-2e6e-472b-a4dc-b15722b1bf5e-kube-api-access-hlr2t\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:20:28.172342 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:28.172339    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5l2hc\" (UniqueName: \"kubernetes.io/projected/ffc20dee-6524-4a5e-a8ff-1142b97b66f7-kube-api-access-5l2hc\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:20:28.436054 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:28.436024    2555 generic.go:356] "Generic (PLEG): container finished" podID="ffc20dee-6524-4a5e-a8ff-1142b97b66f7" containerID="4427d76ff34963611e65f116c9bd1c3177f9e5b041be354c886cd2af66c0e329" exitCode=0
Sep 15 12:20:28.436314 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:28.436071    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-294781-bbcfb7766-2h6m9"
Sep 15 12:20:28.436314 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:28.436111    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-294781-bbcfb7766-2h6m9" event={"ID":"ffc20dee-6524-4a5e-a8ff-1142b97b66f7","Type":"ContainerDied","Data":"4427d76ff34963611e65f116c9bd1c3177f9e5b041be354c886cd2af66c0e329"}
Sep 15 12:20:28.436314 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:28.436160    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-294781-bbcfb7766-2h6m9" event={"ID":"ffc20dee-6524-4a5e-a8ff-1142b97b66f7","Type":"ContainerDied","Data":"d5642b2fe2fd99b7b541607656d621ad0d24a1296268d0d1c30da925e7ab0708"}
Sep 15 12:20:28.436314 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:28.436182    2555 scope.go:117] "RemoveContainer" containerID="4427d76ff34963611e65f116c9bd1c3177f9e5b041be354c886cd2af66c0e329"
Sep 15 12:20:28.437751 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:28.437728    2555 generic.go:356] "Generic (PLEG): container finished" podID="513d30bc-2e6e-472b-a4dc-b15722b1bf5e" containerID="2f25954715a4495d569e88ae6fd32d2566981b9bcce0e78b0044b21f569ef47c" exitCode=0
Sep 15 12:20:28.437869 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:28.437764    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-294781-6bd9d6cbb9-zl7w5"
Sep 15 12:20:28.437869 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:28.437821    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-294781-6bd9d6cbb9-zl7w5" event={"ID":"513d30bc-2e6e-472b-a4dc-b15722b1bf5e","Type":"ContainerDied","Data":"2f25954715a4495d569e88ae6fd32d2566981b9bcce0e78b0044b21f569ef47c"}
Sep 15 12:20:28.437869 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:28.437867    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-294781-6bd9d6cbb9-zl7w5" event={"ID":"513d30bc-2e6e-472b-a4dc-b15722b1bf5e","Type":"ContainerDied","Data":"94d1d1d41e2f377a99c712d1af41dff824227ea6f2d5c6b6b9280c65eeeb70a6"}
Sep 15 12:20:28.450577 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:28.450428    2555 scope.go:117] "RemoveContainer" containerID="4427d76ff34963611e65f116c9bd1c3177f9e5b041be354c886cd2af66c0e329"
Sep 15 12:20:28.451319 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:20:28.451291    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4427d76ff34963611e65f116c9bd1c3177f9e5b041be354c886cd2af66c0e329\": container with ID starting with 4427d76ff34963611e65f116c9bd1c3177f9e5b041be354c886cd2af66c0e329 not found: ID does not exist" containerID="4427d76ff34963611e65f116c9bd1c3177f9e5b041be354c886cd2af66c0e329"
Sep 15 12:20:28.451411 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:28.451330    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4427d76ff34963611e65f116c9bd1c3177f9e5b041be354c886cd2af66c0e329"} err="failed to get container status \"4427d76ff34963611e65f116c9bd1c3177f9e5b041be354c886cd2af66c0e329\": rpc error: code = NotFound desc = could not find container \"4427d76ff34963611e65f116c9bd1c3177f9e5b041be354c886cd2af66c0e329\": container with ID starting with 4427d76ff34963611e65f116c9bd1c3177f9e5b041be354c886cd2af66c0e329 not found: ID does not exist"
Sep 15 12:20:28.451411 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:28.451354    2555 scope.go:117] "RemoveContainer" containerID="2f25954715a4495d569e88ae6fd32d2566981b9bcce0e78b0044b21f569ef47c"
Sep 15 12:20:28.460474 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:28.460457    2555 scope.go:117] "RemoveContainer" containerID="2f25954715a4495d569e88ae6fd32d2566981b9bcce0e78b0044b21f569ef47c"
Sep 15 12:20:28.460754 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:20:28.460736    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f25954715a4495d569e88ae6fd32d2566981b9bcce0e78b0044b21f569ef47c\": container with ID starting with 2f25954715a4495d569e88ae6fd32d2566981b9bcce0e78b0044b21f569ef47c not found: ID does not exist" containerID="2f25954715a4495d569e88ae6fd32d2566981b9bcce0e78b0044b21f569ef47c"
Sep 15 12:20:28.460824 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:28.460768    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f25954715a4495d569e88ae6fd32d2566981b9bcce0e78b0044b21f569ef47c"} err="failed to get container status \"2f25954715a4495d569e88ae6fd32d2566981b9bcce0e78b0044b21f569ef47c\": rpc error: code = NotFound desc = could not find container \"2f25954715a4495d569e88ae6fd32d2566981b9bcce0e78b0044b21f569ef47c\": container with ID starting with 2f25954715a4495d569e88ae6fd32d2566981b9bcce0e78b0044b21f569ef47c not found: ID does not exist"
Sep 15 12:20:28.464275 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:28.464246    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-294781-bbcfb7766-2h6m9"]
Sep 15 12:20:28.467888 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:28.467865    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-294781-bbcfb7766-2h6m9"]
Sep 15 12:20:28.484395 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:28.484365    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-294781-6bd9d6cbb9-zl7w5"]
Sep 15 12:20:28.488558 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:28.488536    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-294781-6bd9d6cbb9-zl7w5"]
Sep 15 12:20:29.817023 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:29.816981    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="513d30bc-2e6e-472b-a4dc-b15722b1bf5e" path="/var/lib/kubelet/pods/513d30bc-2e6e-472b-a4dc-b15722b1bf5e/volumes"
Sep 15 12:20:29.817576 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:29.817555    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ffc20dee-6524-4a5e-a8ff-1142b97b66f7" path="/var/lib/kubelet/pods/ffc20dee-6524-4a5e-a8ff-1142b97b66f7/volumes"
Sep 15 12:20:31.724922 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:31.724856    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-294781/explicit-model-294781-kserve-7575b9cb55-4xssj" podUID="1e48a7af-305e-452b-973a-af750c3c102b" containerName="main" containerID="cri-o://6d97c187e02de582dda5cd4e4aca0068a348cd9d5df5da4257a292c2b24ac763" gracePeriod=45
Sep 15 12:20:31.980832 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:31.980772    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-294781/explicit-model-294781-kserve-7575b9cb55-4xssj"
Sep 15 12:20:32.111105 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:32.111073    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e48a7af-305e-452b-973a-af750c3c102b-tls-certs\") pod \"1e48a7af-305e-452b-973a-af750c3c102b\" (UID: \"1e48a7af-305e-452b-973a-af750c3c102b\") "
Sep 15 12:20:32.111290 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:32.111122    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e48a7af-305e-452b-973a-af750c3c102b-tmp-dir\") pod \"1e48a7af-305e-452b-973a-af750c3c102b\" (UID: \"1e48a7af-305e-452b-973a-af750c3c102b\") "
Sep 15 12:20:32.111290 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:32.111151    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e48a7af-305e-452b-973a-af750c3c102b-model-cache\") pod \"1e48a7af-305e-452b-973a-af750c3c102b\" (UID: \"1e48a7af-305e-452b-973a-af750c3c102b\") "
Sep 15 12:20:32.111290 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:32.111244    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m5c5x\" (UniqueName: \"kubernetes.io/projected/1e48a7af-305e-452b-973a-af750c3c102b-kube-api-access-m5c5x\") pod \"1e48a7af-305e-452b-973a-af750c3c102b\" (UID: \"1e48a7af-305e-452b-973a-af750c3c102b\") "
Sep 15 12:20:32.111451 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:32.111294    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e48a7af-305e-452b-973a-af750c3c102b-home\") pod \"1e48a7af-305e-452b-973a-af750c3c102b\" (UID: \"1e48a7af-305e-452b-973a-af750c3c102b\") "
Sep 15 12:20:32.111451 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:32.111317    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e48a7af-305e-452b-973a-af750c3c102b-dshm\") pod \"1e48a7af-305e-452b-973a-af750c3c102b\" (UID: \"1e48a7af-305e-452b-973a-af750c3c102b\") "
Sep 15 12:20:32.111451 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:32.111430    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e48a7af-305e-452b-973a-af750c3c102b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1e48a7af-305e-452b-973a-af750c3c102b" (UID: "1e48a7af-305e-452b-973a-af750c3c102b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:20:32.111607 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:32.111460    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e48a7af-305e-452b-973a-af750c3c102b-model-cache" (OuterVolumeSpecName: "model-cache") pod "1e48a7af-305e-452b-973a-af750c3c102b" (UID: "1e48a7af-305e-452b-973a-af750c3c102b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:20:32.111607 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:32.111563    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e48a7af-305e-452b-973a-af750c3c102b-home" (OuterVolumeSpecName: "home") pod "1e48a7af-305e-452b-973a-af750c3c102b" (UID: "1e48a7af-305e-452b-973a-af750c3c102b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:20:32.111718 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:32.111684    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e48a7af-305e-452b-973a-af750c3c102b-model-cache\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:20:32.111718 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:32.111696    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e48a7af-305e-452b-973a-af750c3c102b-home\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:20:32.111718 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:32.111705    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e48a7af-305e-452b-973a-af750c3c102b-tmp-dir\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:20:32.113413 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:32.113380    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e48a7af-305e-452b-973a-af750c3c102b-dshm" (OuterVolumeSpecName: "dshm") pod "1e48a7af-305e-452b-973a-af750c3c102b" (UID: "1e48a7af-305e-452b-973a-af750c3c102b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:20:32.113539 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:32.113478    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e48a7af-305e-452b-973a-af750c3c102b-kube-api-access-m5c5x" (OuterVolumeSpecName: "kube-api-access-m5c5x") pod "1e48a7af-305e-452b-973a-af750c3c102b" (UID: "1e48a7af-305e-452b-973a-af750c3c102b"). InnerVolumeSpecName "kube-api-access-m5c5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:20:32.113539 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:32.113507    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e48a7af-305e-452b-973a-af750c3c102b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1e48a7af-305e-452b-973a-af750c3c102b" (UID: "1e48a7af-305e-452b-973a-af750c3c102b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:20:32.212204 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:32.212168    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5c5x\" (UniqueName: \"kubernetes.io/projected/1e48a7af-305e-452b-973a-af750c3c102b-kube-api-access-m5c5x\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:20:32.212204 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:32.212196    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e48a7af-305e-452b-973a-af750c3c102b-dshm\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:20:32.212204 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:32.212207    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e48a7af-305e-452b-973a-af750c3c102b-tls-certs\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:20:32.464773 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:32.464741    2555 generic.go:356] "Generic (PLEG): container finished" podID="1e48a7af-305e-452b-973a-af750c3c102b" containerID="6d97c187e02de582dda5cd4e4aca0068a348cd9d5df5da4257a292c2b24ac763" exitCode=0
Sep 15 12:20:32.464983 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:32.464793    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-294781/explicit-model-294781-kserve-7575b9cb55-4xssj"
Sep 15 12:20:32.464983 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:32.464826    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-294781/explicit-model-294781-kserve-7575b9cb55-4xssj" event={"ID":"1e48a7af-305e-452b-973a-af750c3c102b","Type":"ContainerDied","Data":"6d97c187e02de582dda5cd4e4aca0068a348cd9d5df5da4257a292c2b24ac763"}
Sep 15 12:20:32.464983 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:32.464868    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-294781/explicit-model-294781-kserve-7575b9cb55-4xssj" event={"ID":"1e48a7af-305e-452b-973a-af750c3c102b","Type":"ContainerDied","Data":"f4ec566d0102159dec1bd1cc20848d4f74ece58a41fd92345d2ed81a352b3e5c"}
Sep 15 12:20:32.464983 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:32.464884    2555 scope.go:117] "RemoveContainer" containerID="6d97c187e02de582dda5cd4e4aca0068a348cd9d5df5da4257a292c2b24ac763"
Sep 15 12:20:32.474909 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:32.474890    2555 scope.go:117] "RemoveContainer" containerID="6d97c187e02de582dda5cd4e4aca0068a348cd9d5df5da4257a292c2b24ac763"
Sep 15 12:20:32.475211 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:20:32.475193    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d97c187e02de582dda5cd4e4aca0068a348cd9d5df5da4257a292c2b24ac763\": container with ID starting with 6d97c187e02de582dda5cd4e4aca0068a348cd9d5df5da4257a292c2b24ac763 not found: ID does not exist" containerID="6d97c187e02de582dda5cd4e4aca0068a348cd9d5df5da4257a292c2b24ac763"
Sep 15 12:20:32.475291 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:32.475220    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d97c187e02de582dda5cd4e4aca0068a348cd9d5df5da4257a292c2b24ac763"} err="failed to get container status \"6d97c187e02de582dda5cd4e4aca0068a348cd9d5df5da4257a292c2b24ac763\": rpc error: code = NotFound desc = could not find container \"6d97c187e02de582dda5cd4e4aca0068a348cd9d5df5da4257a292c2b24ac763\": container with ID starting with 6d97c187e02de582dda5cd4e4aca0068a348cd9d5df5da4257a292c2b24ac763 not found: ID does not exist"
Sep 15 12:20:32.490204 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:32.490172    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-294781/explicit-model-294781-kserve-7575b9cb55-4xssj"]
Sep 15 12:20:32.496828 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:32.496799    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-294781/explicit-model-294781-kserve-7575b9cb55-4xssj"]
Sep 15 12:20:33.818270 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:33.818227    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e48a7af-305e-452b-973a-af750c3c102b" path="/var/lib/kubelet/pods/1e48a7af-305e-452b-973a-af750c3c102b/volumes"
Sep 15 12:20:36.361096 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:36.361060    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-166e5c-openshift-default-5c447444b8-k65r8"]
Sep 15 12:20:36.361533 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:36.361437    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="513d30bc-2e6e-472b-a4dc-b15722b1bf5e" containerName="payload-processing"
Sep 15 12:20:36.361533 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:36.361449    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="513d30bc-2e6e-472b-a4dc-b15722b1bf5e" containerName="payload-processing"
Sep 15 12:20:36.361533 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:36.361469    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e48a7af-305e-452b-973a-af750c3c102b" containerName="main"
Sep 15 12:20:36.361533 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:36.361475    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e48a7af-305e-452b-973a-af750c3c102b" containerName="main"
Sep 15 12:20:36.361533 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:36.361487    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffc20dee-6524-4a5e-a8ff-1142b97b66f7" containerName="payload-pre-processing"
Sep 15 12:20:36.361533 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:36.361492    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffc20dee-6524-4a5e-a8ff-1142b97b66f7" containerName="payload-pre-processing"
Sep 15 12:20:36.361760 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:36.361552    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e48a7af-305e-452b-973a-af750c3c102b" containerName="main"
Sep 15 12:20:36.361760 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:36.361558    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="513d30bc-2e6e-472b-a4dc-b15722b1bf5e" containerName="payload-processing"
Sep 15 12:20:36.361760 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:36.361567    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffc20dee-6524-4a5e-a8ff-1142b97b66f7" containerName="payload-pre-processing"
Sep 15 12:20:36.365792 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:36.365772    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-166e5c-openshift-default-5c447444b8-k65r8"
Sep 15 12:20:36.369078 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:36.369053    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 15 12:20:36.369235 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:36.369053    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-166e5c-openshift-default-dockercfg-vsrnc\""
Sep 15 12:20:36.381757 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:36.381725    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-166e5c-openshift-default-5c447444b8-k65r8"]
Sep 15 12:20:36.459456 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:36.459417    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8c62c8a-d028-4034-a3f0-9c7a368bbc4e-workload-socket\") pod \"e2e-noten-166e5c-openshift-default-5c447444b8-k65r8\" (UID: \"d8c62c8a-d028-4034-a3f0-9c7a368bbc4e\") " pod="openshift-ingress/e2e-noten-166e5c-openshift-default-5c447444b8-k65r8"
Sep 15 12:20:36.459456 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:36.459458    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8c62c8a-d028-4034-a3f0-9c7a368bbc4e-workload-certs\") pod \"e2e-noten-166e5c-openshift-default-5c447444b8-k65r8\" (UID: \"d8c62c8a-d028-4034-a3f0-9c7a368bbc4e\") " pod="openshift-ingress/e2e-noten-166e5c-openshift-default-5c447444b8-k65r8"
Sep 15 12:20:36.459674 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:36.459485    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8c62c8a-d028-4034-a3f0-9c7a368bbc4e-istio-token\") pod \"e2e-noten-166e5c-openshift-default-5c447444b8-k65r8\" (UID: \"d8c62c8a-d028-4034-a3f0-9c7a368bbc4e\") " pod="openshift-ingress/e2e-noten-166e5c-openshift-default-5c447444b8-k65r8"
Sep 15 12:20:36.459674 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:36.459567    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8c62c8a-d028-4034-a3f0-9c7a368bbc4e-istio-podinfo\") pod \"e2e-noten-166e5c-openshift-default-5c447444b8-k65r8\" (UID: \"d8c62c8a-d028-4034-a3f0-9c7a368bbc4e\") " pod="openshift-ingress/e2e-noten-166e5c-openshift-default-5c447444b8-k65r8"
Sep 15 12:20:36.459674 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:36.459618    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8c62c8a-d028-4034-a3f0-9c7a368bbc4e-credential-socket\") pod \"e2e-noten-166e5c-openshift-default-5c447444b8-k65r8\" (UID: \"d8c62c8a-d028-4034-a3f0-9c7a368bbc4e\") " pod="openshift-ingress/e2e-noten-166e5c-openshift-default-5c447444b8-k65r8"
Sep 15 12:20:36.459674 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:36.459643    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8c62c8a-d028-4034-a3f0-9c7a368bbc4e-istiod-ca-cert\") pod \"e2e-noten-166e5c-openshift-default-5c447444b8-k65r8\" (UID: \"d8c62c8a-d028-4034-a3f0-9c7a368bbc4e\") " pod="openshift-ingress/e2e-noten-166e5c-openshift-default-5c447444b8-k65r8"
Sep 15 12:20:36.459864 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:36.459689    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8c62c8a-d028-4034-a3f0-9c7a368bbc4e-istio-data\") pod \"e2e-noten-166e5c-openshift-default-5c447444b8-k65r8\" (UID: \"d8c62c8a-d028-4034-a3f0-9c7a368bbc4e\") " pod="openshift-ingress/e2e-noten-166e5c-openshift-default-5c447444b8-k65r8"
Sep 15 12:20:36.459864 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:36.459795    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8c62c8a-d028-4034-a3f0-9c7a368bbc4e-istio-envoy\") pod \"e2e-noten-166e5c-openshift-default-5c447444b8-k65r8\" (UID: \"d8c62c8a-d028-4034-a3f0-9c7a368bbc4e\") " pod="openshift-ingress/e2e-noten-166e5c-openshift-default-5c447444b8-k65r8"
Sep 15 12:20:36.459864 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:36.459838    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65bm8\" (UniqueName: \"kubernetes.io/projected/d8c62c8a-d028-4034-a3f0-9c7a368bbc4e-kube-api-access-65bm8\") pod \"e2e-noten-166e5c-openshift-default-5c447444b8-k65r8\" (UID: \"d8c62c8a-d028-4034-a3f0-9c7a368bbc4e\") " pod="openshift-ingress/e2e-noten-166e5c-openshift-default-5c447444b8-k65r8"
Sep 15 12:20:36.560434 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:36.560390    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8c62c8a-d028-4034-a3f0-9c7a368bbc4e-istio-token\") pod \"e2e-noten-166e5c-openshift-default-5c447444b8-k65r8\" (UID: \"d8c62c8a-d028-4034-a3f0-9c7a368bbc4e\") " pod="openshift-ingress/e2e-noten-166e5c-openshift-default-5c447444b8-k65r8"
Sep 15 12:20:36.560434 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:36.560437    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8c62c8a-d028-4034-a3f0-9c7a368bbc4e-istio-podinfo\") pod \"e2e-noten-166e5c-openshift-default-5c447444b8-k65r8\" (UID: \"d8c62c8a-d028-4034-a3f0-9c7a368bbc4e\") " pod="openshift-ingress/e2e-noten-166e5c-openshift-default-5c447444b8-k65r8"
Sep 15 12:20:36.560730 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:36.560471    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8c62c8a-d028-4034-a3f0-9c7a368bbc4e-credential-socket\") pod \"e2e-noten-166e5c-openshift-default-5c447444b8-k65r8\" (UID: \"d8c62c8a-d028-4034-a3f0-9c7a368bbc4e\") " pod="openshift-ingress/e2e-noten-166e5c-openshift-default-5c447444b8-k65r8"
Sep 15 12:20:36.560730 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:36.560495    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8c62c8a-d028-4034-a3f0-9c7a368bbc4e-istiod-ca-cert\") pod \"e2e-noten-166e5c-openshift-default-5c447444b8-k65r8\" (UID: \"d8c62c8a-d028-4034-a3f0-9c7a368bbc4e\") " pod="openshift-ingress/e2e-noten-166e5c-openshift-default-5c447444b8-k65r8"
Sep 15 12:20:36.560730 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:36.560521    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8c62c8a-d028-4034-a3f0-9c7a368bbc4e-istio-data\") pod \"e2e-noten-166e5c-openshift-default-5c447444b8-k65r8\" (UID: \"d8c62c8a-d028-4034-a3f0-9c7a368bbc4e\") " pod="openshift-ingress/e2e-noten-166e5c-openshift-default-5c447444b8-k65r8"
Sep 15 12:20:36.560730 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:36.560601    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8c62c8a-d028-4034-a3f0-9c7a368bbc4e-istio-envoy\") pod \"e2e-noten-166e5c-openshift-default-5c447444b8-k65r8\" (UID: \"d8c62c8a-d028-4034-a3f0-9c7a368bbc4e\") " pod="openshift-ingress/e2e-noten-166e5c-openshift-default-5c447444b8-k65r8"
Sep 15 12:20:36.560730 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:36.560637    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65bm8\" (UniqueName: \"kubernetes.io/projected/d8c62c8a-d028-4034-a3f0-9c7a368bbc4e-kube-api-access-65bm8\") pod \"e2e-noten-166e5c-openshift-default-5c447444b8-k65r8\" (UID: \"d8c62c8a-d028-4034-a3f0-9c7a368bbc4e\") " pod="openshift-ingress/e2e-noten-166e5c-openshift-default-5c447444b8-k65r8"
Sep 15 12:20:36.560730 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:36.560668    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8c62c8a-d028-4034-a3f0-9c7a368bbc4e-workload-socket\") pod \"e2e-noten-166e5c-openshift-default-5c447444b8-k65r8\" (UID: \"d8c62c8a-d028-4034-a3f0-9c7a368bbc4e\") " pod="openshift-ingress/e2e-noten-166e5c-openshift-default-5c447444b8-k65r8"
Sep 15 12:20:36.561041 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:36.560759    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8c62c8a-d028-4034-a3f0-9c7a368bbc4e-workload-certs\") pod \"e2e-noten-166e5c-openshift-default-5c447444b8-k65r8\" (UID: \"d8c62c8a-d028-4034-a3f0-9c7a368bbc4e\") " pod="openshift-ingress/e2e-noten-166e5c-openshift-default-5c447444b8-k65r8"
Sep 15 12:20:36.561041 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:36.560954    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8c62c8a-d028-4034-a3f0-9c7a368bbc4e-credential-socket\") pod \"e2e-noten-166e5c-openshift-default-5c447444b8-k65r8\" (UID: \"d8c62c8a-d028-4034-a3f0-9c7a368bbc4e\") " pod="openshift-ingress/e2e-noten-166e5c-openshift-default-5c447444b8-k65r8"
Sep 15 12:20:36.561041 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:36.561022    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8c62c8a-d028-4034-a3f0-9c7a368bbc4e-istio-data\") pod \"e2e-noten-166e5c-openshift-default-5c447444b8-k65r8\" (UID: \"d8c62c8a-d028-4034-a3f0-9c7a368bbc4e\") " pod="openshift-ingress/e2e-noten-166e5c-openshift-default-5c447444b8-k65r8"
Sep 15 12:20:36.561229 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:36.561054    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8c62c8a-d028-4034-a3f0-9c7a368bbc4e-workload-certs\") pod \"e2e-noten-166e5c-openshift-default-5c447444b8-k65r8\" (UID: \"d8c62c8a-d028-4034-a3f0-9c7a368bbc4e\") " pod="openshift-ingress/e2e-noten-166e5c-openshift-default-5c447444b8-k65r8"
Sep 15 12:20:36.561229 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:36.561095    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8c62c8a-d028-4034-a3f0-9c7a368bbc4e-workload-socket\") pod \"e2e-noten-166e5c-openshift-default-5c447444b8-k65r8\" (UID: \"d8c62c8a-d028-4034-a3f0-9c7a368bbc4e\") " pod="openshift-ingress/e2e-noten-166e5c-openshift-default-5c447444b8-k65r8"
Sep 15 12:20:36.561521 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:36.561503    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8c62c8a-d028-4034-a3f0-9c7a368bbc4e-istiod-ca-cert\") pod \"e2e-noten-166e5c-openshift-default-5c447444b8-k65r8\" (UID: \"d8c62c8a-d028-4034-a3f0-9c7a368bbc4e\") " pod="openshift-ingress/e2e-noten-166e5c-openshift-default-5c447444b8-k65r8"
Sep 15 12:20:36.562823 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:36.562802    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8c62c8a-d028-4034-a3f0-9c7a368bbc4e-istio-envoy\") pod \"e2e-noten-166e5c-openshift-default-5c447444b8-k65r8\" (UID: \"d8c62c8a-d028-4034-a3f0-9c7a368bbc4e\") " pod="openshift-ingress/e2e-noten-166e5c-openshift-default-5c447444b8-k65r8"
Sep 15 12:20:36.563099 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:36.563081    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8c62c8a-d028-4034-a3f0-9c7a368bbc4e-istio-podinfo\") pod \"e2e-noten-166e5c-openshift-default-5c447444b8-k65r8\" (UID: \"d8c62c8a-d028-4034-a3f0-9c7a368bbc4e\") " pod="openshift-ingress/e2e-noten-166e5c-openshift-default-5c447444b8-k65r8"
Sep 15 12:20:36.571389 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:36.571360    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8c62c8a-d028-4034-a3f0-9c7a368bbc4e-istio-token\") pod \"e2e-noten-166e5c-openshift-default-5c447444b8-k65r8\" (UID: \"d8c62c8a-d028-4034-a3f0-9c7a368bbc4e\") " pod="openshift-ingress/e2e-noten-166e5c-openshift-default-5c447444b8-k65r8"
Sep 15 12:20:36.571621 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:36.571525    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65bm8\" (UniqueName: \"kubernetes.io/projected/d8c62c8a-d028-4034-a3f0-9c7a368bbc4e-kube-api-access-65bm8\") pod \"e2e-noten-166e5c-openshift-default-5c447444b8-k65r8\" (UID: \"d8c62c8a-d028-4034-a3f0-9c7a368bbc4e\") " pod="openshift-ingress/e2e-noten-166e5c-openshift-default-5c447444b8-k65r8"
Sep 15 12:20:36.676475 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:36.676433    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-166e5c-openshift-default-5c447444b8-k65r8"
Sep 15 12:20:37.011460 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:37.011428    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-166e5c-openshift-default-5c447444b8-k65r8"]
Sep 15 12:20:37.012294 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:20:37.012232    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8c62c8a_d028_4034_a3f0_9c7a368bbc4e.slice/crio-7d78c4410cd1c7e4ce1b73b6f82444ef965f4aa56c09e901d6e368c1dca41b21 WatchSource:0}: Error finding container 7d78c4410cd1c7e4ce1b73b6f82444ef965f4aa56c09e901d6e368c1dca41b21: Status 404 returned error can't find the container with id 7d78c4410cd1c7e4ce1b73b6f82444ef965f4aa56c09e901d6e368c1dca41b21
Sep 15 12:20:37.014838 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:37.014797    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:20:37.014917 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:37.014893    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:20:37.014963 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:37.014942    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:20:37.495709 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:37.495674    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-166e5c-openshift-default-5c447444b8-k65r8" event={"ID":"d8c62c8a-d028-4034-a3f0-9c7a368bbc4e","Type":"ContainerStarted","Data":"14d5d072e0575e41a5af7aa46deda0f5b9ca0edbc022ccb39f68de95f3c70615"}
Sep 15 12:20:37.496161 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:37.495730    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-166e5c-openshift-default-5c447444b8-k65r8" event={"ID":"d8c62c8a-d028-4034-a3f0-9c7a368bbc4e","Type":"ContainerStarted","Data":"7d78c4410cd1c7e4ce1b73b6f82444ef965f4aa56c09e901d6e368c1dca41b21"}
Sep 15 12:20:37.533766 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:37.533709    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-166e5c-openshift-default-5c447444b8-k65r8" podStartSLOduration=1.5336915119999999 podStartE2EDuration="1.533691512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:20:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:20:37.53087192 +0000 UTC m=+1094.321794432" watchObservedRunningTime="2026-09-15 12:20:37.533691512 +0000 UTC m=+1094.324614022"
Sep 15 12:20:37.676564 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:37.676509    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-166e5c-openshift-default-5c447444b8-k65r8"
Sep 15 12:20:38.682578 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:38.682545    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-166e5c-openshift-default-5c447444b8-k65r8"
Sep 15 12:20:39.262154 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:39.262113    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5gkxs"]
Sep 15 12:20:39.507371 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:39.507329    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-166e5c-openshift-default-5c447444b8-k65r8"
Sep 15 12:20:39.508411 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:20:39.508384    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-166e5c-openshift-default-5c447444b8-k65r8"
Sep 15 12:21:10.387171 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:21:10.387129    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5gkxs"]
Sep 15 12:21:30.518389 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:21:30.518352    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-166e5c/noten-model-166e5c-kserve-5f746d946c-snkzb"]
Sep 15 12:21:30.523343 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:21:30.523319    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-166e5c/noten-model-166e5c-kserve-5f746d946c-snkzb"
Sep 15 12:21:30.527475 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:21:30.527448    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-166e5c\"/\"kube-root-ca.crt\""
Sep 15 12:21:30.527620 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:21:30.527448    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-166e5c\"/\"noten-model-166e5c-kserve-self-signed-certs\""
Sep 15 12:21:30.527620 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:21:30.527607    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-166e5c\"/\"openshift-service-ca.crt\""
Sep 15 12:21:30.527729 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:21:30.527641    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-166e5c\"/\"default-dockercfg-64hk7\""
Sep 15 12:21:30.536115 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:21:30.536090    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-166e5c/noten-model-166e5c-kserve-5f746d946c-snkzb"]
Sep 15 12:21:30.614520 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:21:30.614481    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b04a7844-49ed-48a4-8ef0-7aad3eee1aea-home\") pod \"noten-model-166e5c-kserve-5f746d946c-snkzb\" (UID: \"b04a7844-49ed-48a4-8ef0-7aad3eee1aea\") " pod="ai-tenant-e2e-noten-166e5c/noten-model-166e5c-kserve-5f746d946c-snkzb"
Sep 15 12:21:30.614520 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:21:30.614518    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b04a7844-49ed-48a4-8ef0-7aad3eee1aea-dshm\") pod \"noten-model-166e5c-kserve-5f746d946c-snkzb\" (UID: \"b04a7844-49ed-48a4-8ef0-7aad3eee1aea\") " pod="ai-tenant-e2e-noten-166e5c/noten-model-166e5c-kserve-5f746d946c-snkzb"
Sep 15 12:21:30.614727 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:21:30.614593    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b04a7844-49ed-48a4-8ef0-7aad3eee1aea-tmp-dir\") pod \"noten-model-166e5c-kserve-5f746d946c-snkzb\" (UID: \"b04a7844-49ed-48a4-8ef0-7aad3eee1aea\") " pod="ai-tenant-e2e-noten-166e5c/noten-model-166e5c-kserve-5f746d946c-snkzb"
Sep 15 12:21:30.614727 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:21:30.614633    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b04a7844-49ed-48a4-8ef0-7aad3eee1aea-model-cache\") pod \"noten-model-166e5c-kserve-5f746d946c-snkzb\" (UID: \"b04a7844-49ed-48a4-8ef0-7aad3eee1aea\") " pod="ai-tenant-e2e-noten-166e5c/noten-model-166e5c-kserve-5f746d946c-snkzb"
Sep 15 12:21:30.614727 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:21:30.614723    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b04a7844-49ed-48a4-8ef0-7aad3eee1aea-tls-certs\") pod \"noten-model-166e5c-kserve-5f746d946c-snkzb\" (UID: \"b04a7844-49ed-48a4-8ef0-7aad3eee1aea\") " pod="ai-tenant-e2e-noten-166e5c/noten-model-166e5c-kserve-5f746d946c-snkzb"
Sep 15 12:21:30.614832 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:21:30.614742    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mq7wp\" (UniqueName: \"kubernetes.io/projected/b04a7844-49ed-48a4-8ef0-7aad3eee1aea-kube-api-access-mq7wp\") pod \"noten-model-166e5c-kserve-5f746d946c-snkzb\" (UID: \"b04a7844-49ed-48a4-8ef0-7aad3eee1aea\") " pod="ai-tenant-e2e-noten-166e5c/noten-model-166e5c-kserve-5f746d946c-snkzb"
Sep 15 12:21:30.715969 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:21:30.715915    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b04a7844-49ed-48a4-8ef0-7aad3eee1aea-tmp-dir\") pod \"noten-model-166e5c-kserve-5f746d946c-snkzb\" (UID: \"b04a7844-49ed-48a4-8ef0-7aad3eee1aea\") " pod="ai-tenant-e2e-noten-166e5c/noten-model-166e5c-kserve-5f746d946c-snkzb"
Sep 15 12:21:30.715969 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:21:30.715969    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b04a7844-49ed-48a4-8ef0-7aad3eee1aea-model-cache\") pod \"noten-model-166e5c-kserve-5f746d946c-snkzb\" (UID: \"b04a7844-49ed-48a4-8ef0-7aad3eee1aea\") " pod="ai-tenant-e2e-noten-166e5c/noten-model-166e5c-kserve-5f746d946c-snkzb"
Sep 15 12:21:30.716239 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:21:30.716039    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b04a7844-49ed-48a4-8ef0-7aad3eee1aea-tls-certs\") pod \"noten-model-166e5c-kserve-5f746d946c-snkzb\" (UID: \"b04a7844-49ed-48a4-8ef0-7aad3eee1aea\") " pod="ai-tenant-e2e-noten-166e5c/noten-model-166e5c-kserve-5f746d946c-snkzb"
Sep 15 12:21:30.716239 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:21:30.716065    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq7wp\" (UniqueName: \"kubernetes.io/projected/b04a7844-49ed-48a4-8ef0-7aad3eee1aea-kube-api-access-mq7wp\") pod \"noten-model-166e5c-kserve-5f746d946c-snkzb\" (UID: \"b04a7844-49ed-48a4-8ef0-7aad3eee1aea\") " pod="ai-tenant-e2e-noten-166e5c/noten-model-166e5c-kserve-5f746d946c-snkzb"
Sep 15 12:21:30.716239 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:21:30.716151    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b04a7844-49ed-48a4-8ef0-7aad3eee1aea-home\") pod \"noten-model-166e5c-kserve-5f746d946c-snkzb\" (UID: \"b04a7844-49ed-48a4-8ef0-7aad3eee1aea\") " pod="ai-tenant-e2e-noten-166e5c/noten-model-166e5c-kserve-5f746d946c-snkzb"
Sep 15 12:21:30.716239 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:21:30.716174    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b04a7844-49ed-48a4-8ef0-7aad3eee1aea-dshm\") pod \"noten-model-166e5c-kserve-5f746d946c-snkzb\" (UID: \"b04a7844-49ed-48a4-8ef0-7aad3eee1aea\") " pod="ai-tenant-e2e-noten-166e5c/noten-model-166e5c-kserve-5f746d946c-snkzb"
Sep 15 12:21:30.716487 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:21:30.716373    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b04a7844-49ed-48a4-8ef0-7aad3eee1aea-tmp-dir\") pod \"noten-model-166e5c-kserve-5f746d946c-snkzb\" (UID: \"b04a7844-49ed-48a4-8ef0-7aad3eee1aea\") " pod="ai-tenant-e2e-noten-166e5c/noten-model-166e5c-kserve-5f746d946c-snkzb"
Sep 15 12:21:30.716487 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:21:30.716430    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b04a7844-49ed-48a4-8ef0-7aad3eee1aea-model-cache\") pod \"noten-model-166e5c-kserve-5f746d946c-snkzb\" (UID: \"b04a7844-49ed-48a4-8ef0-7aad3eee1aea\") " pod="ai-tenant-e2e-noten-166e5c/noten-model-166e5c-kserve-5f746d946c-snkzb"
Sep 15 12:21:30.716588 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:21:30.716490    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b04a7844-49ed-48a4-8ef0-7aad3eee1aea-home\") pod \"noten-model-166e5c-kserve-5f746d946c-snkzb\" (UID: \"b04a7844-49ed-48a4-8ef0-7aad3eee1aea\") " pod="ai-tenant-e2e-noten-166e5c/noten-model-166e5c-kserve-5f746d946c-snkzb"
Sep 15 12:21:30.718339 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:21:30.718322    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b04a7844-49ed-48a4-8ef0-7aad3eee1aea-dshm\") pod \"noten-model-166e5c-kserve-5f746d946c-snkzb\" (UID: \"b04a7844-49ed-48a4-8ef0-7aad3eee1aea\") " pod="ai-tenant-e2e-noten-166e5c/noten-model-166e5c-kserve-5f746d946c-snkzb"
Sep 15 12:21:30.718739 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:21:30.718715    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b04a7844-49ed-48a4-8ef0-7aad3eee1aea-tls-certs\") pod \"noten-model-166e5c-kserve-5f746d946c-snkzb\" (UID: \"b04a7844-49ed-48a4-8ef0-7aad3eee1aea\") " pod="ai-tenant-e2e-noten-166e5c/noten-model-166e5c-kserve-5f746d946c-snkzb"
Sep 15 12:21:30.728637 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:21:30.728613    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq7wp\" (UniqueName: \"kubernetes.io/projected/b04a7844-49ed-48a4-8ef0-7aad3eee1aea-kube-api-access-mq7wp\") pod \"noten-model-166e5c-kserve-5f746d946c-snkzb\" (UID: \"b04a7844-49ed-48a4-8ef0-7aad3eee1aea\") " pod="ai-tenant-e2e-noten-166e5c/noten-model-166e5c-kserve-5f746d946c-snkzb"
Sep 15 12:21:30.833380 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:21:30.833277    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-166e5c/noten-model-166e5c-kserve-5f746d946c-snkzb"
Sep 15 12:21:31.115945 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:21:31.115920    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-166e5c/noten-model-166e5c-kserve-5f746d946c-snkzb"]
Sep 15 12:21:31.117768 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:21:31.117741    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb04a7844_49ed_48a4_8ef0_7aad3eee1aea.slice/crio-b7e0255ee2838c52c48c62d2888bd0c2cc92caccdabf096ae839f7dcb49a11cf WatchSource:0}: Error finding container b7e0255ee2838c52c48c62d2888bd0c2cc92caccdabf096ae839f7dcb49a11cf: Status 404 returned error can't find the container with id b7e0255ee2838c52c48c62d2888bd0c2cc92caccdabf096ae839f7dcb49a11cf
Sep 15 12:21:31.830963 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:21:31.830926    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-166e5c/noten-model-166e5c-kserve-5f746d946c-snkzb" event={"ID":"b04a7844-49ed-48a4-8ef0-7aad3eee1aea","Type":"ContainerStarted","Data":"4a6c036d6ec67d4c19f5e3cffac473ec301776f76bc259cc30f119be039f33c8"}
Sep 15 12:21:31.830963 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:21:31.830965    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-166e5c/noten-model-166e5c-kserve-5f746d946c-snkzb" event={"ID":"b04a7844-49ed-48a4-8ef0-7aad3eee1aea","Type":"ContainerStarted","Data":"b7e0255ee2838c52c48c62d2888bd0c2cc92caccdabf096ae839f7dcb49a11cf"}
Sep 15 12:21:31.852196 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:21:31.852142    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-166e5c/noten-model-166e5c-kserve-5f746d946c-snkzb" podStartSLOduration=1.852123501 podStartE2EDuration="1.852123501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:21:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:21:31.851155025 +0000 UTC m=+1148.642077559" watchObservedRunningTime="2026-09-15 12:21:31.852123501 +0000 UTC m=+1148.643046012"
Sep 15 12:21:40.834453 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:21:40.834412    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-166e5c/noten-model-166e5c-kserve-5f746d946c-snkzb"
Sep 15 12:21:40.834904 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:21:40.834466    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-166e5c/noten-model-166e5c-kserve-5f746d946c-snkzb"
Sep 15 12:21:40.846958 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:21:40.846930    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-166e5c/noten-model-166e5c-kserve-5f746d946c-snkzb"
Sep 15 12:21:40.894154 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:21:40.894120    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-166e5c/noten-model-166e5c-kserve-5f746d946c-snkzb"
Sep 15 12:21:41.638016 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:21:41.637972    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5gkxs"]
Sep 15 12:21:44.670956 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:21:44.670918    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ba366c-5c8b5d7794-rkhhq"]
Sep 15 12:21:44.674642 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:21:44.674619    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ba366c-5c8b5d7794-rkhhq"
Sep 15 12:21:44.677446 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:21:44.677418    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-ba366c\""
Sep 15 12:21:44.677815 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:21:44.677799    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-ba366c-dockercfg-bbzzf\""
Sep 15 12:21:44.746984 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:21:44.746949    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ba366c-5c8b5d7794-rkhhq"]
Sep 15 12:21:44.763158 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:21:44.763124    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcj4d\" (UniqueName: \"kubernetes.io/projected/0770ba7c-72be-4fb5-bf05-bef6114ce762-kube-api-access-jcj4d\") pod \"payload-pre-processing-e2e-shared-b-w2-ba366c-5c8b5d7794-rkhhq\" (UID: \"0770ba7c-72be-4fb5-bf05-bef6114ce762\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ba366c-5c8b5d7794-rkhhq"
Sep 15 12:21:44.763342 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:21:44.763190    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0770ba7c-72be-4fb5-bf05-bef6114ce762-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-ba366c-5c8b5d7794-rkhhq\" (UID: \"0770ba7c-72be-4fb5-bf05-bef6114ce762\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ba366c-5c8b5d7794-rkhhq"
Sep 15 12:21:44.864144 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:21:44.864105    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0770ba7c-72be-4fb5-bf05-bef6114ce762-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-ba366c-5c8b5d7794-rkhhq\" (UID: \"0770ba7c-72be-4fb5-bf05-bef6114ce762\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ba366c-5c8b5d7794-rkhhq"
Sep 15 12:21:44.864360 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:21:44.864237    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcj4d\" (UniqueName: \"kubernetes.io/projected/0770ba7c-72be-4fb5-bf05-bef6114ce762-kube-api-access-jcj4d\") pod \"payload-pre-processing-e2e-shared-b-w2-ba366c-5c8b5d7794-rkhhq\" (UID: \"0770ba7c-72be-4fb5-bf05-bef6114ce762\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ba366c-5c8b5d7794-rkhhq"
Sep 15 12:21:44.864877 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:21:44.864855    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0770ba7c-72be-4fb5-bf05-bef6114ce762-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-ba366c-5c8b5d7794-rkhhq\" (UID: \"0770ba7c-72be-4fb5-bf05-bef6114ce762\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ba366c-5c8b5d7794-rkhhq"
Sep 15 12:21:44.874866 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:21:44.874844    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcj4d\" (UniqueName: \"kubernetes.io/projected/0770ba7c-72be-4fb5-bf05-bef6114ce762-kube-api-access-jcj4d\") pod \"payload-pre-processing-e2e-shared-b-w2-ba366c-5c8b5d7794-rkhhq\" (UID: \"0770ba7c-72be-4fb5-bf05-bef6114ce762\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ba366c-5c8b5d7794-rkhhq"
Sep 15 12:21:44.985313 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:21:44.985195    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ba366c-5c8b5d7794-rkhhq"
Sep 15 12:21:45.264481 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:21:45.264454    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ba366c-5c8b5d7794-rkhhq"]
Sep 15 12:21:45.267414 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:21:45.267378    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0770ba7c_72be_4fb5_bf05_bef6114ce762.slice/crio-e65ed4386f9977302e9eb9a6ec126e8757a4aebe596b66fc07c4e10e5b834c79 WatchSource:0}: Error finding container e65ed4386f9977302e9eb9a6ec126e8757a4aebe596b66fc07c4e10e5b834c79: Status 404 returned error can't find the container with id e65ed4386f9977302e9eb9a6ec126e8757a4aebe596b66fc07c4e10e5b834c79
Sep 15 12:21:45.911786 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:21:45.911697    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ba366c-5c8b5d7794-rkhhq" event={"ID":"0770ba7c-72be-4fb5-bf05-bef6114ce762","Type":"ContainerStarted","Data":"0bf8a778559a73a9ed1864b8237333aa4af6083d2816cd66417135d3900739bc"}
Sep 15 12:21:45.911786 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:21:45.911760    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ba366c-5c8b5d7794-rkhhq"
Sep 15 12:21:45.911786 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:21:45.911783    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ba366c-5c8b5d7794-rkhhq" event={"ID":"0770ba7c-72be-4fb5-bf05-bef6114ce762","Type":"ContainerStarted","Data":"e65ed4386f9977302e9eb9a6ec126e8757a4aebe596b66fc07c4e10e5b834c79"}
Sep 15 12:21:45.938767 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:21:45.935015    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ba366c-5c8b5d7794-rkhhq" podStartSLOduration=1.934994776 podStartE2EDuration="1.934994776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:21:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:21:45.929812145 +0000 UTC m=+1162.720734668" watchObservedRunningTime="2026-09-15 12:21:45.934994776 +0000 UTC m=+1162.725917289"
Sep 15 12:21:54.863935 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:21:54.863889    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-4a9453/test-model-4a9453-kserve-657c74dd4f-qbbxr"]
Sep 15 12:21:56.919728 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:21:56.919687    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ba366c-5c8b5d7794-rkhhq"
Sep 15 12:22:00.327232 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:00.327195    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-680229/distinct-2-w0-680229-kserve-858ddb69cc-pfn5m"]
Sep 15 12:22:00.330302 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:00.330280    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-680229/distinct-2-w0-680229-kserve-858ddb69cc-pfn5m"
Sep 15 12:22:00.333091 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:00.333067    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-680229\"/\"distinct-2-w0-680229-kserve-self-signed-certs\""
Sep 15 12:22:00.345605 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:00.345575    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-680229/distinct-2-w0-680229-kserve-858ddb69cc-pfn5m"]
Sep 15 12:22:00.421216 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:00.421177    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7970f806-b3df-4c68-9388-db776b5fc8fb-model-cache\") pod \"distinct-2-w0-680229-kserve-858ddb69cc-pfn5m\" (UID: \"7970f806-b3df-4c68-9388-db776b5fc8fb\") " pod="e2e-models-e2e-worker-w0-680229/distinct-2-w0-680229-kserve-858ddb69cc-pfn5m"
Sep 15 12:22:00.421216 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:00.421218    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7970f806-b3df-4c68-9388-db776b5fc8fb-home\") pod \"distinct-2-w0-680229-kserve-858ddb69cc-pfn5m\" (UID: \"7970f806-b3df-4c68-9388-db776b5fc8fb\") " pod="e2e-models-e2e-worker-w0-680229/distinct-2-w0-680229-kserve-858ddb69cc-pfn5m"
Sep 15 12:22:00.421485 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:00.421247    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7970f806-b3df-4c68-9388-db776b5fc8fb-dshm\") pod \"distinct-2-w0-680229-kserve-858ddb69cc-pfn5m\" (UID: \"7970f806-b3df-4c68-9388-db776b5fc8fb\") " pod="e2e-models-e2e-worker-w0-680229/distinct-2-w0-680229-kserve-858ddb69cc-pfn5m"
Sep 15 12:22:00.421485 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:00.421313    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7970f806-b3df-4c68-9388-db776b5fc8fb-tls-certs\") pod \"distinct-2-w0-680229-kserve-858ddb69cc-pfn5m\" (UID: \"7970f806-b3df-4c68-9388-db776b5fc8fb\") " pod="e2e-models-e2e-worker-w0-680229/distinct-2-w0-680229-kserve-858ddb69cc-pfn5m"
Sep 15 12:22:00.421485 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:00.421355    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hz2sp\" (UniqueName: \"kubernetes.io/projected/7970f806-b3df-4c68-9388-db776b5fc8fb-kube-api-access-hz2sp\") pod \"distinct-2-w0-680229-kserve-858ddb69cc-pfn5m\" (UID: \"7970f806-b3df-4c68-9388-db776b5fc8fb\") " pod="e2e-models-e2e-worker-w0-680229/distinct-2-w0-680229-kserve-858ddb69cc-pfn5m"
Sep 15 12:22:00.421485 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:00.421449    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7970f806-b3df-4c68-9388-db776b5fc8fb-tmp-dir\") pod \"distinct-2-w0-680229-kserve-858ddb69cc-pfn5m\" (UID: \"7970f806-b3df-4c68-9388-db776b5fc8fb\") " pod="e2e-models-e2e-worker-w0-680229/distinct-2-w0-680229-kserve-858ddb69cc-pfn5m"
Sep 15 12:22:00.522855 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:00.522813    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7970f806-b3df-4c68-9388-db776b5fc8fb-model-cache\") pod \"distinct-2-w0-680229-kserve-858ddb69cc-pfn5m\" (UID: \"7970f806-b3df-4c68-9388-db776b5fc8fb\") " pod="e2e-models-e2e-worker-w0-680229/distinct-2-w0-680229-kserve-858ddb69cc-pfn5m"
Sep 15 12:22:00.522855 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:00.522858    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7970f806-b3df-4c68-9388-db776b5fc8fb-home\") pod \"distinct-2-w0-680229-kserve-858ddb69cc-pfn5m\" (UID: \"7970f806-b3df-4c68-9388-db776b5fc8fb\") " pod="e2e-models-e2e-worker-w0-680229/distinct-2-w0-680229-kserve-858ddb69cc-pfn5m"
Sep 15 12:22:00.523120 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:00.522889    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7970f806-b3df-4c68-9388-db776b5fc8fb-dshm\") pod \"distinct-2-w0-680229-kserve-858ddb69cc-pfn5m\" (UID: \"7970f806-b3df-4c68-9388-db776b5fc8fb\") " pod="e2e-models-e2e-worker-w0-680229/distinct-2-w0-680229-kserve-858ddb69cc-pfn5m"
Sep 15 12:22:00.523120 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:00.522906    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7970f806-b3df-4c68-9388-db776b5fc8fb-tls-certs\") pod \"distinct-2-w0-680229-kserve-858ddb69cc-pfn5m\" (UID: \"7970f806-b3df-4c68-9388-db776b5fc8fb\") " pod="e2e-models-e2e-worker-w0-680229/distinct-2-w0-680229-kserve-858ddb69cc-pfn5m"
Sep 15 12:22:00.523120 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:00.523042    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hz2sp\" (UniqueName: \"kubernetes.io/projected/7970f806-b3df-4c68-9388-db776b5fc8fb-kube-api-access-hz2sp\") pod \"distinct-2-w0-680229-kserve-858ddb69cc-pfn5m\" (UID: \"7970f806-b3df-4c68-9388-db776b5fc8fb\") " pod="e2e-models-e2e-worker-w0-680229/distinct-2-w0-680229-kserve-858ddb69cc-pfn5m"
Sep 15 12:22:00.523315 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:00.523192    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7970f806-b3df-4c68-9388-db776b5fc8fb-tmp-dir\") pod \"distinct-2-w0-680229-kserve-858ddb69cc-pfn5m\" (UID: \"7970f806-b3df-4c68-9388-db776b5fc8fb\") " pod="e2e-models-e2e-worker-w0-680229/distinct-2-w0-680229-kserve-858ddb69cc-pfn5m"
Sep 15 12:22:00.523391 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:00.523320    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7970f806-b3df-4c68-9388-db776b5fc8fb-model-cache\") pod \"distinct-2-w0-680229-kserve-858ddb69cc-pfn5m\" (UID: \"7970f806-b3df-4c68-9388-db776b5fc8fb\") " pod="e2e-models-e2e-worker-w0-680229/distinct-2-w0-680229-kserve-858ddb69cc-pfn5m"
Sep 15 12:22:00.523391 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:00.523343    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7970f806-b3df-4c68-9388-db776b5fc8fb-home\") pod \"distinct-2-w0-680229-kserve-858ddb69cc-pfn5m\" (UID: \"7970f806-b3df-4c68-9388-db776b5fc8fb\") " pod="e2e-models-e2e-worker-w0-680229/distinct-2-w0-680229-kserve-858ddb69cc-pfn5m"
Sep 15 12:22:00.523582 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:00.523561    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7970f806-b3df-4c68-9388-db776b5fc8fb-tmp-dir\") pod \"distinct-2-w0-680229-kserve-858ddb69cc-pfn5m\" (UID: \"7970f806-b3df-4c68-9388-db776b5fc8fb\") " pod="e2e-models-e2e-worker-w0-680229/distinct-2-w0-680229-kserve-858ddb69cc-pfn5m"
Sep 15 12:22:00.525237 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:00.525220    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7970f806-b3df-4c68-9388-db776b5fc8fb-dshm\") pod \"distinct-2-w0-680229-kserve-858ddb69cc-pfn5m\" (UID: \"7970f806-b3df-4c68-9388-db776b5fc8fb\") " pod="e2e-models-e2e-worker-w0-680229/distinct-2-w0-680229-kserve-858ddb69cc-pfn5m"
Sep 15 12:22:00.525542 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:00.525524    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7970f806-b3df-4c68-9388-db776b5fc8fb-tls-certs\") pod \"distinct-2-w0-680229-kserve-858ddb69cc-pfn5m\" (UID: \"7970f806-b3df-4c68-9388-db776b5fc8fb\") " pod="e2e-models-e2e-worker-w0-680229/distinct-2-w0-680229-kserve-858ddb69cc-pfn5m"
Sep 15 12:22:00.534436 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:00.534412    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hz2sp\" (UniqueName: \"kubernetes.io/projected/7970f806-b3df-4c68-9388-db776b5fc8fb-kube-api-access-hz2sp\") pod \"distinct-2-w0-680229-kserve-858ddb69cc-pfn5m\" (UID: \"7970f806-b3df-4c68-9388-db776b5fc8fb\") " pod="e2e-models-e2e-worker-w0-680229/distinct-2-w0-680229-kserve-858ddb69cc-pfn5m"
Sep 15 12:22:00.640358 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:00.640324    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-680229/distinct-2-w0-680229-kserve-858ddb69cc-pfn5m"
Sep 15 12:22:00.916034 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:00.916007    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-680229/distinct-2-w0-680229-kserve-858ddb69cc-pfn5m"]
Sep 15 12:22:00.917370 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:22:00.917343    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7970f806_b3df_4c68_9388_db776b5fc8fb.slice/crio-b5b8ba9c8b7f31b131898d038f04238264a48ff71abb1d54b69f7f368842fbdc WatchSource:0}: Error finding container b5b8ba9c8b7f31b131898d038f04238264a48ff71abb1d54b69f7f368842fbdc: Status 404 returned error can't find the container with id b5b8ba9c8b7f31b131898d038f04238264a48ff71abb1d54b69f7f368842fbdc
Sep 15 12:22:01.001847 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:01.001801    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-680229/distinct-2-w0-680229-kserve-858ddb69cc-pfn5m" event={"ID":"7970f806-b3df-4c68-9388-db776b5fc8fb","Type":"ContainerStarted","Data":"2de83679c23c5e9dd32f45a27aa476d97fd75235dd1b6956679646691d65fe80"}
Sep 15 12:22:01.002026 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:01.001862    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-680229/distinct-2-w0-680229-kserve-858ddb69cc-pfn5m" event={"ID":"7970f806-b3df-4c68-9388-db776b5fc8fb","Type":"ContainerStarted","Data":"b5b8ba9c8b7f31b131898d038f04238264a48ff71abb1d54b69f7f368842fbdc"}
Sep 15 12:22:01.025077 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:01.025012    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-680229/distinct-2-w0-680229-kserve-858ddb69cc-pfn5m" podStartSLOduration=1.024991357 podStartE2EDuration="1.024991357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:22:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:22:01.023492462 +0000 UTC m=+1177.814414964" watchObservedRunningTime="2026-09-15 12:22:01.024991357 +0000 UTC m=+1177.815913870"
Sep 15 12:22:01.382857 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:01.382826    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-166e5c-openshift-default-5c447444b8-k65r8"]
Sep 15 12:22:01.383221 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:01.383083    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-166e5c-openshift-default-5c447444b8-k65r8" podUID="d8c62c8a-d028-4034-a3f0-9c7a368bbc4e" containerName="istio-proxy" containerID="cri-o://14d5d072e0575e41a5af7aa46deda0f5b9ca0edbc022ccb39f68de95f3c70615" gracePeriod=30
Sep 15 12:22:03.611217 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:03.611176    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-166e5c/noten-model-166e5c-kserve-5f746d946c-snkzb"]
Sep 15 12:22:04.194185 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:04.194147    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-69d7ba/test-model-69d7ba-kserve-8dc8bcc6c-gft8f"]
Sep 15 12:22:06.632463 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:06.632431    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-166e5c-openshift-default-5c447444b8-k65r8"
Sep 15 12:22:06.685965 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:06.685867    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8c62c8a-d028-4034-a3f0-9c7a368bbc4e-istiod-ca-cert\") pod \"d8c62c8a-d028-4034-a3f0-9c7a368bbc4e\" (UID: \"d8c62c8a-d028-4034-a3f0-9c7a368bbc4e\") "
Sep 15 12:22:06.685965 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:06.685951    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8c62c8a-d028-4034-a3f0-9c7a368bbc4e-istio-envoy\") pod \"d8c62c8a-d028-4034-a3f0-9c7a368bbc4e\" (UID: \"d8c62c8a-d028-4034-a3f0-9c7a368bbc4e\") "
Sep 15 12:22:06.686207 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:06.686002    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8c62c8a-d028-4034-a3f0-9c7a368bbc4e-workload-certs\") pod \"d8c62c8a-d028-4034-a3f0-9c7a368bbc4e\" (UID: \"d8c62c8a-d028-4034-a3f0-9c7a368bbc4e\") "
Sep 15 12:22:06.686207 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:06.686030    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8c62c8a-d028-4034-a3f0-9c7a368bbc4e-istio-podinfo\") pod \"d8c62c8a-d028-4034-a3f0-9c7a368bbc4e\" (UID: \"d8c62c8a-d028-4034-a3f0-9c7a368bbc4e\") "
Sep 15 12:22:06.686207 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:06.686058    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8c62c8a-d028-4034-a3f0-9c7a368bbc4e-credential-socket\") pod \"d8c62c8a-d028-4034-a3f0-9c7a368bbc4e\" (UID: \"d8c62c8a-d028-4034-a3f0-9c7a368bbc4e\") "
Sep 15 12:22:06.686207 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:06.686102    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8c62c8a-d028-4034-a3f0-9c7a368bbc4e-workload-socket\") pod \"d8c62c8a-d028-4034-a3f0-9c7a368bbc4e\" (UID: \"d8c62c8a-d028-4034-a3f0-9c7a368bbc4e\") "
Sep 15 12:22:06.686207 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:06.686161    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-65bm8\" (UniqueName: \"kubernetes.io/projected/d8c62c8a-d028-4034-a3f0-9c7a368bbc4e-kube-api-access-65bm8\") pod \"d8c62c8a-d028-4034-a3f0-9c7a368bbc4e\" (UID: \"d8c62c8a-d028-4034-a3f0-9c7a368bbc4e\") "
Sep 15 12:22:06.686519 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:06.686236    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8c62c8a-d028-4034-a3f0-9c7a368bbc4e-istio-token\") pod \"d8c62c8a-d028-4034-a3f0-9c7a368bbc4e\" (UID: \"d8c62c8a-d028-4034-a3f0-9c7a368bbc4e\") "
Sep 15 12:22:06.686519 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:06.686301    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8c62c8a-d028-4034-a3f0-9c7a368bbc4e-istio-data\") pod \"d8c62c8a-d028-4034-a3f0-9c7a368bbc4e\" (UID: \"d8c62c8a-d028-4034-a3f0-9c7a368bbc4e\") "
Sep 15 12:22:06.686519 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:06.686325    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8c62c8a-d028-4034-a3f0-9c7a368bbc4e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d8c62c8a-d028-4034-a3f0-9c7a368bbc4e" (UID: "d8c62c8a-d028-4034-a3f0-9c7a368bbc4e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:22:06.686519 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:06.686493    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8c62c8a-d028-4034-a3f0-9c7a368bbc4e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d8c62c8a-d028-4034-a3f0-9c7a368bbc4e" (UID: "d8c62c8a-d028-4034-a3f0-9c7a368bbc4e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:22:06.686740 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:06.686595    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8c62c8a-d028-4034-a3f0-9c7a368bbc4e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d8c62c8a-d028-4034-a3f0-9c7a368bbc4e" (UID: "d8c62c8a-d028-4034-a3f0-9c7a368bbc4e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:22:06.686740 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:06.686715    2555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8c62c8a-d028-4034-a3f0-9c7a368bbc4e-workload-certs\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:22:06.686740 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:06.686718    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8c62c8a-d028-4034-a3f0-9c7a368bbc4e-istio-data" (OuterVolumeSpecName: "istio-data") pod "d8c62c8a-d028-4034-a3f0-9c7a368bbc4e" (UID: "d8c62c8a-d028-4034-a3f0-9c7a368bbc4e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:22:06.686740 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:06.686733    2555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8c62c8a-d028-4034-a3f0-9c7a368bbc4e-workload-socket\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:22:06.686958 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:06.686747    2555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8c62c8a-d028-4034-a3f0-9c7a368bbc4e-istiod-ca-cert\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:22:06.686958 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:06.686798    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8c62c8a-d028-4034-a3f0-9c7a368bbc4e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d8c62c8a-d028-4034-a3f0-9c7a368bbc4e" (UID: "d8c62c8a-d028-4034-a3f0-9c7a368bbc4e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:22:06.689094 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:06.689065    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8c62c8a-d028-4034-a3f0-9c7a368bbc4e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d8c62c8a-d028-4034-a3f0-9c7a368bbc4e" (UID: "d8c62c8a-d028-4034-a3f0-9c7a368bbc4e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:22:06.689229 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:06.689112    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d8c62c8a-d028-4034-a3f0-9c7a368bbc4e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d8c62c8a-d028-4034-a3f0-9c7a368bbc4e" (UID: "d8c62c8a-d028-4034-a3f0-9c7a368bbc4e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 12:22:06.689229 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:06.689138    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8c62c8a-d028-4034-a3f0-9c7a368bbc4e-istio-token" (OuterVolumeSpecName: "istio-token") pod "d8c62c8a-d028-4034-a3f0-9c7a368bbc4e" (UID: "d8c62c8a-d028-4034-a3f0-9c7a368bbc4e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:22:06.689229 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:06.689159    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8c62c8a-d028-4034-a3f0-9c7a368bbc4e-kube-api-access-65bm8" (OuterVolumeSpecName: "kube-api-access-65bm8") pod "d8c62c8a-d028-4034-a3f0-9c7a368bbc4e" (UID: "d8c62c8a-d028-4034-a3f0-9c7a368bbc4e"). InnerVolumeSpecName "kube-api-access-65bm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:22:06.787897 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:06.787858    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65bm8\" (UniqueName: \"kubernetes.io/projected/d8c62c8a-d028-4034-a3f0-9c7a368bbc4e-kube-api-access-65bm8\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:22:06.787897 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:06.787891    2555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8c62c8a-d028-4034-a3f0-9c7a368bbc4e-istio-token\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:22:06.787897 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:06.787901    2555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8c62c8a-d028-4034-a3f0-9c7a368bbc4e-istio-data\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:22:06.787897 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:06.787908    2555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8c62c8a-d028-4034-a3f0-9c7a368bbc4e-istio-envoy\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:22:06.788169 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:06.787916    2555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8c62c8a-d028-4034-a3f0-9c7a368bbc4e-istio-podinfo\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:22:06.788169 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:06.787924    2555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8c62c8a-d028-4034-a3f0-9c7a368bbc4e-credential-socket\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:22:07.037610 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:07.037522    2555 generic.go:356] "Generic (PLEG): container finished" podID="d8c62c8a-d028-4034-a3f0-9c7a368bbc4e" containerID="14d5d072e0575e41a5af7aa46deda0f5b9ca0edbc022ccb39f68de95f3c70615" exitCode=0
Sep 15 12:22:07.037610 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:07.037573    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-166e5c-openshift-default-5c447444b8-k65r8"
Sep 15 12:22:07.037801 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:07.037614    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-166e5c-openshift-default-5c447444b8-k65r8" event={"ID":"d8c62c8a-d028-4034-a3f0-9c7a368bbc4e","Type":"ContainerDied","Data":"14d5d072e0575e41a5af7aa46deda0f5b9ca0edbc022ccb39f68de95f3c70615"}
Sep 15 12:22:07.037801 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:07.037673    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-166e5c-openshift-default-5c447444b8-k65r8" event={"ID":"d8c62c8a-d028-4034-a3f0-9c7a368bbc4e","Type":"ContainerDied","Data":"7d78c4410cd1c7e4ce1b73b6f82444ef965f4aa56c09e901d6e368c1dca41b21"}
Sep 15 12:22:07.037801 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:07.037695    2555 scope.go:117] "RemoveContainer" containerID="14d5d072e0575e41a5af7aa46deda0f5b9ca0edbc022ccb39f68de95f3c70615"
Sep 15 12:22:07.047828 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:07.047804    2555 scope.go:117] "RemoveContainer" containerID="14d5d072e0575e41a5af7aa46deda0f5b9ca0edbc022ccb39f68de95f3c70615"
Sep 15 12:22:07.048171 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:22:07.048146    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14d5d072e0575e41a5af7aa46deda0f5b9ca0edbc022ccb39f68de95f3c70615\": container with ID starting with 14d5d072e0575e41a5af7aa46deda0f5b9ca0edbc022ccb39f68de95f3c70615 not found: ID does not exist" containerID="14d5d072e0575e41a5af7aa46deda0f5b9ca0edbc022ccb39f68de95f3c70615"
Sep 15 12:22:07.048256 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:07.048181    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14d5d072e0575e41a5af7aa46deda0f5b9ca0edbc022ccb39f68de95f3c70615"} err="failed to get container status \"14d5d072e0575e41a5af7aa46deda0f5b9ca0edbc022ccb39f68de95f3c70615\": rpc error: code = NotFound desc = could not find container \"14d5d072e0575e41a5af7aa46deda0f5b9ca0edbc022ccb39f68de95f3c70615\": container with ID starting with 14d5d072e0575e41a5af7aa46deda0f5b9ca0edbc022ccb39f68de95f3c70615 not found: ID does not exist"
Sep 15 12:22:07.082589 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:07.082559    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-166e5c-openshift-default-5c447444b8-k65r8"]
Sep 15 12:22:07.094850 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:07.094821    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-166e5c-openshift-default-5c447444b8-k65r8"]
Sep 15 12:22:07.818055 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:07.818020    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8c62c8a-d028-4034-a3f0-9c7a368bbc4e" path="/var/lib/kubelet/pods/d8c62c8a-d028-4034-a3f0-9c7a368bbc4e/volumes"
Sep 15 12:22:09.883155 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:09.883109    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-4a9453/test-model-4a9453-kserve-657c74dd4f-qbbxr" podUID="78cf1c6e-98e8-4296-97ed-00140d47133d" containerName="main" containerID="cri-o://7291105c3578a3f6ce82e5ece08104b032f79bb1ee59ff94171d9729f8453cf8" gracePeriod=45
Sep 15 12:22:10.061216 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:10.061188    2555 generic.go:356] "Generic (PLEG): container finished" podID="78cf1c6e-98e8-4296-97ed-00140d47133d" containerID="7291105c3578a3f6ce82e5ece08104b032f79bb1ee59ff94171d9729f8453cf8" exitCode=0
Sep 15 12:22:10.061403 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:10.061275    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4a9453/test-model-4a9453-kserve-657c74dd4f-qbbxr" event={"ID":"78cf1c6e-98e8-4296-97ed-00140d47133d","Type":"ContainerDied","Data":"7291105c3578a3f6ce82e5ece08104b032f79bb1ee59ff94171d9729f8453cf8"}
Sep 15 12:22:10.121596 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:10.121572    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4a9453/test-model-4a9453-kserve-657c74dd4f-qbbxr"
Sep 15 12:22:10.222484 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:10.222388    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78cf1c6e-98e8-4296-97ed-00140d47133d-tmp-dir\") pod \"78cf1c6e-98e8-4296-97ed-00140d47133d\" (UID: \"78cf1c6e-98e8-4296-97ed-00140d47133d\") "
Sep 15 12:22:10.222484 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:10.222442    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78cf1c6e-98e8-4296-97ed-00140d47133d-tls-certs\") pod \"78cf1c6e-98e8-4296-97ed-00140d47133d\" (UID: \"78cf1c6e-98e8-4296-97ed-00140d47133d\") "
Sep 15 12:22:10.222721 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:10.222499    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78cf1c6e-98e8-4296-97ed-00140d47133d-dshm\") pod \"78cf1c6e-98e8-4296-97ed-00140d47133d\" (UID: \"78cf1c6e-98e8-4296-97ed-00140d47133d\") "
Sep 15 12:22:10.222721 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:10.222560    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vcxgq\" (UniqueName: \"kubernetes.io/projected/78cf1c6e-98e8-4296-97ed-00140d47133d-kube-api-access-vcxgq\") pod \"78cf1c6e-98e8-4296-97ed-00140d47133d\" (UID: \"78cf1c6e-98e8-4296-97ed-00140d47133d\") "
Sep 15 12:22:10.222721 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:10.222587    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78cf1c6e-98e8-4296-97ed-00140d47133d-home\") pod \"78cf1c6e-98e8-4296-97ed-00140d47133d\" (UID: \"78cf1c6e-98e8-4296-97ed-00140d47133d\") "
Sep 15 12:22:10.222721 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:10.222613    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78cf1c6e-98e8-4296-97ed-00140d47133d-model-cache\") pod \"78cf1c6e-98e8-4296-97ed-00140d47133d\" (UID: \"78cf1c6e-98e8-4296-97ed-00140d47133d\") "
Sep 15 12:22:10.222721 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:10.222643    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78cf1c6e-98e8-4296-97ed-00140d47133d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "78cf1c6e-98e8-4296-97ed-00140d47133d" (UID: "78cf1c6e-98e8-4296-97ed-00140d47133d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:22:10.222990 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:10.222910    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78cf1c6e-98e8-4296-97ed-00140d47133d-tmp-dir\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:22:10.222990 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:10.222926    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78cf1c6e-98e8-4296-97ed-00140d47133d-model-cache" (OuterVolumeSpecName: "model-cache") pod "78cf1c6e-98e8-4296-97ed-00140d47133d" (UID: "78cf1c6e-98e8-4296-97ed-00140d47133d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:22:10.222990 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:10.222942    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78cf1c6e-98e8-4296-97ed-00140d47133d-home" (OuterVolumeSpecName: "home") pod "78cf1c6e-98e8-4296-97ed-00140d47133d" (UID: "78cf1c6e-98e8-4296-97ed-00140d47133d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:22:10.224781 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:10.224752    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78cf1c6e-98e8-4296-97ed-00140d47133d-dshm" (OuterVolumeSpecName: "dshm") pod "78cf1c6e-98e8-4296-97ed-00140d47133d" (UID: "78cf1c6e-98e8-4296-97ed-00140d47133d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:22:10.224781 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:10.224764    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78cf1c6e-98e8-4296-97ed-00140d47133d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "78cf1c6e-98e8-4296-97ed-00140d47133d" (UID: "78cf1c6e-98e8-4296-97ed-00140d47133d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:22:10.224781 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:10.224778    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78cf1c6e-98e8-4296-97ed-00140d47133d-kube-api-access-vcxgq" (OuterVolumeSpecName: "kube-api-access-vcxgq") pod "78cf1c6e-98e8-4296-97ed-00140d47133d" (UID: "78cf1c6e-98e8-4296-97ed-00140d47133d"). InnerVolumeSpecName "kube-api-access-vcxgq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:22:10.323844 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:10.323802    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78cf1c6e-98e8-4296-97ed-00140d47133d-home\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:22:10.323844 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:10.323836    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78cf1c6e-98e8-4296-97ed-00140d47133d-model-cache\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:22:10.323844 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:10.323846    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78cf1c6e-98e8-4296-97ed-00140d47133d-tls-certs\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:22:10.323844 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:10.323854    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78cf1c6e-98e8-4296-97ed-00140d47133d-dshm\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:22:10.324114 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:10.323862    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcxgq\" (UniqueName: \"kubernetes.io/projected/78cf1c6e-98e8-4296-97ed-00140d47133d-kube-api-access-vcxgq\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:22:10.641057 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:10.641022    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-680229/distinct-2-w0-680229-kserve-858ddb69cc-pfn5m"
Sep 15 12:22:10.641226 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:10.641081    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-680229/distinct-2-w0-680229-kserve-858ddb69cc-pfn5m"
Sep 15 12:22:10.657524 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:10.657492    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-680229/distinct-2-w0-680229-kserve-858ddb69cc-pfn5m"
Sep 15 12:22:11.074699 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:11.074613    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4a9453/test-model-4a9453-kserve-657c74dd4f-qbbxr"
Sep 15 12:22:11.074699 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:11.074650    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4a9453/test-model-4a9453-kserve-657c74dd4f-qbbxr" event={"ID":"78cf1c6e-98e8-4296-97ed-00140d47133d","Type":"ContainerDied","Data":"52d853e7e42f6f2d761e361b0594a75cefe691fd01beb2a4957ea381fa8d0258"}
Sep 15 12:22:11.075392 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:11.074711    2555 scope.go:117] "RemoveContainer" containerID="7291105c3578a3f6ce82e5ece08104b032f79bb1ee59ff94171d9729f8453cf8"
Sep 15 12:22:11.087081 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:11.087053    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-680229/distinct-2-w0-680229-kserve-858ddb69cc-pfn5m"
Sep 15 12:22:11.098148 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:11.098120    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-4a9453/test-model-4a9453-kserve-657c74dd4f-qbbxr"]
Sep 15 12:22:11.106598 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:11.106572    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-4a9453/test-model-4a9453-kserve-657c74dd4f-qbbxr"]
Sep 15 12:22:11.817993 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:11.817957    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78cf1c6e-98e8-4296-97ed-00140d47133d" path="/var/lib/kubelet/pods/78cf1c6e-98e8-4296-97ed-00140d47133d/volumes"
Sep 15 12:22:11.917362 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:11.917326    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-9ba48f/e2e-conflict-b-a7f06f2e-kserve-7ff6c7b9db-4p67k"]
Sep 15 12:22:11.917718 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:11.917704    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8c62c8a-d028-4034-a3f0-9c7a368bbc4e" containerName="istio-proxy"
Sep 15 12:22:11.917765 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:11.917720    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8c62c8a-d028-4034-a3f0-9c7a368bbc4e" containerName="istio-proxy"
Sep 15 12:22:11.917765 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:11.917735    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78cf1c6e-98e8-4296-97ed-00140d47133d" containerName="main"
Sep 15 12:22:11.917765 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:11.917741    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="78cf1c6e-98e8-4296-97ed-00140d47133d" containerName="main"
Sep 15 12:22:11.917868 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:11.917815    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="78cf1c6e-98e8-4296-97ed-00140d47133d" containerName="main"
Sep 15 12:22:11.917868 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:11.917825    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8c62c8a-d028-4034-a3f0-9c7a368bbc4e" containerName="istio-proxy"
Sep 15 12:22:11.922172 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:11.922150    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9ba48f/e2e-conflict-b-a7f06f2e-kserve-7ff6c7b9db-4p67k"
Sep 15 12:22:11.929417 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:11.929391    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-9ba48f\"/\"e2e-conflict-b-a7f06f2e-kserve-self-signed-certs\""
Sep 15 12:22:11.941466 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:11.941436    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-9ba48f/e2e-conflict-b-a7f06f2e-kserve-7ff6c7b9db-4p67k"]
Sep 15 12:22:12.039068 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:12.039026    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4c30438-42bc-4d52-be9a-0af5a15eff23-tls-certs\") pod \"e2e-conflict-b-a7f06f2e-kserve-7ff6c7b9db-4p67k\" (UID: \"a4c30438-42bc-4d52-be9a-0af5a15eff23\") " pod="e2e-models-e2e-worker-w4-9ba48f/e2e-conflict-b-a7f06f2e-kserve-7ff6c7b9db-4p67k"
Sep 15 12:22:12.039068 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:12.039067    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6c26\" (UniqueName: \"kubernetes.io/projected/a4c30438-42bc-4d52-be9a-0af5a15eff23-kube-api-access-s6c26\") pod \"e2e-conflict-b-a7f06f2e-kserve-7ff6c7b9db-4p67k\" (UID: \"a4c30438-42bc-4d52-be9a-0af5a15eff23\") " pod="e2e-models-e2e-worker-w4-9ba48f/e2e-conflict-b-a7f06f2e-kserve-7ff6c7b9db-4p67k"
Sep 15 12:22:12.039332 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:12.039101    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4c30438-42bc-4d52-be9a-0af5a15eff23-tmp-dir\") pod \"e2e-conflict-b-a7f06f2e-kserve-7ff6c7b9db-4p67k\" (UID: \"a4c30438-42bc-4d52-be9a-0af5a15eff23\") " pod="e2e-models-e2e-worker-w4-9ba48f/e2e-conflict-b-a7f06f2e-kserve-7ff6c7b9db-4p67k"
Sep 15 12:22:12.039332 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:12.039165    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4c30438-42bc-4d52-be9a-0af5a15eff23-model-cache\") pod \"e2e-conflict-b-a7f06f2e-kserve-7ff6c7b9db-4p67k\" (UID: \"a4c30438-42bc-4d52-be9a-0af5a15eff23\") " pod="e2e-models-e2e-worker-w4-9ba48f/e2e-conflict-b-a7f06f2e-kserve-7ff6c7b9db-4p67k"
Sep 15 12:22:12.039332 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:12.039196    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4c30438-42bc-4d52-be9a-0af5a15eff23-home\") pod \"e2e-conflict-b-a7f06f2e-kserve-7ff6c7b9db-4p67k\" (UID: \"a4c30438-42bc-4d52-be9a-0af5a15eff23\") " pod="e2e-models-e2e-worker-w4-9ba48f/e2e-conflict-b-a7f06f2e-kserve-7ff6c7b9db-4p67k"
Sep 15 12:22:12.039332 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:12.039317    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4c30438-42bc-4d52-be9a-0af5a15eff23-dshm\") pod \"e2e-conflict-b-a7f06f2e-kserve-7ff6c7b9db-4p67k\" (UID: \"a4c30438-42bc-4d52-be9a-0af5a15eff23\") " pod="e2e-models-e2e-worker-w4-9ba48f/e2e-conflict-b-a7f06f2e-kserve-7ff6c7b9db-4p67k"
Sep 15 12:22:12.140099 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:12.140065    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4c30438-42bc-4d52-be9a-0af5a15eff23-tls-certs\") pod \"e2e-conflict-b-a7f06f2e-kserve-7ff6c7b9db-4p67k\" (UID: \"a4c30438-42bc-4d52-be9a-0af5a15eff23\") " pod="e2e-models-e2e-worker-w4-9ba48f/e2e-conflict-b-a7f06f2e-kserve-7ff6c7b9db-4p67k"
Sep 15 12:22:12.140528 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:12.140107    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6c26\" (UniqueName: \"kubernetes.io/projected/a4c30438-42bc-4d52-be9a-0af5a15eff23-kube-api-access-s6c26\") pod \"e2e-conflict-b-a7f06f2e-kserve-7ff6c7b9db-4p67k\" (UID: \"a4c30438-42bc-4d52-be9a-0af5a15eff23\") " pod="e2e-models-e2e-worker-w4-9ba48f/e2e-conflict-b-a7f06f2e-kserve-7ff6c7b9db-4p67k"
Sep 15 12:22:12.140528 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:12.140145    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4c30438-42bc-4d52-be9a-0af5a15eff23-tmp-dir\") pod \"e2e-conflict-b-a7f06f2e-kserve-7ff6c7b9db-4p67k\" (UID: \"a4c30438-42bc-4d52-be9a-0af5a15eff23\") " pod="e2e-models-e2e-worker-w4-9ba48f/e2e-conflict-b-a7f06f2e-kserve-7ff6c7b9db-4p67k"
Sep 15 12:22:12.140528 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:12.140175    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4c30438-42bc-4d52-be9a-0af5a15eff23-model-cache\") pod \"e2e-conflict-b-a7f06f2e-kserve-7ff6c7b9db-4p67k\" (UID: \"a4c30438-42bc-4d52-be9a-0af5a15eff23\") " pod="e2e-models-e2e-worker-w4-9ba48f/e2e-conflict-b-a7f06f2e-kserve-7ff6c7b9db-4p67k"
Sep 15 12:22:12.140528 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:12.140192    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4c30438-42bc-4d52-be9a-0af5a15eff23-home\") pod \"e2e-conflict-b-a7f06f2e-kserve-7ff6c7b9db-4p67k\" (UID: \"a4c30438-42bc-4d52-be9a-0af5a15eff23\") " pod="e2e-models-e2e-worker-w4-9ba48f/e2e-conflict-b-a7f06f2e-kserve-7ff6c7b9db-4p67k"
Sep 15 12:22:12.140528 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:12.140447    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4c30438-42bc-4d52-be9a-0af5a15eff23-dshm\") pod \"e2e-conflict-b-a7f06f2e-kserve-7ff6c7b9db-4p67k\" (UID: \"a4c30438-42bc-4d52-be9a-0af5a15eff23\") " pod="e2e-models-e2e-worker-w4-9ba48f/e2e-conflict-b-a7f06f2e-kserve-7ff6c7b9db-4p67k"
Sep 15 12:22:12.140809 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:12.140580    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4c30438-42bc-4d52-be9a-0af5a15eff23-tmp-dir\") pod \"e2e-conflict-b-a7f06f2e-kserve-7ff6c7b9db-4p67k\" (UID: \"a4c30438-42bc-4d52-be9a-0af5a15eff23\") " pod="e2e-models-e2e-worker-w4-9ba48f/e2e-conflict-b-a7f06f2e-kserve-7ff6c7b9db-4p67k"
Sep 15 12:22:12.140809 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:12.140604    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4c30438-42bc-4d52-be9a-0af5a15eff23-model-cache\") pod \"e2e-conflict-b-a7f06f2e-kserve-7ff6c7b9db-4p67k\" (UID: \"a4c30438-42bc-4d52-be9a-0af5a15eff23\") " pod="e2e-models-e2e-worker-w4-9ba48f/e2e-conflict-b-a7f06f2e-kserve-7ff6c7b9db-4p67k"
Sep 15 12:22:12.140809 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:12.140654    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4c30438-42bc-4d52-be9a-0af5a15eff23-home\") pod \"e2e-conflict-b-a7f06f2e-kserve-7ff6c7b9db-4p67k\" (UID: \"a4c30438-42bc-4d52-be9a-0af5a15eff23\") " pod="e2e-models-e2e-worker-w4-9ba48f/e2e-conflict-b-a7f06f2e-kserve-7ff6c7b9db-4p67k"
Sep 15 12:22:12.142588 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:12.142567    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4c30438-42bc-4d52-be9a-0af5a15eff23-dshm\") pod \"e2e-conflict-b-a7f06f2e-kserve-7ff6c7b9db-4p67k\" (UID: \"a4c30438-42bc-4d52-be9a-0af5a15eff23\") " pod="e2e-models-e2e-worker-w4-9ba48f/e2e-conflict-b-a7f06f2e-kserve-7ff6c7b9db-4p67k"
Sep 15 12:22:12.142729 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:12.142713    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4c30438-42bc-4d52-be9a-0af5a15eff23-tls-certs\") pod \"e2e-conflict-b-a7f06f2e-kserve-7ff6c7b9db-4p67k\" (UID: \"a4c30438-42bc-4d52-be9a-0af5a15eff23\") " pod="e2e-models-e2e-worker-w4-9ba48f/e2e-conflict-b-a7f06f2e-kserve-7ff6c7b9db-4p67k"
Sep 15 12:22:12.151966 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:12.151937    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6c26\" (UniqueName: \"kubernetes.io/projected/a4c30438-42bc-4d52-be9a-0af5a15eff23-kube-api-access-s6c26\") pod \"e2e-conflict-b-a7f06f2e-kserve-7ff6c7b9db-4p67k\" (UID: \"a4c30438-42bc-4d52-be9a-0af5a15eff23\") " pod="e2e-models-e2e-worker-w4-9ba48f/e2e-conflict-b-a7f06f2e-kserve-7ff6c7b9db-4p67k"
Sep 15 12:22:12.233651 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:12.233607    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9ba48f/e2e-conflict-b-a7f06f2e-kserve-7ff6c7b9db-4p67k"
Sep 15 12:22:12.512868 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:12.512840    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-9ba48f/e2e-conflict-b-a7f06f2e-kserve-7ff6c7b9db-4p67k"]
Sep 15 12:22:12.514782 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:22:12.514748    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda4c30438_42bc_4d52_be9a_0af5a15eff23.slice/crio-f79031a144a47bb2a2d77559e2f5d65b335857cfc930fc3f4826f727244d081c WatchSource:0}: Error finding container f79031a144a47bb2a2d77559e2f5d65b335857cfc930fc3f4826f727244d081c: Status 404 returned error can't find the container with id f79031a144a47bb2a2d77559e2f5d65b335857cfc930fc3f4826f727244d081c
Sep 15 12:22:13.090424 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:13.090375    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9ba48f/e2e-conflict-b-a7f06f2e-kserve-7ff6c7b9db-4p67k" event={"ID":"a4c30438-42bc-4d52-be9a-0af5a15eff23","Type":"ContainerStarted","Data":"04859da4770b91800ba1940a2c431323db09d4daf95154b38567b3cd25f6a0ed"}
Sep 15 12:22:13.090424 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:13.090432    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9ba48f/e2e-conflict-b-a7f06f2e-kserve-7ff6c7b9db-4p67k" event={"ID":"a4c30438-42bc-4d52-be9a-0af5a15eff23","Type":"ContainerStarted","Data":"f79031a144a47bb2a2d77559e2f5d65b335857cfc930fc3f4826f727244d081c"}
Sep 15 12:22:13.117014 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:13.116953    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-9ba48f/e2e-conflict-b-a7f06f2e-kserve-7ff6c7b9db-4p67k" podStartSLOduration=2.11693367 podStartE2EDuration="2.11693367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:22:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:22:13.114733893 +0000 UTC m=+1189.905656406" watchObservedRunningTime="2026-09-15 12:22:13.11693367 +0000 UTC m=+1189.907856213"
Sep 15 12:22:18.631018 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:18.630973    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-166e5c/noten-model-166e5c-kserve-5f746d946c-snkzb" podUID="b04a7844-49ed-48a4-8ef0-7aad3eee1aea" containerName="main" containerID="cri-o://4a6c036d6ec67d4c19f5e3cffac473ec301776f76bc259cc30f119be039f33c8" gracePeriod=45
Sep 15 12:22:18.884357 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:18.884338    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-166e5c/noten-model-166e5c-kserve-5f746d946c-snkzb"
Sep 15 12:22:19.008411 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:19.008366    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b04a7844-49ed-48a4-8ef0-7aad3eee1aea-tls-certs\") pod \"b04a7844-49ed-48a4-8ef0-7aad3eee1aea\" (UID: \"b04a7844-49ed-48a4-8ef0-7aad3eee1aea\") "
Sep 15 12:22:19.008582 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:19.008430    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mq7wp\" (UniqueName: \"kubernetes.io/projected/b04a7844-49ed-48a4-8ef0-7aad3eee1aea-kube-api-access-mq7wp\") pod \"b04a7844-49ed-48a4-8ef0-7aad3eee1aea\" (UID: \"b04a7844-49ed-48a4-8ef0-7aad3eee1aea\") "
Sep 15 12:22:19.008582 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:19.008482    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b04a7844-49ed-48a4-8ef0-7aad3eee1aea-tmp-dir\") pod \"b04a7844-49ed-48a4-8ef0-7aad3eee1aea\" (UID: \"b04a7844-49ed-48a4-8ef0-7aad3eee1aea\") "
Sep 15 12:22:19.008582 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:19.008516    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b04a7844-49ed-48a4-8ef0-7aad3eee1aea-home\") pod \"b04a7844-49ed-48a4-8ef0-7aad3eee1aea\" (UID: \"b04a7844-49ed-48a4-8ef0-7aad3eee1aea\") "
Sep 15 12:22:19.008582 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:19.008547    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b04a7844-49ed-48a4-8ef0-7aad3eee1aea-model-cache\") pod \"b04a7844-49ed-48a4-8ef0-7aad3eee1aea\" (UID: \"b04a7844-49ed-48a4-8ef0-7aad3eee1aea\") "
Sep 15 12:22:19.008792 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:19.008609    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b04a7844-49ed-48a4-8ef0-7aad3eee1aea-dshm\") pod \"b04a7844-49ed-48a4-8ef0-7aad3eee1aea\" (UID: \"b04a7844-49ed-48a4-8ef0-7aad3eee1aea\") "
Sep 15 12:22:19.008792 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:19.008756    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b04a7844-49ed-48a4-8ef0-7aad3eee1aea-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b04a7844-49ed-48a4-8ef0-7aad3eee1aea" (UID: "b04a7844-49ed-48a4-8ef0-7aad3eee1aea"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:22:19.008898 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:19.008816    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b04a7844-49ed-48a4-8ef0-7aad3eee1aea-home" (OuterVolumeSpecName: "home") pod "b04a7844-49ed-48a4-8ef0-7aad3eee1aea" (UID: "b04a7844-49ed-48a4-8ef0-7aad3eee1aea"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:22:19.008963 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:19.008898    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b04a7844-49ed-48a4-8ef0-7aad3eee1aea-model-cache" (OuterVolumeSpecName: "model-cache") pod "b04a7844-49ed-48a4-8ef0-7aad3eee1aea" (UID: "b04a7844-49ed-48a4-8ef0-7aad3eee1aea"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:22:19.009079 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:19.009061    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b04a7844-49ed-48a4-8ef0-7aad3eee1aea-tmp-dir\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:22:19.009132 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:19.009081    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b04a7844-49ed-48a4-8ef0-7aad3eee1aea-home\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:22:19.009132 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:19.009093    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b04a7844-49ed-48a4-8ef0-7aad3eee1aea-model-cache\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:22:19.010862 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:19.010831    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b04a7844-49ed-48a4-8ef0-7aad3eee1aea-kube-api-access-mq7wp" (OuterVolumeSpecName: "kube-api-access-mq7wp") pod "b04a7844-49ed-48a4-8ef0-7aad3eee1aea" (UID: "b04a7844-49ed-48a4-8ef0-7aad3eee1aea"). InnerVolumeSpecName "kube-api-access-mq7wp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:22:19.010862 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:19.010853    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b04a7844-49ed-48a4-8ef0-7aad3eee1aea-dshm" (OuterVolumeSpecName: "dshm") pod "b04a7844-49ed-48a4-8ef0-7aad3eee1aea" (UID: "b04a7844-49ed-48a4-8ef0-7aad3eee1aea"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:22:19.011006 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:19.010841    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b04a7844-49ed-48a4-8ef0-7aad3eee1aea-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b04a7844-49ed-48a4-8ef0-7aad3eee1aea" (UID: "b04a7844-49ed-48a4-8ef0-7aad3eee1aea"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:22:19.110913 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:19.110860    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b04a7844-49ed-48a4-8ef0-7aad3eee1aea-tls-certs\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:22:19.111115 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:19.110922    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mq7wp\" (UniqueName: \"kubernetes.io/projected/b04a7844-49ed-48a4-8ef0-7aad3eee1aea-kube-api-access-mq7wp\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:22:19.111115 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:19.110952    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b04a7844-49ed-48a4-8ef0-7aad3eee1aea-dshm\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:22:19.131159 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:19.131108    2555 generic.go:356] "Generic (PLEG): container finished" podID="b04a7844-49ed-48a4-8ef0-7aad3eee1aea" containerID="4a6c036d6ec67d4c19f5e3cffac473ec301776f76bc259cc30f119be039f33c8" exitCode=0
Sep 15 12:22:19.131385 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:19.131169    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-166e5c/noten-model-166e5c-kserve-5f746d946c-snkzb"
Sep 15 12:22:19.131385 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:19.131193    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-166e5c/noten-model-166e5c-kserve-5f746d946c-snkzb" event={"ID":"b04a7844-49ed-48a4-8ef0-7aad3eee1aea","Type":"ContainerDied","Data":"4a6c036d6ec67d4c19f5e3cffac473ec301776f76bc259cc30f119be039f33c8"}
Sep 15 12:22:19.131385 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:19.131247    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-166e5c/noten-model-166e5c-kserve-5f746d946c-snkzb" event={"ID":"b04a7844-49ed-48a4-8ef0-7aad3eee1aea","Type":"ContainerDied","Data":"b7e0255ee2838c52c48c62d2888bd0c2cc92caccdabf096ae839f7dcb49a11cf"}
Sep 15 12:22:19.131385 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:19.131291    2555 scope.go:117] "RemoveContainer" containerID="4a6c036d6ec67d4c19f5e3cffac473ec301776f76bc259cc30f119be039f33c8"
Sep 15 12:22:19.141715 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:19.141691    2555 scope.go:117] "RemoveContainer" containerID="4a6c036d6ec67d4c19f5e3cffac473ec301776f76bc259cc30f119be039f33c8"
Sep 15 12:22:19.142023 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:22:19.142004    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a6c036d6ec67d4c19f5e3cffac473ec301776f76bc259cc30f119be039f33c8\": container with ID starting with 4a6c036d6ec67d4c19f5e3cffac473ec301776f76bc259cc30f119be039f33c8 not found: ID does not exist" containerID="4a6c036d6ec67d4c19f5e3cffac473ec301776f76bc259cc30f119be039f33c8"
Sep 15 12:22:19.142069 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:19.142035    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a6c036d6ec67d4c19f5e3cffac473ec301776f76bc259cc30f119be039f33c8"} err="failed to get container status \"4a6c036d6ec67d4c19f5e3cffac473ec301776f76bc259cc30f119be039f33c8\": rpc error: code = NotFound desc = could not find container \"4a6c036d6ec67d4c19f5e3cffac473ec301776f76bc259cc30f119be039f33c8\": container with ID starting with 4a6c036d6ec67d4c19f5e3cffac473ec301776f76bc259cc30f119be039f33c8 not found: ID does not exist"
Sep 15 12:22:19.161284 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:19.161241    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-166e5c/noten-model-166e5c-kserve-5f746d946c-snkzb"]
Sep 15 12:22:19.164126 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:19.164097    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-166e5c/noten-model-166e5c-kserve-5f746d946c-snkzb"]
Sep 15 12:22:19.213519 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:19.213479    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-69d7ba/test-model-69d7ba-kserve-8dc8bcc6c-gft8f" podUID="84978e46-13ee-4363-a728-9510efe618bb" containerName="main" containerID="cri-o://a9579d1eb1f7d5f47264c9e0e1715985afb364f47071a0fb6b65968d5305a1e5" gracePeriod=45
Sep 15 12:22:19.455121 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:19.455096    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-69d7ba/test-model-69d7ba-kserve-8dc8bcc6c-gft8f"
Sep 15 12:22:19.614976 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:19.614935    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84978e46-13ee-4363-a728-9510efe618bb-home\") pod \"84978e46-13ee-4363-a728-9510efe618bb\" (UID: \"84978e46-13ee-4363-a728-9510efe618bb\") "
Sep 15 12:22:19.615180 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:19.615005    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84978e46-13ee-4363-a728-9510efe618bb-tmp-dir\") pod \"84978e46-13ee-4363-a728-9510efe618bb\" (UID: \"84978e46-13ee-4363-a728-9510efe618bb\") "
Sep 15 12:22:19.615180 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:19.615038    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84978e46-13ee-4363-a728-9510efe618bb-dshm\") pod \"84978e46-13ee-4363-a728-9510efe618bb\" (UID: \"84978e46-13ee-4363-a728-9510efe618bb\") "
Sep 15 12:22:19.615180 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:19.615140    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84978e46-13ee-4363-a728-9510efe618bb-tls-certs\") pod \"84978e46-13ee-4363-a728-9510efe618bb\" (UID: \"84978e46-13ee-4363-a728-9510efe618bb\") "
Sep 15 12:22:19.615180 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:19.615177    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l8jwb\" (UniqueName: \"kubernetes.io/projected/84978e46-13ee-4363-a728-9510efe618bb-kube-api-access-l8jwb\") pod \"84978e46-13ee-4363-a728-9510efe618bb\" (UID: \"84978e46-13ee-4363-a728-9510efe618bb\") "
Sep 15 12:22:19.615432 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:19.615219    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84978e46-13ee-4363-a728-9510efe618bb-model-cache\") pod \"84978e46-13ee-4363-a728-9510efe618bb\" (UID: \"84978e46-13ee-4363-a728-9510efe618bb\") "
Sep 15 12:22:19.615432 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:19.615240    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84978e46-13ee-4363-a728-9510efe618bb-home" (OuterVolumeSpecName: "home") pod "84978e46-13ee-4363-a728-9510efe618bb" (UID: "84978e46-13ee-4363-a728-9510efe618bb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:22:19.615432 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:19.615292    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84978e46-13ee-4363-a728-9510efe618bb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "84978e46-13ee-4363-a728-9510efe618bb" (UID: "84978e46-13ee-4363-a728-9510efe618bb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:22:19.615599 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:19.615582    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84978e46-13ee-4363-a728-9510efe618bb-home\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:22:19.615655 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:19.615609    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84978e46-13ee-4363-a728-9510efe618bb-tmp-dir\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:22:19.615655 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:19.615580    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84978e46-13ee-4363-a728-9510efe618bb-model-cache" (OuterVolumeSpecName: "model-cache") pod "84978e46-13ee-4363-a728-9510efe618bb" (UID: "84978e46-13ee-4363-a728-9510efe618bb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:22:19.617418 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:19.617392    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84978e46-13ee-4363-a728-9510efe618bb-kube-api-access-l8jwb" (OuterVolumeSpecName: "kube-api-access-l8jwb") pod "84978e46-13ee-4363-a728-9510efe618bb" (UID: "84978e46-13ee-4363-a728-9510efe618bb"). InnerVolumeSpecName "kube-api-access-l8jwb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:22:19.617505 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:19.617398    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84978e46-13ee-4363-a728-9510efe618bb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "84978e46-13ee-4363-a728-9510efe618bb" (UID: "84978e46-13ee-4363-a728-9510efe618bb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:22:19.617505 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:19.617402    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84978e46-13ee-4363-a728-9510efe618bb-dshm" (OuterVolumeSpecName: "dshm") pod "84978e46-13ee-4363-a728-9510efe618bb" (UID: "84978e46-13ee-4363-a728-9510efe618bb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:22:19.716686 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:19.716589    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84978e46-13ee-4363-a728-9510efe618bb-dshm\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:22:19.716686 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:19.716622    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84978e46-13ee-4363-a728-9510efe618bb-tls-certs\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:22:19.716686 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:19.716632    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8jwb\" (UniqueName: \"kubernetes.io/projected/84978e46-13ee-4363-a728-9510efe618bb-kube-api-access-l8jwb\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:22:19.716686 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:19.716641    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84978e46-13ee-4363-a728-9510efe618bb-model-cache\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:22:19.818795 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:19.818759    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b04a7844-49ed-48a4-8ef0-7aad3eee1aea" path="/var/lib/kubelet/pods/b04a7844-49ed-48a4-8ef0-7aad3eee1aea/volumes"
Sep 15 12:22:20.138830 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:20.138792    2555 generic.go:356] "Generic (PLEG): container finished" podID="84978e46-13ee-4363-a728-9510efe618bb" containerID="a9579d1eb1f7d5f47264c9e0e1715985afb364f47071a0fb6b65968d5305a1e5" exitCode=0
Sep 15 12:22:20.138830 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:20.138834    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-69d7ba/test-model-69d7ba-kserve-8dc8bcc6c-gft8f"
Sep 15 12:22:20.139090 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:20.138881    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-69d7ba/test-model-69d7ba-kserve-8dc8bcc6c-gft8f" event={"ID":"84978e46-13ee-4363-a728-9510efe618bb","Type":"ContainerDied","Data":"a9579d1eb1f7d5f47264c9e0e1715985afb364f47071a0fb6b65968d5305a1e5"}
Sep 15 12:22:20.139090 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:20.138926    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-69d7ba/test-model-69d7ba-kserve-8dc8bcc6c-gft8f" event={"ID":"84978e46-13ee-4363-a728-9510efe618bb","Type":"ContainerDied","Data":"1ec88fdbea5fd462dec34e5a8a3f3aa95a5682f5d118cc4a10a79507c0c1e853"}
Sep 15 12:22:20.139090 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:20.138946    2555 scope.go:117] "RemoveContainer" containerID="a9579d1eb1f7d5f47264c9e0e1715985afb364f47071a0fb6b65968d5305a1e5"
Sep 15 12:22:20.148073 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:20.148050    2555 scope.go:117] "RemoveContainer" containerID="a9579d1eb1f7d5f47264c9e0e1715985afb364f47071a0fb6b65968d5305a1e5"
Sep 15 12:22:20.148403 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:22:20.148382    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9579d1eb1f7d5f47264c9e0e1715985afb364f47071a0fb6b65968d5305a1e5\": container with ID starting with a9579d1eb1f7d5f47264c9e0e1715985afb364f47071a0fb6b65968d5305a1e5 not found: ID does not exist" containerID="a9579d1eb1f7d5f47264c9e0e1715985afb364f47071a0fb6b65968d5305a1e5"
Sep 15 12:22:20.148462 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:20.148411    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9579d1eb1f7d5f47264c9e0e1715985afb364f47071a0fb6b65968d5305a1e5"} err="failed to get container status \"a9579d1eb1f7d5f47264c9e0e1715985afb364f47071a0fb6b65968d5305a1e5\": rpc error: code = NotFound desc = could not find container \"a9579d1eb1f7d5f47264c9e0e1715985afb364f47071a0fb6b65968d5305a1e5\": container with ID starting with a9579d1eb1f7d5f47264c9e0e1715985afb364f47071a0fb6b65968d5305a1e5 not found: ID does not exist"
Sep 15 12:22:20.159575 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:20.159546    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-69d7ba/test-model-69d7ba-kserve-8dc8bcc6c-gft8f"]
Sep 15 12:22:20.164515 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:20.164492    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-69d7ba/test-model-69d7ba-kserve-8dc8bcc6c-gft8f"]
Sep 15 12:22:20.725278 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:20.725224    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5gkxs"]
Sep 15 12:22:21.817982 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:21.817947    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84978e46-13ee-4363-a728-9510efe618bb" path="/var/lib/kubelet/pods/84978e46-13ee-4363-a728-9510efe618bb/volumes"
Sep 15 12:22:22.234556 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:22.234519    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9ba48f/e2e-conflict-b-a7f06f2e-kserve-7ff6c7b9db-4p67k"
Sep 15 12:22:22.234740 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:22.234582    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-9ba48f/e2e-conflict-b-a7f06f2e-kserve-7ff6c7b9db-4p67k"
Sep 15 12:22:22.248075 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:22.248038    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-9ba48f/e2e-conflict-b-a7f06f2e-kserve-7ff6c7b9db-4p67k"
Sep 15 12:22:23.174182 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:23.174146    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9ba48f/e2e-conflict-b-a7f06f2e-kserve-7ff6c7b9db-4p67k"
Sep 15 12:22:23.947298 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:23.947244    2555 scope.go:117] "RemoveContainer" containerID="2fc84f48edadb897c1c7971821d696f27dbec59923459fcbe8f612a443f29f25"
Sep 15 12:22:25.880323 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:25.880287    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-69d7ba-tphf9"]
Sep 15 12:22:25.880812 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:25.880797    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b04a7844-49ed-48a4-8ef0-7aad3eee1aea" containerName="main"
Sep 15 12:22:25.880859 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:25.880815    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="b04a7844-49ed-48a4-8ef0-7aad3eee1aea" containerName="main"
Sep 15 12:22:25.880859 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:25.880830    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84978e46-13ee-4363-a728-9510efe618bb" containerName="main"
Sep 15 12:22:25.880859 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:25.880839    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="84978e46-13ee-4363-a728-9510efe618bb" containerName="main"
Sep 15 12:22:25.880964 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:25.880953    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="84978e46-13ee-4363-a728-9510efe618bb" containerName="main"
Sep 15 12:22:25.881203 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:25.880971    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="b04a7844-49ed-48a4-8ef0-7aad3eee1aea" containerName="main"
Sep 15 12:22:25.885613 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:25.885591    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-69d7ba-tphf9"
Sep 15 12:22:25.888154 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:25.888131    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tbvkt\""
Sep 15 12:22:25.888317 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:25.888212    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 12:22:25.893845 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:25.893788    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-69d7ba-tphf9"]
Sep 15 12:22:25.977463 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:25.977428    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bae9775-b774-4b57-aa52-dcb8ea9f0fc8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-69d7ba-tphf9\" (UID: \"3bae9775-b774-4b57-aa52-dcb8ea9f0fc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-69d7ba-tphf9"
Sep 15 12:22:26.078385 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:26.078339    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bae9775-b774-4b57-aa52-dcb8ea9f0fc8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-69d7ba-tphf9\" (UID: \"3bae9775-b774-4b57-aa52-dcb8ea9f0fc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-69d7ba-tphf9"
Sep 15 12:22:26.078991 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:26.078970    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bae9775-b774-4b57-aa52-dcb8ea9f0fc8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-69d7ba-tphf9\" (UID: \"3bae9775-b774-4b57-aa52-dcb8ea9f0fc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-69d7ba-tphf9"
Sep 15 12:22:26.198024 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:26.197987    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-69d7ba-tphf9"
Sep 15 12:22:26.480448 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:26.480421    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-69d7ba-tphf9"]
Sep 15 12:22:26.481756 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:22:26.481728    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bae9775_b774_4b57_aa52_dcb8ea9f0fc8.slice/crio-fe46e8ff7e9f1ab842f782f55893925b8dfc984d30ec30cc0fe1a477ace47b57 WatchSource:0}: Error finding container fe46e8ff7e9f1ab842f782f55893925b8dfc984d30ec30cc0fe1a477ace47b57: Status 404 returned error can't find the container with id fe46e8ff7e9f1ab842f782f55893925b8dfc984d30ec30cc0fe1a477ace47b57
Sep 15 12:22:27.187160 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:27.187120    2555 generic.go:356] "Generic (PLEG): container finished" podID="3bae9775-b774-4b57-aa52-dcb8ea9f0fc8" containerID="8471f1a08b78e7718094171000300434b72cc7f4b9c4a2ffb901fbd4c4957d6d" exitCode=0
Sep 15 12:22:27.187578 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:27.187195    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-69d7ba-tphf9" event={"ID":"3bae9775-b774-4b57-aa52-dcb8ea9f0fc8","Type":"ContainerDied","Data":"8471f1a08b78e7718094171000300434b72cc7f4b9c4a2ffb901fbd4c4957d6d"}
Sep 15 12:22:27.187578 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:27.187237    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-69d7ba-tphf9" event={"ID":"3bae9775-b774-4b57-aa52-dcb8ea9f0fc8","Type":"ContainerStarted","Data":"fe46e8ff7e9f1ab842f782f55893925b8dfc984d30ec30cc0fe1a477ace47b57"}
Sep 15 12:22:28.329704 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:28.329678    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-69d7ba-tphf9"
Sep 15 12:22:28.504527 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:28.504435    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bae9775-b774-4b57-aa52-dcb8ea9f0fc8-maas-api-service-ca\") pod \"3bae9775-b774-4b57-aa52-dcb8ea9f0fc8\" (UID: \"3bae9775-b774-4b57-aa52-dcb8ea9f0fc8\") "
Sep 15 12:22:28.504813 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:28.504790    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bae9775-b774-4b57-aa52-dcb8ea9f0fc8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3bae9775-b774-4b57-aa52-dcb8ea9f0fc8" (UID: "3bae9775-b774-4b57-aa52-dcb8ea9f0fc8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:22:28.605691 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:28.605657    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bae9775-b774-4b57-aa52-dcb8ea9f0fc8-maas-api-service-ca\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:22:29.199368 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:29.199340    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-69d7ba-tphf9"
Sep 15 12:22:29.199368 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:29.199350    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-69d7ba-tphf9" event={"ID":"3bae9775-b774-4b57-aa52-dcb8ea9f0fc8","Type":"ContainerDied","Data":"fe46e8ff7e9f1ab842f782f55893925b8dfc984d30ec30cc0fe1a477ace47b57"}
Sep 15 12:22:29.199605 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:29.199404    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe46e8ff7e9f1ab842f782f55893925b8dfc984d30ec30cc0fe1a477ace47b57"
Sep 15 12:22:32.101960 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:32.101929    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-80b72486-c6db495fc-wh478"]
Sep 15 12:22:32.102350 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:32.102307    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bae9775-b774-4b57-aa52-dcb8ea9f0fc8" containerName="revoke-keys"
Sep 15 12:22:32.102350 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:32.102319    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bae9775-b774-4b57-aa52-dcb8ea9f0fc8" containerName="revoke-keys"
Sep 15 12:22:32.102426 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:32.102406    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bae9775-b774-4b57-aa52-dcb8ea9f0fc8" containerName="revoke-keys"
Sep 15 12:22:32.105554 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:32.105534    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-80b72486-c6db495fc-wh478"
Sep 15 12:22:32.108865 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:32.108840    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-80b72486\""
Sep 15 12:22:32.109934 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:32.109898    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-80b72486-dockercfg-kxkh4\""
Sep 15 12:22:32.117472 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:32.117443    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-80b72486-c6db495fc-wh478"]
Sep 15 12:22:32.239769 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:32.239726    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5f405e8-7619-473a-b352-77642b71c32f-plugins-config-volume\") pod \"payload-processing-e2e-mt-80b72486-c6db495fc-wh478\" (UID: \"b5f405e8-7619-473a-b352-77642b71c32f\") " pod="openshift-ingress/payload-processing-e2e-mt-80b72486-c6db495fc-wh478"
Sep 15 12:22:32.239959 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:32.239861    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkwmq\" (UniqueName: \"kubernetes.io/projected/b5f405e8-7619-473a-b352-77642b71c32f-kube-api-access-wkwmq\") pod \"payload-processing-e2e-mt-80b72486-c6db495fc-wh478\" (UID: \"b5f405e8-7619-473a-b352-77642b71c32f\") " pod="openshift-ingress/payload-processing-e2e-mt-80b72486-c6db495fc-wh478"
Sep 15 12:22:32.340572 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:32.340537    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5f405e8-7619-473a-b352-77642b71c32f-plugins-config-volume\") pod \"payload-processing-e2e-mt-80b72486-c6db495fc-wh478\" (UID: \"b5f405e8-7619-473a-b352-77642b71c32f\") " pod="openshift-ingress/payload-processing-e2e-mt-80b72486-c6db495fc-wh478"
Sep 15 12:22:32.340766 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:32.340599    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkwmq\" (UniqueName: \"kubernetes.io/projected/b5f405e8-7619-473a-b352-77642b71c32f-kube-api-access-wkwmq\") pod \"payload-processing-e2e-mt-80b72486-c6db495fc-wh478\" (UID: \"b5f405e8-7619-473a-b352-77642b71c32f\") " pod="openshift-ingress/payload-processing-e2e-mt-80b72486-c6db495fc-wh478"
Sep 15 12:22:32.341249 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:32.341229    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5f405e8-7619-473a-b352-77642b71c32f-plugins-config-volume\") pod \"payload-processing-e2e-mt-80b72486-c6db495fc-wh478\" (UID: \"b5f405e8-7619-473a-b352-77642b71c32f\") " pod="openshift-ingress/payload-processing-e2e-mt-80b72486-c6db495fc-wh478"
Sep 15 12:22:32.349578 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:32.349546    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkwmq\" (UniqueName: \"kubernetes.io/projected/b5f405e8-7619-473a-b352-77642b71c32f-kube-api-access-wkwmq\") pod \"payload-processing-e2e-mt-80b72486-c6db495fc-wh478\" (UID: \"b5f405e8-7619-473a-b352-77642b71c32f\") " pod="openshift-ingress/payload-processing-e2e-mt-80b72486-c6db495fc-wh478"
Sep 15 12:22:32.416931 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:32.416898    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-80b72486-c6db495fc-wh478"
Sep 15 12:22:32.702154 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:32.696582    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-80b72486-c6db495fc-wh478"]
Sep 15 12:22:33.226841 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:33.226804    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-80b72486-c6db495fc-wh478" event={"ID":"b5f405e8-7619-473a-b352-77642b71c32f","Type":"ContainerStarted","Data":"249851ae123e257a6a71d9feb0f2f30a466ae499cd954722fd2f7885033f6a9b"}
Sep 15 12:22:33.227288 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:33.226862    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-80b72486-c6db495fc-wh478" event={"ID":"b5f405e8-7619-473a-b352-77642b71c32f","Type":"ContainerStarted","Data":"a43f942e10b1590c46f2876c66adfcba9496d3a34c394e2644e88c936fb0708e"}
Sep 15 12:22:33.227288 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:33.226955    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-80b72486-c6db495fc-wh478"
Sep 15 12:22:33.256458 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:33.256397    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-80b72486-c6db495fc-wh478" podStartSLOduration=1.256381094 podStartE2EDuration="1.256381094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:22:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:22:33.254507984 +0000 UTC m=+1210.045430530" watchObservedRunningTime="2026-09-15 12:22:33.256381094 +0000 UTC m=+1210.047303605"
Sep 15 12:22:36.841854 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:36.841816    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-69d7ba-c88fbdb46-m92g7"]
Sep 15 12:22:36.842378 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:36.842058    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-69d7ba-c88fbdb46-m92g7" podUID="27308750-531b-4905-9f23-bd738daec2d0" containerName="payload-pre-processing" containerID="cri-o://8e697429fcaf852c8ccc14c4d15f15c7a7df8b5f7e054404313b90bf5e451b92" gracePeriod=30
Sep 15 12:22:37.100659 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:37.100181    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-69d7ba-tphf9"]
Sep 15 12:22:37.108956 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:37.108918    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-69d7ba-tphf9"]
Sep 15 12:22:37.110926 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:37.110897    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-69d7ba-c88fbdb46-m92g7"
Sep 15 12:22:37.251693 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:37.251661    2555 generic.go:356] "Generic (PLEG): container finished" podID="27308750-531b-4905-9f23-bd738daec2d0" containerID="8e697429fcaf852c8ccc14c4d15f15c7a7df8b5f7e054404313b90bf5e451b92" exitCode=0
Sep 15 12:22:37.251870 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:37.251718    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-69d7ba-c88fbdb46-m92g7" event={"ID":"27308750-531b-4905-9f23-bd738daec2d0","Type":"ContainerDied","Data":"8e697429fcaf852c8ccc14c4d15f15c7a7df8b5f7e054404313b90bf5e451b92"}
Sep 15 12:22:37.251870 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:37.251762    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-69d7ba-c88fbdb46-m92g7" event={"ID":"27308750-531b-4905-9f23-bd738daec2d0","Type":"ContainerDied","Data":"4599467381f46217cecc13d225967fd489e2315f6f9332811cd66b6a8d8ddf0e"}
Sep 15 12:22:37.251870 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:37.251786    2555 scope.go:117] "RemoveContainer" containerID="8e697429fcaf852c8ccc14c4d15f15c7a7df8b5f7e054404313b90bf5e451b92"
Sep 15 12:22:37.251870 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:37.251721    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-69d7ba-c88fbdb46-m92g7"
Sep 15 12:22:37.261163 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:37.261148    2555 scope.go:117] "RemoveContainer" containerID="8e697429fcaf852c8ccc14c4d15f15c7a7df8b5f7e054404313b90bf5e451b92"
Sep 15 12:22:37.261466 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:22:37.261445    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e697429fcaf852c8ccc14c4d15f15c7a7df8b5f7e054404313b90bf5e451b92\": container with ID starting with 8e697429fcaf852c8ccc14c4d15f15c7a7df8b5f7e054404313b90bf5e451b92 not found: ID does not exist" containerID="8e697429fcaf852c8ccc14c4d15f15c7a7df8b5f7e054404313b90bf5e451b92"
Sep 15 12:22:37.261551 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:37.261473    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e697429fcaf852c8ccc14c4d15f15c7a7df8b5f7e054404313b90bf5e451b92"} err="failed to get container status \"8e697429fcaf852c8ccc14c4d15f15c7a7df8b5f7e054404313b90bf5e451b92\": rpc error: code = NotFound desc = could not find container \"8e697429fcaf852c8ccc14c4d15f15c7a7df8b5f7e054404313b90bf5e451b92\": container with ID starting with 8e697429fcaf852c8ccc14c4d15f15c7a7df8b5f7e054404313b90bf5e451b92 not found: ID does not exist"
Sep 15 12:22:37.287886 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:37.287853    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s5cgc\" (UniqueName: \"kubernetes.io/projected/27308750-531b-4905-9f23-bd738daec2d0-kube-api-access-s5cgc\") pod \"27308750-531b-4905-9f23-bd738daec2d0\" (UID: \"27308750-531b-4905-9f23-bd738daec2d0\") "
Sep 15 12:22:37.288014 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:37.287962    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27308750-531b-4905-9f23-bd738daec2d0-plugins-config-volume\") pod \"27308750-531b-4905-9f23-bd738daec2d0\" (UID: \"27308750-531b-4905-9f23-bd738daec2d0\") "
Sep 15 12:22:37.288372 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:37.288350    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27308750-531b-4905-9f23-bd738daec2d0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "27308750-531b-4905-9f23-bd738daec2d0" (UID: "27308750-531b-4905-9f23-bd738daec2d0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:22:37.290047 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:37.290018    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27308750-531b-4905-9f23-bd738daec2d0-kube-api-access-s5cgc" (OuterVolumeSpecName: "kube-api-access-s5cgc") pod "27308750-531b-4905-9f23-bd738daec2d0" (UID: "27308750-531b-4905-9f23-bd738daec2d0"). InnerVolumeSpecName "kube-api-access-s5cgc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:22:37.389153 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:37.389123    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5cgc\" (UniqueName: \"kubernetes.io/projected/27308750-531b-4905-9f23-bd738daec2d0-kube-api-access-s5cgc\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:22:37.389153 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:37.389155    2555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27308750-531b-4905-9f23-bd738daec2d0-plugins-config-volume\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:22:37.585601 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:37.585564    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-69d7ba-c88fbdb46-m92g7"]
Sep 15 12:22:37.587969 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:37.587946    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-69d7ba-c88fbdb46-m92g7"]
Sep 15 12:22:37.823548 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:37.823464    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27308750-531b-4905-9f23-bd738daec2d0" path="/var/lib/kubelet/pods/27308750-531b-4905-9f23-bd738daec2d0/volumes"
Sep 15 12:22:37.823806 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:37.823793    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3bae9775-b774-4b57-aa52-dcb8ea9f0fc8" path="/var/lib/kubelet/pods/3bae9775-b774-4b57-aa52-dcb8ea9f0fc8/volumes"
Sep 15 12:22:44.235453 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:44.235419    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-80b72486-c6db495fc-wh478"
Sep 15 12:22:51.750420 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:51.750381    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5gkxs"]
Sep 15 12:22:55.962550 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:55.962509    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-1dc430-54b65bcb7c-l42f8"]
Sep 15 12:22:55.962926 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:55.962890    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27308750-531b-4905-9f23-bd738daec2d0" containerName="payload-pre-processing"
Sep 15 12:22:55.962926 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:55.962902    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="27308750-531b-4905-9f23-bd738daec2d0" containerName="payload-pre-processing"
Sep 15 12:22:55.963017 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:55.962991    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="27308750-531b-4905-9f23-bd738daec2d0" containerName="payload-pre-processing"
Sep 15 12:22:55.966029 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:55.966012    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1dc430-54b65bcb7c-l42f8"
Sep 15 12:22:55.969877 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:55.969852    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-1dc430\""
Sep 15 12:22:55.970026 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:55.969896    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-1dc430-dockercfg-rm9bk\""
Sep 15 12:22:55.978733 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:55.978708    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-1dc430-54b65bcb7c-l42f8"]
Sep 15 12:22:56.067833 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:56.067798    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d529d4a5-2f68-419d-b739-f0e3b2abdd81-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-1dc430-54b65bcb7c-l42f8\" (UID: \"d529d4a5-2f68-419d-b739-f0e3b2abdd81\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1dc430-54b65bcb7c-l42f8"
Sep 15 12:22:56.067833 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:56.067834    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9xfr\" (UniqueName: \"kubernetes.io/projected/d529d4a5-2f68-419d-b739-f0e3b2abdd81-kube-api-access-d9xfr\") pod \"payload-processing-e2e-shared-a-w3-1dc430-54b65bcb7c-l42f8\" (UID: \"d529d4a5-2f68-419d-b739-f0e3b2abdd81\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1dc430-54b65bcb7c-l42f8"
Sep 15 12:22:56.169124 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:56.169082    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d529d4a5-2f68-419d-b739-f0e3b2abdd81-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-1dc430-54b65bcb7c-l42f8\" (UID: \"d529d4a5-2f68-419d-b739-f0e3b2abdd81\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1dc430-54b65bcb7c-l42f8"
Sep 15 12:22:56.169124 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:56.169124    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9xfr\" (UniqueName: \"kubernetes.io/projected/d529d4a5-2f68-419d-b739-f0e3b2abdd81-kube-api-access-d9xfr\") pod \"payload-processing-e2e-shared-a-w3-1dc430-54b65bcb7c-l42f8\" (UID: \"d529d4a5-2f68-419d-b739-f0e3b2abdd81\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1dc430-54b65bcb7c-l42f8"
Sep 15 12:22:56.169826 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:56.169806    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d529d4a5-2f68-419d-b739-f0e3b2abdd81-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-1dc430-54b65bcb7c-l42f8\" (UID: \"d529d4a5-2f68-419d-b739-f0e3b2abdd81\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1dc430-54b65bcb7c-l42f8"
Sep 15 12:22:56.178143 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:56.178114    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9xfr\" (UniqueName: \"kubernetes.io/projected/d529d4a5-2f68-419d-b739-f0e3b2abdd81-kube-api-access-d9xfr\") pod \"payload-processing-e2e-shared-a-w3-1dc430-54b65bcb7c-l42f8\" (UID: \"d529d4a5-2f68-419d-b739-f0e3b2abdd81\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1dc430-54b65bcb7c-l42f8"
Sep 15 12:22:56.277850 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:56.277762    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1dc430-54b65bcb7c-l42f8"
Sep 15 12:22:56.554957 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:56.554933    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-1dc430-54b65bcb7c-l42f8"]
Sep 15 12:22:56.556461 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:22:56.556437    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd529d4a5_2f68_419d_b739_f0e3b2abdd81.slice/crio-ce05a23ec3f54dad1ecbbbfba6e1333f4324a0f5a76da00121d0743790531143 WatchSource:0}: Error finding container ce05a23ec3f54dad1ecbbbfba6e1333f4324a0f5a76da00121d0743790531143: Status 404 returned error can't find the container with id ce05a23ec3f54dad1ecbbbfba6e1333f4324a0f5a76da00121d0743790531143
Sep 15 12:22:57.374988 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:57.374956    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1dc430-54b65bcb7c-l42f8" event={"ID":"d529d4a5-2f68-419d-b739-f0e3b2abdd81","Type":"ContainerStarted","Data":"48601119675edacf519d389b18eb34458cd4cbcccf2b531adf2f9669449529d6"}
Sep 15 12:22:57.375535 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:57.375006    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1dc430-54b65bcb7c-l42f8" event={"ID":"d529d4a5-2f68-419d-b739-f0e3b2abdd81","Type":"ContainerStarted","Data":"ce05a23ec3f54dad1ecbbbfba6e1333f4324a0f5a76da00121d0743790531143"}
Sep 15 12:22:57.375535 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:57.375028    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1dc430-54b65bcb7c-l42f8"
Sep 15 12:22:57.397857 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:22:57.397801    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1dc430-54b65bcb7c-l42f8" podStartSLOduration=2.3977832279999998 podStartE2EDuration="2.397783228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:22:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:22:57.395561713 +0000 UTC m=+1234.186484242" watchObservedRunningTime="2026-09-15 12:22:57.397783228 +0000 UTC m=+1234.188705738"
Sep 15 12:23:08.384675 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:23:08.384643    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1dc430-54b65bcb7c-l42f8"
Sep 15 12:23:15.337773 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:23:15.337735    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-80b72486/e2e-lifecycle-model-80b72486-kserve-65bcc7dd9f-r2sdl"]
Sep 15 12:23:15.341224 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:23:15.341202    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-80b72486/e2e-lifecycle-model-80b72486-kserve-65bcc7dd9f-r2sdl"
Sep 15 12:23:15.345240 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:23:15.345215    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-80b72486\"/\"default-dockercfg-cx25w\""
Sep 15 12:23:15.345419 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:23:15.345313    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-80b72486\"/\"openshift-service-ca.crt\""
Sep 15 12:23:15.345419 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:23:15.345347    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-80b72486\"/\"kube-root-ca.crt\""
Sep 15 12:23:15.345419 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:23:15.345313    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-80b72486\"/\"e2e-lifecycle-model-80b72486-kserve-self-signed-certs\""
Sep 15 12:23:15.355100 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:23:15.355068    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-80b72486/e2e-lifecycle-model-80b72486-kserve-65bcc7dd9f-r2sdl"]
Sep 15 12:23:15.464489 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:23:15.464449    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a5b3e4b-ac7a-4f3d-8577-3d46a72c807e-home\") pod \"e2e-lifecycle-model-80b72486-kserve-65bcc7dd9f-r2sdl\" (UID: \"5a5b3e4b-ac7a-4f3d-8577-3d46a72c807e\") " pod="ai-tenant-e2e-mt-80b72486/e2e-lifecycle-model-80b72486-kserve-65bcc7dd9f-r2sdl"
Sep 15 12:23:15.464489 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:23:15.464487    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a5b3e4b-ac7a-4f3d-8577-3d46a72c807e-dshm\") pod \"e2e-lifecycle-model-80b72486-kserve-65bcc7dd9f-r2sdl\" (UID: \"5a5b3e4b-ac7a-4f3d-8577-3d46a72c807e\") " pod="ai-tenant-e2e-mt-80b72486/e2e-lifecycle-model-80b72486-kserve-65bcc7dd9f-r2sdl"
Sep 15 12:23:15.464728 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:23:15.464567    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a5b3e4b-ac7a-4f3d-8577-3d46a72c807e-tls-certs\") pod \"e2e-lifecycle-model-80b72486-kserve-65bcc7dd9f-r2sdl\" (UID: \"5a5b3e4b-ac7a-4f3d-8577-3d46a72c807e\") " pod="ai-tenant-e2e-mt-80b72486/e2e-lifecycle-model-80b72486-kserve-65bcc7dd9f-r2sdl"
Sep 15 12:23:15.464728 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:23:15.464604    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-894ts\" (UniqueName: \"kubernetes.io/projected/5a5b3e4b-ac7a-4f3d-8577-3d46a72c807e-kube-api-access-894ts\") pod \"e2e-lifecycle-model-80b72486-kserve-65bcc7dd9f-r2sdl\" (UID: \"5a5b3e4b-ac7a-4f3d-8577-3d46a72c807e\") " pod="ai-tenant-e2e-mt-80b72486/e2e-lifecycle-model-80b72486-kserve-65bcc7dd9f-r2sdl"
Sep 15 12:23:15.464800 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:23:15.464737    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a5b3e4b-ac7a-4f3d-8577-3d46a72c807e-tmp-dir\") pod \"e2e-lifecycle-model-80b72486-kserve-65bcc7dd9f-r2sdl\" (UID: \"5a5b3e4b-ac7a-4f3d-8577-3d46a72c807e\") " pod="ai-tenant-e2e-mt-80b72486/e2e-lifecycle-model-80b72486-kserve-65bcc7dd9f-r2sdl"
Sep 15 12:23:15.464800 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:23:15.464778    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a5b3e4b-ac7a-4f3d-8577-3d46a72c807e-model-cache\") pod \"e2e-lifecycle-model-80b72486-kserve-65bcc7dd9f-r2sdl\" (UID: \"5a5b3e4b-ac7a-4f3d-8577-3d46a72c807e\") " pod="ai-tenant-e2e-mt-80b72486/e2e-lifecycle-model-80b72486-kserve-65bcc7dd9f-r2sdl"
Sep 15 12:23:15.565515 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:23:15.565472    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a5b3e4b-ac7a-4f3d-8577-3d46a72c807e-tmp-dir\") pod \"e2e-lifecycle-model-80b72486-kserve-65bcc7dd9f-r2sdl\" (UID: \"5a5b3e4b-ac7a-4f3d-8577-3d46a72c807e\") " pod="ai-tenant-e2e-mt-80b72486/e2e-lifecycle-model-80b72486-kserve-65bcc7dd9f-r2sdl"
Sep 15 12:23:15.565685 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:23:15.565525    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a5b3e4b-ac7a-4f3d-8577-3d46a72c807e-model-cache\") pod \"e2e-lifecycle-model-80b72486-kserve-65bcc7dd9f-r2sdl\" (UID: \"5a5b3e4b-ac7a-4f3d-8577-3d46a72c807e\") " pod="ai-tenant-e2e-mt-80b72486/e2e-lifecycle-model-80b72486-kserve-65bcc7dd9f-r2sdl"
Sep 15 12:23:15.565685 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:23:15.565668    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a5b3e4b-ac7a-4f3d-8577-3d46a72c807e-home\") pod \"e2e-lifecycle-model-80b72486-kserve-65bcc7dd9f-r2sdl\" (UID: \"5a5b3e4b-ac7a-4f3d-8577-3d46a72c807e\") " pod="ai-tenant-e2e-mt-80b72486/e2e-lifecycle-model-80b72486-kserve-65bcc7dd9f-r2sdl"
Sep 15 12:23:15.565786 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:23:15.565701    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a5b3e4b-ac7a-4f3d-8577-3d46a72c807e-dshm\") pod \"e2e-lifecycle-model-80b72486-kserve-65bcc7dd9f-r2sdl\" (UID: \"5a5b3e4b-ac7a-4f3d-8577-3d46a72c807e\") " pod="ai-tenant-e2e-mt-80b72486/e2e-lifecycle-model-80b72486-kserve-65bcc7dd9f-r2sdl"
Sep 15 12:23:15.565786 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:23:15.565754    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a5b3e4b-ac7a-4f3d-8577-3d46a72c807e-tls-certs\") pod \"e2e-lifecycle-model-80b72486-kserve-65bcc7dd9f-r2sdl\" (UID: \"5a5b3e4b-ac7a-4f3d-8577-3d46a72c807e\") " pod="ai-tenant-e2e-mt-80b72486/e2e-lifecycle-model-80b72486-kserve-65bcc7dd9f-r2sdl"
Sep 15 12:23:15.565786 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:23:15.565774    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-894ts\" (UniqueName: \"kubernetes.io/projected/5a5b3e4b-ac7a-4f3d-8577-3d46a72c807e-kube-api-access-894ts\") pod \"e2e-lifecycle-model-80b72486-kserve-65bcc7dd9f-r2sdl\" (UID: \"5a5b3e4b-ac7a-4f3d-8577-3d46a72c807e\") " pod="ai-tenant-e2e-mt-80b72486/e2e-lifecycle-model-80b72486-kserve-65bcc7dd9f-r2sdl"
Sep 15 12:23:15.565981 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:23:15.565933    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a5b3e4b-ac7a-4f3d-8577-3d46a72c807e-model-cache\") pod \"e2e-lifecycle-model-80b72486-kserve-65bcc7dd9f-r2sdl\" (UID: \"5a5b3e4b-ac7a-4f3d-8577-3d46a72c807e\") " pod="ai-tenant-e2e-mt-80b72486/e2e-lifecycle-model-80b72486-kserve-65bcc7dd9f-r2sdl"
Sep 15 12:23:15.566023 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:23:15.566004    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a5b3e4b-ac7a-4f3d-8577-3d46a72c807e-tmp-dir\") pod \"e2e-lifecycle-model-80b72486-kserve-65bcc7dd9f-r2sdl\" (UID: \"5a5b3e4b-ac7a-4f3d-8577-3d46a72c807e\") " pod="ai-tenant-e2e-mt-80b72486/e2e-lifecycle-model-80b72486-kserve-65bcc7dd9f-r2sdl"
Sep 15 12:23:15.566059 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:23:15.566028    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a5b3e4b-ac7a-4f3d-8577-3d46a72c807e-home\") pod \"e2e-lifecycle-model-80b72486-kserve-65bcc7dd9f-r2sdl\" (UID: \"5a5b3e4b-ac7a-4f3d-8577-3d46a72c807e\") " pod="ai-tenant-e2e-mt-80b72486/e2e-lifecycle-model-80b72486-kserve-65bcc7dd9f-r2sdl"
Sep 15 12:23:15.568183 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:23:15.568158    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a5b3e4b-ac7a-4f3d-8577-3d46a72c807e-dshm\") pod \"e2e-lifecycle-model-80b72486-kserve-65bcc7dd9f-r2sdl\" (UID: \"5a5b3e4b-ac7a-4f3d-8577-3d46a72c807e\") " pod="ai-tenant-e2e-mt-80b72486/e2e-lifecycle-model-80b72486-kserve-65bcc7dd9f-r2sdl"
Sep 15 12:23:15.568390 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:23:15.568370    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a5b3e4b-ac7a-4f3d-8577-3d46a72c807e-tls-certs\") pod \"e2e-lifecycle-model-80b72486-kserve-65bcc7dd9f-r2sdl\" (UID: \"5a5b3e4b-ac7a-4f3d-8577-3d46a72c807e\") " pod="ai-tenant-e2e-mt-80b72486/e2e-lifecycle-model-80b72486-kserve-65bcc7dd9f-r2sdl"
Sep 15 12:23:15.574627 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:23:15.574602    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-894ts\" (UniqueName: \"kubernetes.io/projected/5a5b3e4b-ac7a-4f3d-8577-3d46a72c807e-kube-api-access-894ts\") pod \"e2e-lifecycle-model-80b72486-kserve-65bcc7dd9f-r2sdl\" (UID: \"5a5b3e4b-ac7a-4f3d-8577-3d46a72c807e\") " pod="ai-tenant-e2e-mt-80b72486/e2e-lifecycle-model-80b72486-kserve-65bcc7dd9f-r2sdl"
Sep 15 12:23:15.650786 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:23:15.650741    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-80b72486/e2e-lifecycle-model-80b72486-kserve-65bcc7dd9f-r2sdl"
Sep 15 12:23:15.932779 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:23:15.932741    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5a5b3e4b_ac7a_4f3d_8577_3d46a72c807e.slice/crio-f52dd9f9afa079d9435bdf9b0cf8d0d914377dcdf9fd1ee43e7177c35a261fa0 WatchSource:0}: Error finding container f52dd9f9afa079d9435bdf9b0cf8d0d914377dcdf9fd1ee43e7177c35a261fa0: Status 404 returned error can't find the container with id f52dd9f9afa079d9435bdf9b0cf8d0d914377dcdf9fd1ee43e7177c35a261fa0
Sep 15 12:23:15.933356 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:23:15.933332    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-80b72486/e2e-lifecycle-model-80b72486-kserve-65bcc7dd9f-r2sdl"]
Sep 15 12:23:16.496440 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:23:16.496409    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-80b72486/e2e-lifecycle-model-80b72486-kserve-65bcc7dd9f-r2sdl" event={"ID":"5a5b3e4b-ac7a-4f3d-8577-3d46a72c807e","Type":"ContainerStarted","Data":"8f7aa65fd2b572f2dded70388397b0e3d6402f813b04d5f70d5838880f0479bb"}
Sep 15 12:23:16.496440 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:23:16.496448    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-80b72486/e2e-lifecycle-model-80b72486-kserve-65bcc7dd9f-r2sdl" event={"ID":"5a5b3e4b-ac7a-4f3d-8577-3d46a72c807e","Type":"ContainerStarted","Data":"f52dd9f9afa079d9435bdf9b0cf8d0d914377dcdf9fd1ee43e7177c35a261fa0"}
Sep 15 12:23:18.740631 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:23:18.740568    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-80b72486/e2e-lifecycle-model-80b72486-kserve-65bcc7dd9f-r2sdl" podStartSLOduration=3.74054813 podStartE2EDuration="3.74054813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:23:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:23:16.546790689 +0000 UTC m=+1253.337713226" watchObservedRunningTime="2026-09-15 12:23:18.74054813 +0000 UTC m=+1255.531470701"
Sep 15 12:23:18.741759 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:23:18.741738    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-9ba48f/e2e-conflict-b-a7f06f2e-kserve-7ff6c7b9db-4p67k"]
Sep 15 12:23:23.985957 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:23:23.985924    2555 scope.go:117] "RemoveContainer" containerID="cd5ab9bd0eea0fadf42514092c095488dc54c17cad87bd59eed5f267e575fb6d"
Sep 15 12:23:25.650908 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:23:25.650865    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-80b72486/e2e-lifecycle-model-80b72486-kserve-65bcc7dd9f-r2sdl"
Sep 15 12:23:25.651333 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:23:25.650953    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-80b72486/e2e-lifecycle-model-80b72486-kserve-65bcc7dd9f-r2sdl"
Sep 15 12:23:25.663057 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:23:25.663026    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-80b72486/e2e-lifecycle-model-80b72486-kserve-65bcc7dd9f-r2sdl"
Sep 15 12:23:26.566125 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:23:26.566091    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-80b72486/e2e-lifecycle-model-80b72486-kserve-65bcc7dd9f-r2sdl"
Sep 15 12:23:31.997356 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:23:31.997311    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5gkxs"]
Sep 15 12:23:33.761475 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:23:33.761411    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-9ba48f/e2e-conflict-b-a7f06f2e-kserve-7ff6c7b9db-4p67k" podUID="a4c30438-42bc-4d52-be9a-0af5a15eff23" containerName="main" containerID="cri-o://04859da4770b91800ba1940a2c431323db09d4daf95154b38567b3cd25f6a0ed" gracePeriod=45
Sep 15 12:23:34.007594 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:23:34.007564    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9ba48f/e2e-conflict-b-a7f06f2e-kserve-7ff6c7b9db-4p67k"
Sep 15 12:23:34.046161 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:23:34.046076    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s6c26\" (UniqueName: \"kubernetes.io/projected/a4c30438-42bc-4d52-be9a-0af5a15eff23-kube-api-access-s6c26\") pod \"a4c30438-42bc-4d52-be9a-0af5a15eff23\" (UID: \"a4c30438-42bc-4d52-be9a-0af5a15eff23\") "
Sep 15 12:23:34.046161 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:23:34.046118    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4c30438-42bc-4d52-be9a-0af5a15eff23-model-cache\") pod \"a4c30438-42bc-4d52-be9a-0af5a15eff23\" (UID: \"a4c30438-42bc-4d52-be9a-0af5a15eff23\") "
Sep 15 12:23:34.046161 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:23:34.046162    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4c30438-42bc-4d52-be9a-0af5a15eff23-tls-certs\") pod \"a4c30438-42bc-4d52-be9a-0af5a15eff23\" (UID: \"a4c30438-42bc-4d52-be9a-0af5a15eff23\") "
Sep 15 12:23:34.046480 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:23:34.046187    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4c30438-42bc-4d52-be9a-0af5a15eff23-dshm\") pod \"a4c30438-42bc-4d52-be9a-0af5a15eff23\" (UID: \"a4c30438-42bc-4d52-be9a-0af5a15eff23\") "
Sep 15 12:23:34.046480 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:23:34.046276    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4c30438-42bc-4d52-be9a-0af5a15eff23-tmp-dir\") pod \"a4c30438-42bc-4d52-be9a-0af5a15eff23\" (UID: \"a4c30438-42bc-4d52-be9a-0af5a15eff23\") "
Sep 15 12:23:34.046480 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:23:34.046363    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4c30438-42bc-4d52-be9a-0af5a15eff23-home\") pod \"a4c30438-42bc-4d52-be9a-0af5a15eff23\" (UID: \"a4c30438-42bc-4d52-be9a-0af5a15eff23\") "
Sep 15 12:23:34.046661 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:23:34.046488    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4c30438-42bc-4d52-be9a-0af5a15eff23-model-cache" (OuterVolumeSpecName: "model-cache") pod "a4c30438-42bc-4d52-be9a-0af5a15eff23" (UID: "a4c30438-42bc-4d52-be9a-0af5a15eff23"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:23:34.046661 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:23:34.046601    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4c30438-42bc-4d52-be9a-0af5a15eff23-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a4c30438-42bc-4d52-be9a-0af5a15eff23" (UID: "a4c30438-42bc-4d52-be9a-0af5a15eff23"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:23:34.046768 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:23:34.046673    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4c30438-42bc-4d52-be9a-0af5a15eff23-home" (OuterVolumeSpecName: "home") pod "a4c30438-42bc-4d52-be9a-0af5a15eff23" (UID: "a4c30438-42bc-4d52-be9a-0af5a15eff23"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:23:34.046768 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:23:34.046751    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4c30438-42bc-4d52-be9a-0af5a15eff23-tmp-dir\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:23:34.046768 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:23:34.046768    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4c30438-42bc-4d52-be9a-0af5a15eff23-home\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:23:34.046932 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:23:34.046777    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4c30438-42bc-4d52-be9a-0af5a15eff23-model-cache\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:23:34.048561 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:23:34.048521    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4c30438-42bc-4d52-be9a-0af5a15eff23-kube-api-access-s6c26" (OuterVolumeSpecName: "kube-api-access-s6c26") pod "a4c30438-42bc-4d52-be9a-0af5a15eff23" (UID: "a4c30438-42bc-4d52-be9a-0af5a15eff23"). InnerVolumeSpecName "kube-api-access-s6c26". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:23:34.048740 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:23:34.048718    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4c30438-42bc-4d52-be9a-0af5a15eff23-dshm" (OuterVolumeSpecName: "dshm") pod "a4c30438-42bc-4d52-be9a-0af5a15eff23" (UID: "a4c30438-42bc-4d52-be9a-0af5a15eff23"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:23:34.049387 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:23:34.049361    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4c30438-42bc-4d52-be9a-0af5a15eff23-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a4c30438-42bc-4d52-be9a-0af5a15eff23" (UID: "a4c30438-42bc-4d52-be9a-0af5a15eff23"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:23:34.147708 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:23:34.147674    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6c26\" (UniqueName: \"kubernetes.io/projected/a4c30438-42bc-4d52-be9a-0af5a15eff23-kube-api-access-s6c26\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:23:34.147708 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:23:34.147703    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4c30438-42bc-4d52-be9a-0af5a15eff23-tls-certs\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:23:34.147708 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:23:34.147715    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4c30438-42bc-4d52-be9a-0af5a15eff23-dshm\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:23:34.606418 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:23:34.606387    2555 generic.go:356] "Generic (PLEG): container finished" podID="a4c30438-42bc-4d52-be9a-0af5a15eff23" containerID="04859da4770b91800ba1940a2c431323db09d4daf95154b38567b3cd25f6a0ed" exitCode=0
Sep 15 12:23:34.606590 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:23:34.606427    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9ba48f/e2e-conflict-b-a7f06f2e-kserve-7ff6c7b9db-4p67k"
Sep 15 12:23:34.606590 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:23:34.606432    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9ba48f/e2e-conflict-b-a7f06f2e-kserve-7ff6c7b9db-4p67k" event={"ID":"a4c30438-42bc-4d52-be9a-0af5a15eff23","Type":"ContainerDied","Data":"04859da4770b91800ba1940a2c431323db09d4daf95154b38567b3cd25f6a0ed"}
Sep 15 12:23:34.606590 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:23:34.606528    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9ba48f/e2e-conflict-b-a7f06f2e-kserve-7ff6c7b9db-4p67k" event={"ID":"a4c30438-42bc-4d52-be9a-0af5a15eff23","Type":"ContainerDied","Data":"f79031a144a47bb2a2d77559e2f5d65b335857cfc930fc3f4826f727244d081c"}
Sep 15 12:23:34.606590 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:23:34.606546    2555 scope.go:117] "RemoveContainer" containerID="04859da4770b91800ba1940a2c431323db09d4daf95154b38567b3cd25f6a0ed"
Sep 15 12:23:34.615939 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:23:34.615917    2555 scope.go:117] "RemoveContainer" containerID="04859da4770b91800ba1940a2c431323db09d4daf95154b38567b3cd25f6a0ed"
Sep 15 12:23:34.616225 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:23:34.616205    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04859da4770b91800ba1940a2c431323db09d4daf95154b38567b3cd25f6a0ed\": container with ID starting with 04859da4770b91800ba1940a2c431323db09d4daf95154b38567b3cd25f6a0ed not found: ID does not exist" containerID="04859da4770b91800ba1940a2c431323db09d4daf95154b38567b3cd25f6a0ed"
Sep 15 12:23:34.616311 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:23:34.616239    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04859da4770b91800ba1940a2c431323db09d4daf95154b38567b3cd25f6a0ed"} err="failed to get container status \"04859da4770b91800ba1940a2c431323db09d4daf95154b38567b3cd25f6a0ed\": rpc error: code = NotFound desc = could not find container \"04859da4770b91800ba1940a2c431323db09d4daf95154b38567b3cd25f6a0ed\": container with ID starting with 04859da4770b91800ba1940a2c431323db09d4daf95154b38567b3cd25f6a0ed not found: ID does not exist"
Sep 15 12:23:34.652661 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:23:34.652621    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-9ba48f/e2e-conflict-b-a7f06f2e-kserve-7ff6c7b9db-4p67k"]
Sep 15 12:23:34.656794 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:23:34.656768    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-9ba48f/e2e-conflict-b-a7f06f2e-kserve-7ff6c7b9db-4p67k"]
Sep 15 12:23:35.817640 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:23:35.817604    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4c30438-42bc-4d52-be9a-0af5a15eff23" path="/var/lib/kubelet/pods/a4c30438-42bc-4d52-be9a-0af5a15eff23/volumes"
Sep 15 12:24:06.889687 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:06.889652    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5gkxs"]
Sep 15 12:24:14.133735 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:14.133698    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-9ba48f/distinct-w4-9ba48f-kserve-cf9cd67f-7hsfr"]
Sep 15 12:24:14.134146 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:14.134093    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4c30438-42bc-4d52-be9a-0af5a15eff23" containerName="main"
Sep 15 12:24:14.134146 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:14.134105    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4c30438-42bc-4d52-be9a-0af5a15eff23" containerName="main"
Sep 15 12:24:14.134224 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:14.134185    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4c30438-42bc-4d52-be9a-0af5a15eff23" containerName="main"
Sep 15 12:24:14.137672 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:14.137650    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9ba48f/distinct-w4-9ba48f-kserve-cf9cd67f-7hsfr"
Sep 15 12:24:14.140798 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:14.140775    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-9ba48f\"/\"distinct-w4-9ba48f-kserve-self-signed-certs\""
Sep 15 12:24:14.162286 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:14.162216    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-9ba48f/distinct-w4-9ba48f-kserve-cf9cd67f-7hsfr"]
Sep 15 12:24:14.258571 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:14.258527    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fff9c4da-fbed-47c9-8f8a-b254c4489af3-tls-certs\") pod \"distinct-w4-9ba48f-kserve-cf9cd67f-7hsfr\" (UID: \"fff9c4da-fbed-47c9-8f8a-b254c4489af3\") " pod="e2e-models-e2e-worker-w4-9ba48f/distinct-w4-9ba48f-kserve-cf9cd67f-7hsfr"
Sep 15 12:24:14.258738 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:14.258601    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fff9c4da-fbed-47c9-8f8a-b254c4489af3-home\") pod \"distinct-w4-9ba48f-kserve-cf9cd67f-7hsfr\" (UID: \"fff9c4da-fbed-47c9-8f8a-b254c4489af3\") " pod="e2e-models-e2e-worker-w4-9ba48f/distinct-w4-9ba48f-kserve-cf9cd67f-7hsfr"
Sep 15 12:24:14.258738 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:14.258634    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8t2c\" (UniqueName: \"kubernetes.io/projected/fff9c4da-fbed-47c9-8f8a-b254c4489af3-kube-api-access-d8t2c\") pod \"distinct-w4-9ba48f-kserve-cf9cd67f-7hsfr\" (UID: \"fff9c4da-fbed-47c9-8f8a-b254c4489af3\") " pod="e2e-models-e2e-worker-w4-9ba48f/distinct-w4-9ba48f-kserve-cf9cd67f-7hsfr"
Sep 15 12:24:14.258738 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:14.258687    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fff9c4da-fbed-47c9-8f8a-b254c4489af3-tmp-dir\") pod \"distinct-w4-9ba48f-kserve-cf9cd67f-7hsfr\" (UID: \"fff9c4da-fbed-47c9-8f8a-b254c4489af3\") " pod="e2e-models-e2e-worker-w4-9ba48f/distinct-w4-9ba48f-kserve-cf9cd67f-7hsfr"
Sep 15 12:24:14.258738 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:14.258725    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fff9c4da-fbed-47c9-8f8a-b254c4489af3-model-cache\") pod \"distinct-w4-9ba48f-kserve-cf9cd67f-7hsfr\" (UID: \"fff9c4da-fbed-47c9-8f8a-b254c4489af3\") " pod="e2e-models-e2e-worker-w4-9ba48f/distinct-w4-9ba48f-kserve-cf9cd67f-7hsfr"
Sep 15 12:24:14.258952 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:14.258748    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fff9c4da-fbed-47c9-8f8a-b254c4489af3-dshm\") pod \"distinct-w4-9ba48f-kserve-cf9cd67f-7hsfr\" (UID: \"fff9c4da-fbed-47c9-8f8a-b254c4489af3\") " pod="e2e-models-e2e-worker-w4-9ba48f/distinct-w4-9ba48f-kserve-cf9cd67f-7hsfr"
Sep 15 12:24:14.359737 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:14.359696    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fff9c4da-fbed-47c9-8f8a-b254c4489af3-home\") pod \"distinct-w4-9ba48f-kserve-cf9cd67f-7hsfr\" (UID: \"fff9c4da-fbed-47c9-8f8a-b254c4489af3\") " pod="e2e-models-e2e-worker-w4-9ba48f/distinct-w4-9ba48f-kserve-cf9cd67f-7hsfr"
Sep 15 12:24:14.359737 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:14.359743    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8t2c\" (UniqueName: \"kubernetes.io/projected/fff9c4da-fbed-47c9-8f8a-b254c4489af3-kube-api-access-d8t2c\") pod \"distinct-w4-9ba48f-kserve-cf9cd67f-7hsfr\" (UID: \"fff9c4da-fbed-47c9-8f8a-b254c4489af3\") " pod="e2e-models-e2e-worker-w4-9ba48f/distinct-w4-9ba48f-kserve-cf9cd67f-7hsfr"
Sep 15 12:24:14.360010 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:14.359767    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fff9c4da-fbed-47c9-8f8a-b254c4489af3-tmp-dir\") pod \"distinct-w4-9ba48f-kserve-cf9cd67f-7hsfr\" (UID: \"fff9c4da-fbed-47c9-8f8a-b254c4489af3\") " pod="e2e-models-e2e-worker-w4-9ba48f/distinct-w4-9ba48f-kserve-cf9cd67f-7hsfr"
Sep 15 12:24:14.360010 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:14.359799    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fff9c4da-fbed-47c9-8f8a-b254c4489af3-model-cache\") pod \"distinct-w4-9ba48f-kserve-cf9cd67f-7hsfr\" (UID: \"fff9c4da-fbed-47c9-8f8a-b254c4489af3\") " pod="e2e-models-e2e-worker-w4-9ba48f/distinct-w4-9ba48f-kserve-cf9cd67f-7hsfr"
Sep 15 12:24:14.360010 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:14.359825    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fff9c4da-fbed-47c9-8f8a-b254c4489af3-dshm\") pod \"distinct-w4-9ba48f-kserve-cf9cd67f-7hsfr\" (UID: \"fff9c4da-fbed-47c9-8f8a-b254c4489af3\") " pod="e2e-models-e2e-worker-w4-9ba48f/distinct-w4-9ba48f-kserve-cf9cd67f-7hsfr"
Sep 15 12:24:14.360010 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:14.359873    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fff9c4da-fbed-47c9-8f8a-b254c4489af3-tls-certs\") pod \"distinct-w4-9ba48f-kserve-cf9cd67f-7hsfr\" (UID: \"fff9c4da-fbed-47c9-8f8a-b254c4489af3\") " pod="e2e-models-e2e-worker-w4-9ba48f/distinct-w4-9ba48f-kserve-cf9cd67f-7hsfr"
Sep 15 12:24:14.360206 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:14.360180    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fff9c4da-fbed-47c9-8f8a-b254c4489af3-tmp-dir\") pod \"distinct-w4-9ba48f-kserve-cf9cd67f-7hsfr\" (UID: \"fff9c4da-fbed-47c9-8f8a-b254c4489af3\") " pod="e2e-models-e2e-worker-w4-9ba48f/distinct-w4-9ba48f-kserve-cf9cd67f-7hsfr"
Sep 15 12:24:14.360206 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:14.360197    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fff9c4da-fbed-47c9-8f8a-b254c4489af3-home\") pod \"distinct-w4-9ba48f-kserve-cf9cd67f-7hsfr\" (UID: \"fff9c4da-fbed-47c9-8f8a-b254c4489af3\") " pod="e2e-models-e2e-worker-w4-9ba48f/distinct-w4-9ba48f-kserve-cf9cd67f-7hsfr"
Sep 15 12:24:14.360317 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:14.360247    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fff9c4da-fbed-47c9-8f8a-b254c4489af3-model-cache\") pod \"distinct-w4-9ba48f-kserve-cf9cd67f-7hsfr\" (UID: \"fff9c4da-fbed-47c9-8f8a-b254c4489af3\") " pod="e2e-models-e2e-worker-w4-9ba48f/distinct-w4-9ba48f-kserve-cf9cd67f-7hsfr"
Sep 15 12:24:14.362145 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:14.362126    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fff9c4da-fbed-47c9-8f8a-b254c4489af3-dshm\") pod \"distinct-w4-9ba48f-kserve-cf9cd67f-7hsfr\" (UID: \"fff9c4da-fbed-47c9-8f8a-b254c4489af3\") " pod="e2e-models-e2e-worker-w4-9ba48f/distinct-w4-9ba48f-kserve-cf9cd67f-7hsfr"
Sep 15 12:24:14.362550 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:14.362531    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fff9c4da-fbed-47c9-8f8a-b254c4489af3-tls-certs\") pod \"distinct-w4-9ba48f-kserve-cf9cd67f-7hsfr\" (UID: \"fff9c4da-fbed-47c9-8f8a-b254c4489af3\") " pod="e2e-models-e2e-worker-w4-9ba48f/distinct-w4-9ba48f-kserve-cf9cd67f-7hsfr"
Sep 15 12:24:14.378631 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:14.378600    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8t2c\" (UniqueName: \"kubernetes.io/projected/fff9c4da-fbed-47c9-8f8a-b254c4489af3-kube-api-access-d8t2c\") pod \"distinct-w4-9ba48f-kserve-cf9cd67f-7hsfr\" (UID: \"fff9c4da-fbed-47c9-8f8a-b254c4489af3\") " pod="e2e-models-e2e-worker-w4-9ba48f/distinct-w4-9ba48f-kserve-cf9cd67f-7hsfr"
Sep 15 12:24:14.447326 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:14.447212    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9ba48f/distinct-w4-9ba48f-kserve-cf9cd67f-7hsfr"
Sep 15 12:24:14.724102 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:14.724075    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-9ba48f/distinct-w4-9ba48f-kserve-cf9cd67f-7hsfr"]
Sep 15 12:24:14.725542 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:24:14.725516    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfff9c4da_fbed_47c9_8f8a_b254c4489af3.slice/crio-04b3a554cef1fb6cdd6393b0b6e282e75719876d6d9130a44be8c0c1d2c5d486 WatchSource:0}: Error finding container 04b3a554cef1fb6cdd6393b0b6e282e75719876d6d9130a44be8c0c1d2c5d486: Status 404 returned error can't find the container with id 04b3a554cef1fb6cdd6393b0b6e282e75719876d6d9130a44be8c0c1d2c5d486
Sep 15 12:24:14.843216 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:14.843181    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9ba48f/distinct-w4-9ba48f-kserve-cf9cd67f-7hsfr" event={"ID":"fff9c4da-fbed-47c9-8f8a-b254c4489af3","Type":"ContainerStarted","Data":"f1adb2977a7274f3da7731a46115b2fe0741033c3d5ac7aafddc0fd6279b3edb"}
Sep 15 12:24:14.843428 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:14.843227    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9ba48f/distinct-w4-9ba48f-kserve-cf9cd67f-7hsfr" event={"ID":"fff9c4da-fbed-47c9-8f8a-b254c4489af3","Type":"ContainerStarted","Data":"04b3a554cef1fb6cdd6393b0b6e282e75719876d6d9130a44be8c0c1d2c5d486"}
Sep 15 12:24:14.873149 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:14.873088    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-9ba48f/distinct-w4-9ba48f-kserve-cf9cd67f-7hsfr" podStartSLOduration=0.873067922 podStartE2EDuration="873.067922ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:24:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:24:14.871185255 +0000 UTC m=+1311.662107787" watchObservedRunningTime="2026-09-15 12:24:14.873067922 +0000 UTC m=+1311.663990428"
Sep 15 12:24:21.835619 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:21.835585    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-1dc430/auth-test-model-1dc430-kserve-77cd669cc5-wc628"]
Sep 15 12:24:21.838837 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:21.838820    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1dc430/auth-test-model-1dc430-kserve-77cd669cc5-wc628"
Sep 15 12:24:21.852472 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:21.852442    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-1dc430\"/\"kube-root-ca.crt\""
Sep 15 12:24:21.852627 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:21.852478    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-1dc430\"/\"default-dockercfg-6dht6\""
Sep 15 12:24:21.852627 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:21.852504    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-1dc430\"/\"auth-test-model-1dc430-kserve-self-signed-certs\""
Sep 15 12:24:21.852627 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:21.852516    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-1dc430\"/\"openshift-service-ca.crt\""
Sep 15 12:24:21.862336 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:21.862299    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1dc430/auth-test-model-1dc430-kserve-77cd669cc5-wc628"]
Sep 15 12:24:21.934783 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:21.934729    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/099303bf-d5c6-4c12-bc2c-1f148d4fa19f-dshm\") pod \"auth-test-model-1dc430-kserve-77cd669cc5-wc628\" (UID: \"099303bf-d5c6-4c12-bc2c-1f148d4fa19f\") " pod="ai-tenant-e2e-shared-a-w3-1dc430/auth-test-model-1dc430-kserve-77cd669cc5-wc628"
Sep 15 12:24:21.934996 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:21.934814    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/099303bf-d5c6-4c12-bc2c-1f148d4fa19f-home\") pod \"auth-test-model-1dc430-kserve-77cd669cc5-wc628\" (UID: \"099303bf-d5c6-4c12-bc2c-1f148d4fa19f\") " pod="ai-tenant-e2e-shared-a-w3-1dc430/auth-test-model-1dc430-kserve-77cd669cc5-wc628"
Sep 15 12:24:21.934996 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:21.934853    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/099303bf-d5c6-4c12-bc2c-1f148d4fa19f-model-cache\") pod \"auth-test-model-1dc430-kserve-77cd669cc5-wc628\" (UID: \"099303bf-d5c6-4c12-bc2c-1f148d4fa19f\") " pod="ai-tenant-e2e-shared-a-w3-1dc430/auth-test-model-1dc430-kserve-77cd669cc5-wc628"
Sep 15 12:24:21.934996 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:21.934895    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8pzx\" (UniqueName: \"kubernetes.io/projected/099303bf-d5c6-4c12-bc2c-1f148d4fa19f-kube-api-access-l8pzx\") pod \"auth-test-model-1dc430-kserve-77cd669cc5-wc628\" (UID: \"099303bf-d5c6-4c12-bc2c-1f148d4fa19f\") " pod="ai-tenant-e2e-shared-a-w3-1dc430/auth-test-model-1dc430-kserve-77cd669cc5-wc628"
Sep 15 12:24:21.934996 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:21.934916    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/099303bf-d5c6-4c12-bc2c-1f148d4fa19f-tls-certs\") pod \"auth-test-model-1dc430-kserve-77cd669cc5-wc628\" (UID: \"099303bf-d5c6-4c12-bc2c-1f148d4fa19f\") " pod="ai-tenant-e2e-shared-a-w3-1dc430/auth-test-model-1dc430-kserve-77cd669cc5-wc628"
Sep 15 12:24:21.934996 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:21.934970    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/099303bf-d5c6-4c12-bc2c-1f148d4fa19f-tmp-dir\") pod \"auth-test-model-1dc430-kserve-77cd669cc5-wc628\" (UID: \"099303bf-d5c6-4c12-bc2c-1f148d4fa19f\") " pod="ai-tenant-e2e-shared-a-w3-1dc430/auth-test-model-1dc430-kserve-77cd669cc5-wc628"
Sep 15 12:24:22.036251 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:22.036155    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/099303bf-d5c6-4c12-bc2c-1f148d4fa19f-dshm\") pod \"auth-test-model-1dc430-kserve-77cd669cc5-wc628\" (UID: \"099303bf-d5c6-4c12-bc2c-1f148d4fa19f\") " pod="ai-tenant-e2e-shared-a-w3-1dc430/auth-test-model-1dc430-kserve-77cd669cc5-wc628"
Sep 15 12:24:22.036458 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:22.036308    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/099303bf-d5c6-4c12-bc2c-1f148d4fa19f-home\") pod \"auth-test-model-1dc430-kserve-77cd669cc5-wc628\" (UID: \"099303bf-d5c6-4c12-bc2c-1f148d4fa19f\") " pod="ai-tenant-e2e-shared-a-w3-1dc430/auth-test-model-1dc430-kserve-77cd669cc5-wc628"
Sep 15 12:24:22.036458 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:22.036353    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/099303bf-d5c6-4c12-bc2c-1f148d4fa19f-model-cache\") pod \"auth-test-model-1dc430-kserve-77cd669cc5-wc628\" (UID: \"099303bf-d5c6-4c12-bc2c-1f148d4fa19f\") " pod="ai-tenant-e2e-shared-a-w3-1dc430/auth-test-model-1dc430-kserve-77cd669cc5-wc628"
Sep 15 12:24:22.036458 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:22.036398    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8pzx\" (UniqueName: \"kubernetes.io/projected/099303bf-d5c6-4c12-bc2c-1f148d4fa19f-kube-api-access-l8pzx\") pod \"auth-test-model-1dc430-kserve-77cd669cc5-wc628\" (UID: \"099303bf-d5c6-4c12-bc2c-1f148d4fa19f\") " pod="ai-tenant-e2e-shared-a-w3-1dc430/auth-test-model-1dc430-kserve-77cd669cc5-wc628"
Sep 15 12:24:22.036458 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:22.036421    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/099303bf-d5c6-4c12-bc2c-1f148d4fa19f-tls-certs\") pod \"auth-test-model-1dc430-kserve-77cd669cc5-wc628\" (UID: \"099303bf-d5c6-4c12-bc2c-1f148d4fa19f\") " pod="ai-tenant-e2e-shared-a-w3-1dc430/auth-test-model-1dc430-kserve-77cd669cc5-wc628"
Sep 15 12:24:22.036458 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:22.036454    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/099303bf-d5c6-4c12-bc2c-1f148d4fa19f-tmp-dir\") pod \"auth-test-model-1dc430-kserve-77cd669cc5-wc628\" (UID: \"099303bf-d5c6-4c12-bc2c-1f148d4fa19f\") " pod="ai-tenant-e2e-shared-a-w3-1dc430/auth-test-model-1dc430-kserve-77cd669cc5-wc628"
Sep 15 12:24:22.036770 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:22.036740    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/099303bf-d5c6-4c12-bc2c-1f148d4fa19f-home\") pod \"auth-test-model-1dc430-kserve-77cd669cc5-wc628\" (UID: \"099303bf-d5c6-4c12-bc2c-1f148d4fa19f\") " pod="ai-tenant-e2e-shared-a-w3-1dc430/auth-test-model-1dc430-kserve-77cd669cc5-wc628"
Sep 15 12:24:22.036835 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:22.036795    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/099303bf-d5c6-4c12-bc2c-1f148d4fa19f-model-cache\") pod \"auth-test-model-1dc430-kserve-77cd669cc5-wc628\" (UID: \"099303bf-d5c6-4c12-bc2c-1f148d4fa19f\") " pod="ai-tenant-e2e-shared-a-w3-1dc430/auth-test-model-1dc430-kserve-77cd669cc5-wc628"
Sep 15 12:24:22.036884 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:22.036831    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/099303bf-d5c6-4c12-bc2c-1f148d4fa19f-tmp-dir\") pod \"auth-test-model-1dc430-kserve-77cd669cc5-wc628\" (UID: \"099303bf-d5c6-4c12-bc2c-1f148d4fa19f\") " pod="ai-tenant-e2e-shared-a-w3-1dc430/auth-test-model-1dc430-kserve-77cd669cc5-wc628"
Sep 15 12:24:22.038927 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:22.038902    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/099303bf-d5c6-4c12-bc2c-1f148d4fa19f-tls-certs\") pod \"auth-test-model-1dc430-kserve-77cd669cc5-wc628\" (UID: \"099303bf-d5c6-4c12-bc2c-1f148d4fa19f\") " pod="ai-tenant-e2e-shared-a-w3-1dc430/auth-test-model-1dc430-kserve-77cd669cc5-wc628"
Sep 15 12:24:22.039080 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:22.039056    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/099303bf-d5c6-4c12-bc2c-1f148d4fa19f-dshm\") pod \"auth-test-model-1dc430-kserve-77cd669cc5-wc628\" (UID: \"099303bf-d5c6-4c12-bc2c-1f148d4fa19f\") " pod="ai-tenant-e2e-shared-a-w3-1dc430/auth-test-model-1dc430-kserve-77cd669cc5-wc628"
Sep 15 12:24:22.046979 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:22.046927    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8pzx\" (UniqueName: \"kubernetes.io/projected/099303bf-d5c6-4c12-bc2c-1f148d4fa19f-kube-api-access-l8pzx\") pod \"auth-test-model-1dc430-kserve-77cd669cc5-wc628\" (UID: \"099303bf-d5c6-4c12-bc2c-1f148d4fa19f\") " pod="ai-tenant-e2e-shared-a-w3-1dc430/auth-test-model-1dc430-kserve-77cd669cc5-wc628"
Sep 15 12:24:22.151616 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:22.151334    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1dc430/auth-test-model-1dc430-kserve-77cd669cc5-wc628"
Sep 15 12:24:22.443940 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:22.443904    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1dc430/auth-test-model-1dc430-kserve-77cd669cc5-wc628"]
Sep 15 12:24:22.444090 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:24:22.443950    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod099303bf_d5c6_4c12_bc2c_1f148d4fa19f.slice/crio-94ae2a49dbee18bb125c59da9edcf8c2fe019238e7e12c7c1af46689ddb91e94 WatchSource:0}: Error finding container 94ae2a49dbee18bb125c59da9edcf8c2fe019238e7e12c7c1af46689ddb91e94: Status 404 returned error can't find the container with id 94ae2a49dbee18bb125c59da9edcf8c2fe019238e7e12c7c1af46689ddb91e94
Sep 15 12:24:22.897728 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:22.897690    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1dc430/auth-test-model-1dc430-kserve-77cd669cc5-wc628" event={"ID":"099303bf-d5c6-4c12-bc2c-1f148d4fa19f","Type":"ContainerStarted","Data":"6fcd1853ed7e1b02bb97eb2c3ef34c644e4b32f2ed2b6038e40f271607dcce11"}
Sep 15 12:24:22.897728 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:22.897734    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1dc430/auth-test-model-1dc430-kserve-77cd669cc5-wc628" event={"ID":"099303bf-d5c6-4c12-bc2c-1f148d4fa19f","Type":"ContainerStarted","Data":"94ae2a49dbee18bb125c59da9edcf8c2fe019238e7e12c7c1af46689ddb91e94"}
Sep 15 12:24:22.924961 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:22.924904    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-1dc430/auth-test-model-1dc430-kserve-77cd669cc5-wc628" podStartSLOduration=1.9248873949999998 podStartE2EDuration="1.924887395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:24:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:24:22.922434882 +0000 UTC m=+1319.713357412" watchObservedRunningTime="2026-09-15 12:24:22.924887395 +0000 UTC m=+1319.715809950"
Sep 15 12:24:24.015250 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:24.014231    2555 scope.go:117] "RemoveContainer" containerID="df2c9b99621377ebca9ee46cddf8ae474025c168d5523aff4811cd7e58462c5e"
Sep 15 12:24:24.024410 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:24.024387    2555 scope.go:117] "RemoveContainer" containerID="1b3a7f13e100104113ea1e090b5ec973c9422c65d8a8026e6ee250f11db2a294"
Sep 15 12:24:24.448456 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:24.448414    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-9ba48f/distinct-w4-9ba48f-kserve-cf9cd67f-7hsfr"
Sep 15 12:24:24.448651 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:24.448481    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9ba48f/distinct-w4-9ba48f-kserve-cf9cd67f-7hsfr"
Sep 15 12:24:24.461556 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:24.461528    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-9ba48f/distinct-w4-9ba48f-kserve-cf9cd67f-7hsfr"
Sep 15 12:24:24.920227 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:24.920195    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9ba48f/distinct-w4-9ba48f-kserve-cf9cd67f-7hsfr"
Sep 15 12:24:27.930416 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:27.930378    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-80b72486-c6db495fc-wh478"]
Sep 15 12:24:27.930875 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:27.930617    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-80b72486-c6db495fc-wh478" podUID="b5f405e8-7619-473a-b352-77642b71c32f" containerName="payload-processing" containerID="cri-o://249851ae123e257a6a71d9feb0f2f30a466ae499cd954722fd2f7885033f6a9b" gracePeriod=30
Sep 15 12:24:28.196276 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:28.196204    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-80b72486-c6db495fc-wh478"
Sep 15 12:24:28.294906 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:28.294867    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wkwmq\" (UniqueName: \"kubernetes.io/projected/b5f405e8-7619-473a-b352-77642b71c32f-kube-api-access-wkwmq\") pod \"b5f405e8-7619-473a-b352-77642b71c32f\" (UID: \"b5f405e8-7619-473a-b352-77642b71c32f\") "
Sep 15 12:24:28.295107 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:28.294927    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5f405e8-7619-473a-b352-77642b71c32f-plugins-config-volume\") pod \"b5f405e8-7619-473a-b352-77642b71c32f\" (UID: \"b5f405e8-7619-473a-b352-77642b71c32f\") "
Sep 15 12:24:28.295354 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:28.295325    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5f405e8-7619-473a-b352-77642b71c32f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b5f405e8-7619-473a-b352-77642b71c32f" (UID: "b5f405e8-7619-473a-b352-77642b71c32f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:24:28.297025 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:28.296999    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5f405e8-7619-473a-b352-77642b71c32f-kube-api-access-wkwmq" (OuterVolumeSpecName: "kube-api-access-wkwmq") pod "b5f405e8-7619-473a-b352-77642b71c32f" (UID: "b5f405e8-7619-473a-b352-77642b71c32f"). InnerVolumeSpecName "kube-api-access-wkwmq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:24:28.396198 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:28.396156    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkwmq\" (UniqueName: \"kubernetes.io/projected/b5f405e8-7619-473a-b352-77642b71c32f-kube-api-access-wkwmq\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:24:28.396198 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:28.396187    2555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5f405e8-7619-473a-b352-77642b71c32f-plugins-config-volume\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:24:28.934310 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:28.934250    2555 generic.go:356] "Generic (PLEG): container finished" podID="b5f405e8-7619-473a-b352-77642b71c32f" containerID="249851ae123e257a6a71d9feb0f2f30a466ae499cd954722fd2f7885033f6a9b" exitCode=0
Sep 15 12:24:28.934310 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:28.934309    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-80b72486-c6db495fc-wh478"
Sep 15 12:24:28.934819 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:28.934341    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-80b72486-c6db495fc-wh478" event={"ID":"b5f405e8-7619-473a-b352-77642b71c32f","Type":"ContainerDied","Data":"249851ae123e257a6a71d9feb0f2f30a466ae499cd954722fd2f7885033f6a9b"}
Sep 15 12:24:28.934819 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:28.934392    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-80b72486-c6db495fc-wh478" event={"ID":"b5f405e8-7619-473a-b352-77642b71c32f","Type":"ContainerDied","Data":"a43f942e10b1590c46f2876c66adfcba9496d3a34c394e2644e88c936fb0708e"}
Sep 15 12:24:28.934819 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:28.934413    2555 scope.go:117] "RemoveContainer" containerID="249851ae123e257a6a71d9feb0f2f30a466ae499cd954722fd2f7885033f6a9b"
Sep 15 12:24:28.943966 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:28.943949    2555 scope.go:117] "RemoveContainer" containerID="249851ae123e257a6a71d9feb0f2f30a466ae499cd954722fd2f7885033f6a9b"
Sep 15 12:24:28.944291 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:24:28.944252    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"249851ae123e257a6a71d9feb0f2f30a466ae499cd954722fd2f7885033f6a9b\": container with ID starting with 249851ae123e257a6a71d9feb0f2f30a466ae499cd954722fd2f7885033f6a9b not found: ID does not exist" containerID="249851ae123e257a6a71d9feb0f2f30a466ae499cd954722fd2f7885033f6a9b"
Sep 15 12:24:28.944355 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:28.944295    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"249851ae123e257a6a71d9feb0f2f30a466ae499cd954722fd2f7885033f6a9b"} err="failed to get container status \"249851ae123e257a6a71d9feb0f2f30a466ae499cd954722fd2f7885033f6a9b\": rpc error: code = NotFound desc = could not find container \"249851ae123e257a6a71d9feb0f2f30a466ae499cd954722fd2f7885033f6a9b\": container with ID starting with 249851ae123e257a6a71d9feb0f2f30a466ae499cd954722fd2f7885033f6a9b not found: ID does not exist"
Sep 15 12:24:28.971822 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:28.971761    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-80b72486-c6db495fc-wh478"]
Sep 15 12:24:28.974162 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:28.974128    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-80b72486-c6db495fc-wh478"]
Sep 15 12:24:29.818082 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:29.818047    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5f405e8-7619-473a-b352-77642b71c32f" path="/var/lib/kubelet/pods/b5f405e8-7619-473a-b352-77642b71c32f/volumes"
Sep 15 12:24:32.152979 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:32.152941    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-1dc430/auth-test-model-1dc430-kserve-77cd669cc5-wc628"
Sep 15 12:24:32.153501 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:32.152998    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-1dc430/auth-test-model-1dc430-kserve-77cd669cc5-wc628"
Sep 15 12:24:32.165597 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:32.165561    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-1dc430/auth-test-model-1dc430-kserve-77cd669cc5-wc628"
Sep 15 12:24:32.971741 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:32.971711    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-1dc430/auth-test-model-1dc430-kserve-77cd669cc5-wc628"
Sep 15 12:24:36.242825 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:36.242788    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-80b72486/e2e-lifecycle-model-80b72486-kserve-65bcc7dd9f-r2sdl"]
Sep 15 12:24:40.989888 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:40.989849    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5gkxs"]
Sep 15 12:24:51.262597 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:51.262509    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-80b72486/e2e-lifecycle-model-80b72486-kserve-65bcc7dd9f-r2sdl" podUID="5a5b3e4b-ac7a-4f3d-8577-3d46a72c807e" containerName="main" containerID="cri-o://8f7aa65fd2b572f2dded70388397b0e3d6402f813b04d5f70d5838880f0479bb" gracePeriod=45
Sep 15 12:24:51.514112 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:51.514043    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-80b72486/e2e-lifecycle-model-80b72486-kserve-65bcc7dd9f-r2sdl"
Sep 15 12:24:51.628660 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:51.628625    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a5b3e4b-ac7a-4f3d-8577-3d46a72c807e-dshm\") pod \"5a5b3e4b-ac7a-4f3d-8577-3d46a72c807e\" (UID: \"5a5b3e4b-ac7a-4f3d-8577-3d46a72c807e\") "
Sep 15 12:24:51.628927 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:51.628685    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-894ts\" (UniqueName: \"kubernetes.io/projected/5a5b3e4b-ac7a-4f3d-8577-3d46a72c807e-kube-api-access-894ts\") pod \"5a5b3e4b-ac7a-4f3d-8577-3d46a72c807e\" (UID: \"5a5b3e4b-ac7a-4f3d-8577-3d46a72c807e\") "
Sep 15 12:24:51.628927 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:51.628770    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a5b3e4b-ac7a-4f3d-8577-3d46a72c807e-tls-certs\") pod \"5a5b3e4b-ac7a-4f3d-8577-3d46a72c807e\" (UID: \"5a5b3e4b-ac7a-4f3d-8577-3d46a72c807e\") "
Sep 15 12:24:51.628927 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:51.628790    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a5b3e4b-ac7a-4f3d-8577-3d46a72c807e-tmp-dir\") pod \"5a5b3e4b-ac7a-4f3d-8577-3d46a72c807e\" (UID: \"5a5b3e4b-ac7a-4f3d-8577-3d46a72c807e\") "
Sep 15 12:24:51.628927 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:51.628825    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a5b3e4b-ac7a-4f3d-8577-3d46a72c807e-home\") pod \"5a5b3e4b-ac7a-4f3d-8577-3d46a72c807e\" (UID: \"5a5b3e4b-ac7a-4f3d-8577-3d46a72c807e\") "
Sep 15 12:24:51.628927 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:51.628855    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a5b3e4b-ac7a-4f3d-8577-3d46a72c807e-model-cache\") pod \"5a5b3e4b-ac7a-4f3d-8577-3d46a72c807e\" (UID: \"5a5b3e4b-ac7a-4f3d-8577-3d46a72c807e\") "
Sep 15 12:24:51.629231 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:51.629094    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a5b3e4b-ac7a-4f3d-8577-3d46a72c807e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5a5b3e4b-ac7a-4f3d-8577-3d46a72c807e" (UID: "5a5b3e4b-ac7a-4f3d-8577-3d46a72c807e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:24:51.629231 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:51.629152    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a5b3e4b-ac7a-4f3d-8577-3d46a72c807e-home" (OuterVolumeSpecName: "home") pod "5a5b3e4b-ac7a-4f3d-8577-3d46a72c807e" (UID: "5a5b3e4b-ac7a-4f3d-8577-3d46a72c807e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:24:51.629231 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:51.629181    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a5b3e4b-ac7a-4f3d-8577-3d46a72c807e-model-cache" (OuterVolumeSpecName: "model-cache") pod "5a5b3e4b-ac7a-4f3d-8577-3d46a72c807e" (UID: "5a5b3e4b-ac7a-4f3d-8577-3d46a72c807e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:24:51.630957 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:51.630932    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a5b3e4b-ac7a-4f3d-8577-3d46a72c807e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5a5b3e4b-ac7a-4f3d-8577-3d46a72c807e" (UID: "5a5b3e4b-ac7a-4f3d-8577-3d46a72c807e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:24:51.631061 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:51.630984    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a5b3e4b-ac7a-4f3d-8577-3d46a72c807e-dshm" (OuterVolumeSpecName: "dshm") pod "5a5b3e4b-ac7a-4f3d-8577-3d46a72c807e" (UID: "5a5b3e4b-ac7a-4f3d-8577-3d46a72c807e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:24:51.631061 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:51.630995    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a5b3e4b-ac7a-4f3d-8577-3d46a72c807e-kube-api-access-894ts" (OuterVolumeSpecName: "kube-api-access-894ts") pod "5a5b3e4b-ac7a-4f3d-8577-3d46a72c807e" (UID: "5a5b3e4b-ac7a-4f3d-8577-3d46a72c807e"). InnerVolumeSpecName "kube-api-access-894ts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:24:51.730565 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:51.730520    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a5b3e4b-ac7a-4f3d-8577-3d46a72c807e-tls-certs\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:24:51.730757 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:51.730586    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a5b3e4b-ac7a-4f3d-8577-3d46a72c807e-tmp-dir\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:24:51.730757 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:51.730599    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a5b3e4b-ac7a-4f3d-8577-3d46a72c807e-home\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:24:51.730757 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:51.730611    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a5b3e4b-ac7a-4f3d-8577-3d46a72c807e-model-cache\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:24:51.730757 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:51.730621    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a5b3e4b-ac7a-4f3d-8577-3d46a72c807e-dshm\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:24:51.730757 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:51.730632    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-894ts\" (UniqueName: \"kubernetes.io/projected/5a5b3e4b-ac7a-4f3d-8577-3d46a72c807e-kube-api-access-894ts\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:24:52.080886 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:52.080784    2555 generic.go:356] "Generic (PLEG): container finished" podID="5a5b3e4b-ac7a-4f3d-8577-3d46a72c807e" containerID="8f7aa65fd2b572f2dded70388397b0e3d6402f813b04d5f70d5838880f0479bb" exitCode=0
Sep 15 12:24:52.080886 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:52.080828    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-80b72486/e2e-lifecycle-model-80b72486-kserve-65bcc7dd9f-r2sdl" event={"ID":"5a5b3e4b-ac7a-4f3d-8577-3d46a72c807e","Type":"ContainerDied","Data":"8f7aa65fd2b572f2dded70388397b0e3d6402f813b04d5f70d5838880f0479bb"}
Sep 15 12:24:52.080886 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:52.080842    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-80b72486/e2e-lifecycle-model-80b72486-kserve-65bcc7dd9f-r2sdl"
Sep 15 12:24:52.080886 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:52.080887    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-80b72486/e2e-lifecycle-model-80b72486-kserve-65bcc7dd9f-r2sdl" event={"ID":"5a5b3e4b-ac7a-4f3d-8577-3d46a72c807e","Type":"ContainerDied","Data":"f52dd9f9afa079d9435bdf9b0cf8d0d914377dcdf9fd1ee43e7177c35a261fa0"}
Sep 15 12:24:52.081211 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:52.080912    2555 scope.go:117] "RemoveContainer" containerID="8f7aa65fd2b572f2dded70388397b0e3d6402f813b04d5f70d5838880f0479bb"
Sep 15 12:24:52.090100 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:52.090077    2555 scope.go:117] "RemoveContainer" containerID="8f7aa65fd2b572f2dded70388397b0e3d6402f813b04d5f70d5838880f0479bb"
Sep 15 12:24:52.090437 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:24:52.090414    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f7aa65fd2b572f2dded70388397b0e3d6402f813b04d5f70d5838880f0479bb\": container with ID starting with 8f7aa65fd2b572f2dded70388397b0e3d6402f813b04d5f70d5838880f0479bb not found: ID does not exist" containerID="8f7aa65fd2b572f2dded70388397b0e3d6402f813b04d5f70d5838880f0479bb"
Sep 15 12:24:52.090514 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:52.090448    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f7aa65fd2b572f2dded70388397b0e3d6402f813b04d5f70d5838880f0479bb"} err="failed to get container status \"8f7aa65fd2b572f2dded70388397b0e3d6402f813b04d5f70d5838880f0479bb\": rpc error: code = NotFound desc = could not find container \"8f7aa65fd2b572f2dded70388397b0e3d6402f813b04d5f70d5838880f0479bb\": container with ID starting with 8f7aa65fd2b572f2dded70388397b0e3d6402f813b04d5f70d5838880f0479bb not found: ID does not exist"
Sep 15 12:24:52.102827 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:52.102790    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-80b72486/e2e-lifecycle-model-80b72486-kserve-65bcc7dd9f-r2sdl"]
Sep 15 12:24:52.109780 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:52.109751    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-80b72486/e2e-lifecycle-model-80b72486-kserve-65bcc7dd9f-r2sdl"]
Sep 15 12:24:53.817769 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:53.817731    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a5b3e4b-ac7a-4f3d-8577-3d46a72c807e" path="/var/lib/kubelet/pods/5a5b3e4b-ac7a-4f3d-8577-3d46a72c807e/volumes"
Sep 15 12:24:57.513730 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:57.513690    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-9ba48f/distinct-2-w4-9ba48f-kserve-7fd5987948-s54l2"]
Sep 15 12:24:57.514140 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:57.514122    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a5b3e4b-ac7a-4f3d-8577-3d46a72c807e" containerName="main"
Sep 15 12:24:57.514187 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:57.514143    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a5b3e4b-ac7a-4f3d-8577-3d46a72c807e" containerName="main"
Sep 15 12:24:57.514187 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:57.514173    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5f405e8-7619-473a-b352-77642b71c32f" containerName="payload-processing"
Sep 15 12:24:57.514187 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:57.514179    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5f405e8-7619-473a-b352-77642b71c32f" containerName="payload-processing"
Sep 15 12:24:57.514302 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:57.514256    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5f405e8-7619-473a-b352-77642b71c32f" containerName="payload-processing"
Sep 15 12:24:57.514302 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:57.514277    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a5b3e4b-ac7a-4f3d-8577-3d46a72c807e" containerName="main"
Sep 15 12:24:57.517615 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:57.517595    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9ba48f/distinct-2-w4-9ba48f-kserve-7fd5987948-s54l2"
Sep 15 12:24:57.521337 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:57.521317    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-9ba48f\"/\"distinct-2-w4-9ba48f-kserve-self-signed-certs\""
Sep 15 12:24:57.531172 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:57.531149    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-9ba48f/distinct-2-w4-9ba48f-kserve-7fd5987948-s54l2"]
Sep 15 12:24:57.691063 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:57.691016    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49d7abed-95ea-4dc2-b066-2ea0ddfdff52-model-cache\") pod \"distinct-2-w4-9ba48f-kserve-7fd5987948-s54l2\" (UID: \"49d7abed-95ea-4dc2-b066-2ea0ddfdff52\") " pod="e2e-models-e2e-worker-w4-9ba48f/distinct-2-w4-9ba48f-kserve-7fd5987948-s54l2"
Sep 15 12:24:57.691244 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:57.691110    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49d7abed-95ea-4dc2-b066-2ea0ddfdff52-home\") pod \"distinct-2-w4-9ba48f-kserve-7fd5987948-s54l2\" (UID: \"49d7abed-95ea-4dc2-b066-2ea0ddfdff52\") " pod="e2e-models-e2e-worker-w4-9ba48f/distinct-2-w4-9ba48f-kserve-7fd5987948-s54l2"
Sep 15 12:24:57.691244 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:57.691138    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49d7abed-95ea-4dc2-b066-2ea0ddfdff52-dshm\") pod \"distinct-2-w4-9ba48f-kserve-7fd5987948-s54l2\" (UID: \"49d7abed-95ea-4dc2-b066-2ea0ddfdff52\") " pod="e2e-models-e2e-worker-w4-9ba48f/distinct-2-w4-9ba48f-kserve-7fd5987948-s54l2"
Sep 15 12:24:57.691244 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:57.691191    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49d7abed-95ea-4dc2-b066-2ea0ddfdff52-tmp-dir\") pod \"distinct-2-w4-9ba48f-kserve-7fd5987948-s54l2\" (UID: \"49d7abed-95ea-4dc2-b066-2ea0ddfdff52\") " pod="e2e-models-e2e-worker-w4-9ba48f/distinct-2-w4-9ba48f-kserve-7fd5987948-s54l2"
Sep 15 12:24:57.691244 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:57.691220    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49d7abed-95ea-4dc2-b066-2ea0ddfdff52-tls-certs\") pod \"distinct-2-w4-9ba48f-kserve-7fd5987948-s54l2\" (UID: \"49d7abed-95ea-4dc2-b066-2ea0ddfdff52\") " pod="e2e-models-e2e-worker-w4-9ba48f/distinct-2-w4-9ba48f-kserve-7fd5987948-s54l2"
Sep 15 12:24:57.691244 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:57.691237    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h97t4\" (UniqueName: \"kubernetes.io/projected/49d7abed-95ea-4dc2-b066-2ea0ddfdff52-kube-api-access-h97t4\") pod \"distinct-2-w4-9ba48f-kserve-7fd5987948-s54l2\" (UID: \"49d7abed-95ea-4dc2-b066-2ea0ddfdff52\") " pod="e2e-models-e2e-worker-w4-9ba48f/distinct-2-w4-9ba48f-kserve-7fd5987948-s54l2"
Sep 15 12:24:57.792563 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:57.792457    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49d7abed-95ea-4dc2-b066-2ea0ddfdff52-model-cache\") pod \"distinct-2-w4-9ba48f-kserve-7fd5987948-s54l2\" (UID: \"49d7abed-95ea-4dc2-b066-2ea0ddfdff52\") " pod="e2e-models-e2e-worker-w4-9ba48f/distinct-2-w4-9ba48f-kserve-7fd5987948-s54l2"
Sep 15 12:24:57.792563 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:57.792552    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49d7abed-95ea-4dc2-b066-2ea0ddfdff52-home\") pod \"distinct-2-w4-9ba48f-kserve-7fd5987948-s54l2\" (UID: \"49d7abed-95ea-4dc2-b066-2ea0ddfdff52\") " pod="e2e-models-e2e-worker-w4-9ba48f/distinct-2-w4-9ba48f-kserve-7fd5987948-s54l2"
Sep 15 12:24:57.792563 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:57.792569    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49d7abed-95ea-4dc2-b066-2ea0ddfdff52-dshm\") pod \"distinct-2-w4-9ba48f-kserve-7fd5987948-s54l2\" (UID: \"49d7abed-95ea-4dc2-b066-2ea0ddfdff52\") " pod="e2e-models-e2e-worker-w4-9ba48f/distinct-2-w4-9ba48f-kserve-7fd5987948-s54l2"
Sep 15 12:24:57.792849 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:57.792599    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49d7abed-95ea-4dc2-b066-2ea0ddfdff52-tmp-dir\") pod \"distinct-2-w4-9ba48f-kserve-7fd5987948-s54l2\" (UID: \"49d7abed-95ea-4dc2-b066-2ea0ddfdff52\") " pod="e2e-models-e2e-worker-w4-9ba48f/distinct-2-w4-9ba48f-kserve-7fd5987948-s54l2"
Sep 15 12:24:57.792849 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:57.792625    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49d7abed-95ea-4dc2-b066-2ea0ddfdff52-tls-certs\") pod \"distinct-2-w4-9ba48f-kserve-7fd5987948-s54l2\" (UID: \"49d7abed-95ea-4dc2-b066-2ea0ddfdff52\") " pod="e2e-models-e2e-worker-w4-9ba48f/distinct-2-w4-9ba48f-kserve-7fd5987948-s54l2"
Sep 15 12:24:57.792849 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:57.792644    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h97t4\" (UniqueName: \"kubernetes.io/projected/49d7abed-95ea-4dc2-b066-2ea0ddfdff52-kube-api-access-h97t4\") pod \"distinct-2-w4-9ba48f-kserve-7fd5987948-s54l2\" (UID: \"49d7abed-95ea-4dc2-b066-2ea0ddfdff52\") " pod="e2e-models-e2e-worker-w4-9ba48f/distinct-2-w4-9ba48f-kserve-7fd5987948-s54l2"
Sep 15 12:24:57.793023 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:57.792944    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49d7abed-95ea-4dc2-b066-2ea0ddfdff52-model-cache\") pod \"distinct-2-w4-9ba48f-kserve-7fd5987948-s54l2\" (UID: \"49d7abed-95ea-4dc2-b066-2ea0ddfdff52\") " pod="e2e-models-e2e-worker-w4-9ba48f/distinct-2-w4-9ba48f-kserve-7fd5987948-s54l2"
Sep 15 12:24:57.793023 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:57.792970    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49d7abed-95ea-4dc2-b066-2ea0ddfdff52-home\") pod \"distinct-2-w4-9ba48f-kserve-7fd5987948-s54l2\" (UID: \"49d7abed-95ea-4dc2-b066-2ea0ddfdff52\") " pod="e2e-models-e2e-worker-w4-9ba48f/distinct-2-w4-9ba48f-kserve-7fd5987948-s54l2"
Sep 15 12:24:57.793127 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:57.793070    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49d7abed-95ea-4dc2-b066-2ea0ddfdff52-tmp-dir\") pod \"distinct-2-w4-9ba48f-kserve-7fd5987948-s54l2\" (UID: \"49d7abed-95ea-4dc2-b066-2ea0ddfdff52\") " pod="e2e-models-e2e-worker-w4-9ba48f/distinct-2-w4-9ba48f-kserve-7fd5987948-s54l2"
Sep 15 12:24:57.794936 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:57.794915    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49d7abed-95ea-4dc2-b066-2ea0ddfdff52-dshm\") pod \"distinct-2-w4-9ba48f-kserve-7fd5987948-s54l2\" (UID: \"49d7abed-95ea-4dc2-b066-2ea0ddfdff52\") " pod="e2e-models-e2e-worker-w4-9ba48f/distinct-2-w4-9ba48f-kserve-7fd5987948-s54l2"
Sep 15 12:24:57.795107 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:57.795087    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49d7abed-95ea-4dc2-b066-2ea0ddfdff52-tls-certs\") pod \"distinct-2-w4-9ba48f-kserve-7fd5987948-s54l2\" (UID: \"49d7abed-95ea-4dc2-b066-2ea0ddfdff52\") " pod="e2e-models-e2e-worker-w4-9ba48f/distinct-2-w4-9ba48f-kserve-7fd5987948-s54l2"
Sep 15 12:24:57.801793 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:57.801766    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h97t4\" (UniqueName: \"kubernetes.io/projected/49d7abed-95ea-4dc2-b066-2ea0ddfdff52-kube-api-access-h97t4\") pod \"distinct-2-w4-9ba48f-kserve-7fd5987948-s54l2\" (UID: \"49d7abed-95ea-4dc2-b066-2ea0ddfdff52\") " pod="e2e-models-e2e-worker-w4-9ba48f/distinct-2-w4-9ba48f-kserve-7fd5987948-s54l2"
Sep 15 12:24:57.826257 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:57.826223    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9ba48f/distinct-2-w4-9ba48f-kserve-7fd5987948-s54l2"
Sep 15 12:24:58.102147 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:58.102116    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-9ba48f/distinct-2-w4-9ba48f-kserve-7fd5987948-s54l2"]
Sep 15 12:24:58.104097 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:24:58.104066    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod49d7abed_95ea_4dc2_b066_2ea0ddfdff52.slice/crio-b0d394bb6f8701aa731c950e8bbbfcfca349c5de5a39a25c512aaa21cbd9c98e WatchSource:0}: Error finding container b0d394bb6f8701aa731c950e8bbbfcfca349c5de5a39a25c512aaa21cbd9c98e: Status 404 returned error can't find the container with id b0d394bb6f8701aa731c950e8bbbfcfca349c5de5a39a25c512aaa21cbd9c98e
Sep 15 12:24:58.117169 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:58.117141    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9ba48f/distinct-2-w4-9ba48f-kserve-7fd5987948-s54l2" event={"ID":"49d7abed-95ea-4dc2-b066-2ea0ddfdff52","Type":"ContainerStarted","Data":"b0d394bb6f8701aa731c950e8bbbfcfca349c5de5a39a25c512aaa21cbd9c98e"}
Sep 15 12:24:59.124788 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:24:59.124743    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9ba48f/distinct-2-w4-9ba48f-kserve-7fd5987948-s54l2" event={"ID":"49d7abed-95ea-4dc2-b066-2ea0ddfdff52","Type":"ContainerStarted","Data":"4027c5e63a56bf22e8f49c36d1337fbd5d3182116180c0261a7e9fd46fbe86c8"}
Sep 15 12:25:05.985199 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:25:05.985140    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-9ba48f/distinct-2-w4-9ba48f-kserve-7fd5987948-s54l2" podStartSLOduration=8.985118628 podStartE2EDuration="8.985118628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:24:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:24:59.153648352 +0000 UTC m=+1355.944570875" watchObservedRunningTime="2026-09-15 12:25:05.985118628 +0000 UTC m=+1362.776041140"
Sep 15 12:25:05.987290 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:25:05.987252    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5gkxs"]
Sep 15 12:25:07.826661 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:25:07.826622    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9ba48f/distinct-2-w4-9ba48f-kserve-7fd5987948-s54l2"
Sep 15 12:25:07.827086 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:25:07.826687    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-9ba48f/distinct-2-w4-9ba48f-kserve-7fd5987948-s54l2"
Sep 15 12:25:07.839520 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:25:07.839490    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-9ba48f/distinct-2-w4-9ba48f-kserve-7fd5987948-s54l2"
Sep 15 12:25:08.200369 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:25:08.200338    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9ba48f/distinct-2-w4-9ba48f-kserve-7fd5987948-s54l2"
Sep 15 12:25:14.696906 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:25:14.696866    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-680229/embedding-w0-680229-kserve-5ff6df78d7-tbscw"]
Sep 15 12:25:14.700618 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:25:14.700600    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-680229/embedding-w0-680229-kserve-5ff6df78d7-tbscw"
Sep 15 12:25:14.704995 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:25:14.704969    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-680229\"/\"embedding-w0-680229-kserve-self-signed-certs\""
Sep 15 12:25:14.710618 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:25:14.710594    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-680229/embedding-w0-680229-kserve-5ff6df78d7-tbscw"]
Sep 15 12:25:14.758724 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:25:14.758688    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b03ebf3-5e70-423a-8373-fa6af05da2d8-tmp-dir\") pod \"embedding-w0-680229-kserve-5ff6df78d7-tbscw\" (UID: \"4b03ebf3-5e70-423a-8373-fa6af05da2d8\") " pod="e2e-models-e2e-worker-w0-680229/embedding-w0-680229-kserve-5ff6df78d7-tbscw"
Sep 15 12:25:14.758935 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:25:14.758735    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b03ebf3-5e70-423a-8373-fa6af05da2d8-tls-certs\") pod \"embedding-w0-680229-kserve-5ff6df78d7-tbscw\" (UID: \"4b03ebf3-5e70-423a-8373-fa6af05da2d8\") " pod="e2e-models-e2e-worker-w0-680229/embedding-w0-680229-kserve-5ff6df78d7-tbscw"
Sep 15 12:25:14.758935 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:25:14.758796    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b03ebf3-5e70-423a-8373-fa6af05da2d8-model-cache\") pod \"embedding-w0-680229-kserve-5ff6df78d7-tbscw\" (UID: \"4b03ebf3-5e70-423a-8373-fa6af05da2d8\") " pod="e2e-models-e2e-worker-w0-680229/embedding-w0-680229-kserve-5ff6df78d7-tbscw"
Sep 15 12:25:14.758935 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:25:14.758822    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wdvd\" (UniqueName: \"kubernetes.io/projected/4b03ebf3-5e70-423a-8373-fa6af05da2d8-kube-api-access-6wdvd\") pod \"embedding-w0-680229-kserve-5ff6df78d7-tbscw\" (UID: \"4b03ebf3-5e70-423a-8373-fa6af05da2d8\") " pod="e2e-models-e2e-worker-w0-680229/embedding-w0-680229-kserve-5ff6df78d7-tbscw"
Sep 15 12:25:14.758935 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:25:14.758855    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b03ebf3-5e70-423a-8373-fa6af05da2d8-home\") pod \"embedding-w0-680229-kserve-5ff6df78d7-tbscw\" (UID: \"4b03ebf3-5e70-423a-8373-fa6af05da2d8\") " pod="e2e-models-e2e-worker-w0-680229/embedding-w0-680229-kserve-5ff6df78d7-tbscw"
Sep 15 12:25:14.758935 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:25:14.758889    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b03ebf3-5e70-423a-8373-fa6af05da2d8-dshm\") pod \"embedding-w0-680229-kserve-5ff6df78d7-tbscw\" (UID: \"4b03ebf3-5e70-423a-8373-fa6af05da2d8\") " pod="e2e-models-e2e-worker-w0-680229/embedding-w0-680229-kserve-5ff6df78d7-tbscw"
Sep 15 12:25:14.859985 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:25:14.859942    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b03ebf3-5e70-423a-8373-fa6af05da2d8-tmp-dir\") pod \"embedding-w0-680229-kserve-5ff6df78d7-tbscw\" (UID: \"4b03ebf3-5e70-423a-8373-fa6af05da2d8\") " pod="e2e-models-e2e-worker-w0-680229/embedding-w0-680229-kserve-5ff6df78d7-tbscw"
Sep 15 12:25:14.859985 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:25:14.859985    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b03ebf3-5e70-423a-8373-fa6af05da2d8-tls-certs\") pod \"embedding-w0-680229-kserve-5ff6df78d7-tbscw\" (UID: \"4b03ebf3-5e70-423a-8373-fa6af05da2d8\") " pod="e2e-models-e2e-worker-w0-680229/embedding-w0-680229-kserve-5ff6df78d7-tbscw"
Sep 15 12:25:14.860248 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:25:14.860007    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b03ebf3-5e70-423a-8373-fa6af05da2d8-model-cache\") pod \"embedding-w0-680229-kserve-5ff6df78d7-tbscw\" (UID: \"4b03ebf3-5e70-423a-8373-fa6af05da2d8\") " pod="e2e-models-e2e-worker-w0-680229/embedding-w0-680229-kserve-5ff6df78d7-tbscw"
Sep 15 12:25:14.860248 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:25:14.860024    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wdvd\" (UniqueName: \"kubernetes.io/projected/4b03ebf3-5e70-423a-8373-fa6af05da2d8-kube-api-access-6wdvd\") pod \"embedding-w0-680229-kserve-5ff6df78d7-tbscw\" (UID: \"4b03ebf3-5e70-423a-8373-fa6af05da2d8\") " pod="e2e-models-e2e-worker-w0-680229/embedding-w0-680229-kserve-5ff6df78d7-tbscw"
Sep 15 12:25:14.860248 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:25:14.860046    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b03ebf3-5e70-423a-8373-fa6af05da2d8-home\") pod \"embedding-w0-680229-kserve-5ff6df78d7-tbscw\" (UID: \"4b03ebf3-5e70-423a-8373-fa6af05da2d8\") " pod="e2e-models-e2e-worker-w0-680229/embedding-w0-680229-kserve-5ff6df78d7-tbscw"
Sep 15 12:25:14.860248 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:25:14.860069    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b03ebf3-5e70-423a-8373-fa6af05da2d8-dshm\") pod \"embedding-w0-680229-kserve-5ff6df78d7-tbscw\" (UID: \"4b03ebf3-5e70-423a-8373-fa6af05da2d8\") " pod="e2e-models-e2e-worker-w0-680229/embedding-w0-680229-kserve-5ff6df78d7-tbscw"
Sep 15 12:25:14.860596 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:25:14.860572    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b03ebf3-5e70-423a-8373-fa6af05da2d8-home\") pod \"embedding-w0-680229-kserve-5ff6df78d7-tbscw\" (UID: \"4b03ebf3-5e70-423a-8373-fa6af05da2d8\") " pod="e2e-models-e2e-worker-w0-680229/embedding-w0-680229-kserve-5ff6df78d7-tbscw"
Sep 15 12:25:14.860648 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:25:14.860598    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b03ebf3-5e70-423a-8373-fa6af05da2d8-model-cache\") pod \"embedding-w0-680229-kserve-5ff6df78d7-tbscw\" (UID: \"4b03ebf3-5e70-423a-8373-fa6af05da2d8\") " pod="e2e-models-e2e-worker-w0-680229/embedding-w0-680229-kserve-5ff6df78d7-tbscw"
Sep 15 12:25:14.860648 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:25:14.860605    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b03ebf3-5e70-423a-8373-fa6af05da2d8-tmp-dir\") pod \"embedding-w0-680229-kserve-5ff6df78d7-tbscw\" (UID: \"4b03ebf3-5e70-423a-8373-fa6af05da2d8\") " pod="e2e-models-e2e-worker-w0-680229/embedding-w0-680229-kserve-5ff6df78d7-tbscw"
Sep 15 12:25:14.862417 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:25:14.862392    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b03ebf3-5e70-423a-8373-fa6af05da2d8-dshm\") pod \"embedding-w0-680229-kserve-5ff6df78d7-tbscw\" (UID: \"4b03ebf3-5e70-423a-8373-fa6af05da2d8\") " pod="e2e-models-e2e-worker-w0-680229/embedding-w0-680229-kserve-5ff6df78d7-tbscw"
Sep 15 12:25:14.862592 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:25:14.862574    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b03ebf3-5e70-423a-8373-fa6af05da2d8-tls-certs\") pod \"embedding-w0-680229-kserve-5ff6df78d7-tbscw\" (UID: \"4b03ebf3-5e70-423a-8373-fa6af05da2d8\") " pod="e2e-models-e2e-worker-w0-680229/embedding-w0-680229-kserve-5ff6df78d7-tbscw"
Sep 15 12:25:14.869235 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:25:14.869214    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wdvd\" (UniqueName: \"kubernetes.io/projected/4b03ebf3-5e70-423a-8373-fa6af05da2d8-kube-api-access-6wdvd\") pod \"embedding-w0-680229-kserve-5ff6df78d7-tbscw\" (UID: \"4b03ebf3-5e70-423a-8373-fa6af05da2d8\") " pod="e2e-models-e2e-worker-w0-680229/embedding-w0-680229-kserve-5ff6df78d7-tbscw"
Sep 15 12:25:15.011038 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:25:15.010949    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-680229/embedding-w0-680229-kserve-5ff6df78d7-tbscw"
Sep 15 12:25:15.292739 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:25:15.292454    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-680229/embedding-w0-680229-kserve-5ff6df78d7-tbscw"]
Sep 15 12:25:15.293831 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:25:15.293801    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4b03ebf3_5e70_423a_8373_fa6af05da2d8.slice/crio-a1ec0ee209b684517b81e30945fc719b1a23e5fd54349006eb95fc6522891386 WatchSource:0}: Error finding container a1ec0ee209b684517b81e30945fc719b1a23e5fd54349006eb95fc6522891386: Status 404 returned error can't find the container with id a1ec0ee209b684517b81e30945fc719b1a23e5fd54349006eb95fc6522891386
Sep 15 12:25:16.235782 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:25:16.235741    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-680229/embedding-w0-680229-kserve-5ff6df78d7-tbscw" event={"ID":"4b03ebf3-5e70-423a-8373-fa6af05da2d8","Type":"ContainerStarted","Data":"433d2244a11de198587b96967b996ea2778bbbee52754408f65f5209f182276a"}
Sep 15 12:25:16.236230 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:25:16.235794    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-680229/embedding-w0-680229-kserve-5ff6df78d7-tbscw" event={"ID":"4b03ebf3-5e70-423a-8373-fa6af05da2d8","Type":"ContainerStarted","Data":"a1ec0ee209b684517b81e30945fc719b1a23e5fd54349006eb95fc6522891386"}
Sep 15 12:25:16.262113 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:25:16.262060    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-680229/embedding-w0-680229-kserve-5ff6df78d7-tbscw" podStartSLOduration=2.262044283 podStartE2EDuration="2.262044283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:25:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:25:16.261578938 +0000 UTC m=+1373.052501475" watchObservedRunningTime="2026-09-15 12:25:16.262044283 +0000 UTC m=+1373.052966793"
Sep 15 12:25:25.011771 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:25:25.011727    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-680229/embedding-w0-680229-kserve-5ff6df78d7-tbscw"
Sep 15 12:25:25.011771 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:25:25.011774    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-680229/embedding-w0-680229-kserve-5ff6df78d7-tbscw"
Sep 15 12:25:25.024429 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:25:25.024397    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-680229/embedding-w0-680229-kserve-5ff6df78d7-tbscw"
Sep 15 12:25:25.298238 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:25:25.298147    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-680229/embedding-w0-680229-kserve-5ff6df78d7-tbscw"
Sep 15 12:25:41.189399 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:25:41.189366    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5gkxs"]
Sep 15 12:26:07.482223 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:26:07.482184    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5gkxs"]
Sep 15 12:26:50.381308 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:26:50.381272    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5gkxs"]
Sep 15 12:26:58.929518 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:26:58.929480    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-680229-cxhkh"]
Sep 15 12:26:58.939354 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:26:58.939323    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-680229-cxhkh"
Sep 15 12:26:58.943609 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:26:58.943583    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 12:26:58.943609 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:26:58.943598    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tbvkt\""
Sep 15 12:26:58.964204 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:26:58.964170    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-680229-cxhkh"]
Sep 15 12:26:59.037813 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:26:59.037781    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a21bea94-6968-4905-b70a-57a19fd56052-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-680229-cxhkh\" (UID: \"a21bea94-6968-4905-b70a-57a19fd56052\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-680229-cxhkh"
Sep 15 12:26:59.138940 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:26:59.138911    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a21bea94-6968-4905-b70a-57a19fd56052-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-680229-cxhkh\" (UID: \"a21bea94-6968-4905-b70a-57a19fd56052\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-680229-cxhkh"
Sep 15 12:26:59.139570 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:26:59.139552    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a21bea94-6968-4905-b70a-57a19fd56052-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-680229-cxhkh\" (UID: \"a21bea94-6968-4905-b70a-57a19fd56052\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-680229-cxhkh"
Sep 15 12:26:59.249228 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:26:59.249125    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-680229-cxhkh"
Sep 15 12:26:59.529746 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:26:59.529721    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-680229-cxhkh"]
Sep 15 12:26:59.530393 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:26:59.530366    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda21bea94_6968_4905_b70a_57a19fd56052.slice/crio-eefdef9b67c6cac12fcebc656399003350c7135a64ef904df9f4b691d7ac10dc WatchSource:0}: Error finding container eefdef9b67c6cac12fcebc656399003350c7135a64ef904df9f4b691d7ac10dc: Status 404 returned error can't find the container with id eefdef9b67c6cac12fcebc656399003350c7135a64ef904df9f4b691d7ac10dc
Sep 15 12:26:59.822145 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:26:59.822062    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-680229-cxhkh" event={"ID":"a21bea94-6968-4905-b70a-57a19fd56052","Type":"ContainerStarted","Data":"eefdef9b67c6cac12fcebc656399003350c7135a64ef904df9f4b691d7ac10dc"}
Sep 15 12:27:00.829256 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:00.829223    2555 generic.go:356] "Generic (PLEG): container finished" podID="a21bea94-6968-4905-b70a-57a19fd56052" containerID="bced5b9321245b06bdbec0684eab4a1c321e440b115405dc865dc8ca2aaa6b11" exitCode=0
Sep 15 12:27:00.829726 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:00.829278    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-680229-cxhkh" event={"ID":"a21bea94-6968-4905-b70a-57a19fd56052","Type":"ContainerDied","Data":"bced5b9321245b06bdbec0684eab4a1c321e440b115405dc865dc8ca2aaa6b11"}
Sep 15 12:27:01.968642 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:01.968613    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-680229-cxhkh"
Sep 15 12:27:02.068910 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:02.068875    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a21bea94-6968-4905-b70a-57a19fd56052-maas-api-service-ca\") pod \"a21bea94-6968-4905-b70a-57a19fd56052\" (UID: \"a21bea94-6968-4905-b70a-57a19fd56052\") "
Sep 15 12:27:02.069221 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:02.069199    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a21bea94-6968-4905-b70a-57a19fd56052-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a21bea94-6968-4905-b70a-57a19fd56052" (UID: "a21bea94-6968-4905-b70a-57a19fd56052"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:27:02.169726 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:02.169677    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a21bea94-6968-4905-b70a-57a19fd56052-maas-api-service-ca\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:27:02.843692 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:02.843664    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-680229-cxhkh"
Sep 15 12:27:02.843888 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:02.843704    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-680229-cxhkh" event={"ID":"a21bea94-6968-4905-b70a-57a19fd56052","Type":"ContainerDied","Data":"eefdef9b67c6cac12fcebc656399003350c7135a64ef904df9f4b691d7ac10dc"}
Sep 15 12:27:02.843888 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:02.843754    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eefdef9b67c6cac12fcebc656399003350c7135a64ef904df9f4b691d7ac10dc"
Sep 15 12:27:03.829304 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:03.829250    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-680229/distinct-2-w0-680229-kserve-858ddb69cc-pfn5m"]
Sep 15 12:27:03.839287 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:03.839238    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-680229/distinct-w0-680229-kserve-5fcc68849d-nmsc7"]
Sep 15 12:27:03.859548 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:03.859515    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-680229/embedding-w0-680229-kserve-5ff6df78d7-tbscw"]
Sep 15 12:27:03.876096 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:03.876052    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-680229/premium-w0-680229-kserve-67cfbb7659-xt66w"]
Sep 15 12:27:03.895183 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:03.895144    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-680229/simulator-w0-680229-kserve-6bbc79c558-r5j9c"]
Sep 15 12:27:08.268971 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:08.268938    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-eb9bbe-65578df9fd-ltsvl"]
Sep 15 12:27:08.269468 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:08.269337    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a21bea94-6968-4905-b70a-57a19fd56052" containerName="revoke-keys"
Sep 15 12:27:08.269468 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:08.269350    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a21bea94-6968-4905-b70a-57a19fd56052" containerName="revoke-keys"
Sep 15 12:27:08.269468 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:08.269411    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a21bea94-6968-4905-b70a-57a19fd56052" containerName="revoke-keys"
Sep 15 12:27:08.272590 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:08.272570    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-eb9bbe-65578df9fd-ltsvl"
Sep 15 12:27:08.275429 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:08.275408    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-eb9bbe-dockercfg-llvrf\""
Sep 15 12:27:08.275537 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:08.275431    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-eb9bbe\""
Sep 15 12:27:08.291295 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:08.291253    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-eb9bbe-65578df9fd-ltsvl"]
Sep 15 12:27:08.434078 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:08.434031    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d68fcaa-5381-4a5d-ab48-6dce8a627458-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-eb9bbe-65578df9fd-ltsvl\" (UID: \"7d68fcaa-5381-4a5d-ab48-6dce8a627458\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-eb9bbe-65578df9fd-ltsvl"
Sep 15 12:27:08.434250 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:08.434090    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhgpt\" (UniqueName: \"kubernetes.io/projected/7d68fcaa-5381-4a5d-ab48-6dce8a627458-kube-api-access-vhgpt\") pod \"payload-pre-processing-e2e-api-b-eb9bbe-65578df9fd-ltsvl\" (UID: \"7d68fcaa-5381-4a5d-ab48-6dce8a627458\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-eb9bbe-65578df9fd-ltsvl"
Sep 15 12:27:08.535407 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:08.535318    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d68fcaa-5381-4a5d-ab48-6dce8a627458-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-eb9bbe-65578df9fd-ltsvl\" (UID: \"7d68fcaa-5381-4a5d-ab48-6dce8a627458\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-eb9bbe-65578df9fd-ltsvl"
Sep 15 12:27:08.535407 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:08.535368    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhgpt\" (UniqueName: \"kubernetes.io/projected/7d68fcaa-5381-4a5d-ab48-6dce8a627458-kube-api-access-vhgpt\") pod \"payload-pre-processing-e2e-api-b-eb9bbe-65578df9fd-ltsvl\" (UID: \"7d68fcaa-5381-4a5d-ab48-6dce8a627458\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-eb9bbe-65578df9fd-ltsvl"
Sep 15 12:27:08.535974 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:08.535951    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d68fcaa-5381-4a5d-ab48-6dce8a627458-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-eb9bbe-65578df9fd-ltsvl\" (UID: \"7d68fcaa-5381-4a5d-ab48-6dce8a627458\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-eb9bbe-65578df9fd-ltsvl"
Sep 15 12:27:08.545189 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:08.545160    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhgpt\" (UniqueName: \"kubernetes.io/projected/7d68fcaa-5381-4a5d-ab48-6dce8a627458-kube-api-access-vhgpt\") pod \"payload-pre-processing-e2e-api-b-eb9bbe-65578df9fd-ltsvl\" (UID: \"7d68fcaa-5381-4a5d-ab48-6dce8a627458\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-eb9bbe-65578df9fd-ltsvl"
Sep 15 12:27:08.582848 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:08.582809    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-eb9bbe-65578df9fd-ltsvl"
Sep 15 12:27:08.861595 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:08.861567    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-eb9bbe-65578df9fd-ltsvl"]
Sep 15 12:27:08.863327 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:27:08.863300    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d68fcaa_5381_4a5d_ab48_6dce8a627458.slice/crio-7a9c30ab58b244b7f69b742703e6e116ae061bc4489fcf8f0f92e3689ee3d567 WatchSource:0}: Error finding container 7a9c30ab58b244b7f69b742703e6e116ae061bc4489fcf8f0f92e3689ee3d567: Status 404 returned error can't find the container with id 7a9c30ab58b244b7f69b742703e6e116ae061bc4489fcf8f0f92e3689ee3d567
Sep 15 12:27:08.880383 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:08.880354    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-eb9bbe-65578df9fd-ltsvl" event={"ID":"7d68fcaa-5381-4a5d-ab48-6dce8a627458","Type":"ContainerStarted","Data":"7a9c30ab58b244b7f69b742703e6e116ae061bc4489fcf8f0f92e3689ee3d567"}
Sep 15 12:27:09.892192 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:09.892153    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-eb9bbe-65578df9fd-ltsvl" event={"ID":"7d68fcaa-5381-4a5d-ab48-6dce8a627458","Type":"ContainerStarted","Data":"fcf763912bb9642143ad385f7aeaa70e326108d1fcdf786d5d418b890cf04723"}
Sep 15 12:27:09.892676 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:09.892289    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-eb9bbe-65578df9fd-ltsvl"
Sep 15 12:27:17.834747 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:17.834689    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-eb9bbe-65578df9fd-ltsvl" podStartSLOduration=9.834672725 podStartE2EDuration="9.834672725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:27:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:27:09.916242034 +0000 UTC m=+1486.707164555" watchObservedRunningTime="2026-09-15 12:27:17.834672725 +0000 UTC m=+1494.625595234"
Sep 15 12:27:17.836505 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:17.836484    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-680229-cxhkh"]
Sep 15 12:27:17.840990 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:17.840967    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-680229-cxhkh"]
Sep 15 12:27:18.851167 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:18.851099    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-680229/distinct-2-w0-680229-kserve-858ddb69cc-pfn5m" podUID="7970f806-b3df-4c68-9388-db776b5fc8fb" containerName="main" containerID="cri-o://2de83679c23c5e9dd32f45a27aa476d97fd75235dd1b6956679646691d65fe80" gracePeriod=45
Sep 15 12:27:18.862703 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:18.862663    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-680229/distinct-w0-680229-kserve-5fcc68849d-nmsc7" podUID="3c2b1801-220f-4b66-a232-09aa568820f9" containerName="main" containerID="cri-o://3a9b08fc7ecfe6a4ed3b26fc7a8b779c83b0768b2a68b70ab57c1d86033e596e" gracePeriod=45
Sep 15 12:27:18.879847 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:18.879807    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-680229/embedding-w0-680229-kserve-5ff6df78d7-tbscw" podUID="4b03ebf3-5e70-423a-8373-fa6af05da2d8" containerName="main" containerID="cri-o://433d2244a11de198587b96967b996ea2778bbbee52754408f65f5209f182276a" gracePeriod=45
Sep 15 12:27:18.895816 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:18.895768    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-680229/premium-w0-680229-kserve-67cfbb7659-xt66w" podUID="4759dbe9-eeff-4ef1-aefc-59d9fa2ee19f" containerName="main" containerID="cri-o://4b485a3aba5d2d4cb91d8db2c65d7af0346e540c6a534563e049a071f3d6f75a" gracePeriod=45
Sep 15 12:27:18.914486 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:18.914450    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-680229/simulator-w0-680229-kserve-6bbc79c558-r5j9c" podUID="b3409087-fe02-4ea0-9662-f682cd4fda20" containerName="main" containerID="cri-o://743d8529096a17348a805e86b88a62169f6358809573d5e9eece6ef1d1bc0a3a" gracePeriod=45
Sep 15 12:27:19.211767 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.211740    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-680229/distinct-w0-680229-kserve-5fcc68849d-nmsc7"
Sep 15 12:27:19.240275 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.240232    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-680229/simulator-w0-680229-kserve-6bbc79c558-r5j9c"
Sep 15 12:27:19.243059 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.243036    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3409087-fe02-4ea0-9662-f682cd4fda20-tls-certs\") pod \"b3409087-fe02-4ea0-9662-f682cd4fda20\" (UID: \"b3409087-fe02-4ea0-9662-f682cd4fda20\") "
Sep 15 12:27:19.243206 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.243070    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6sgxx\" (UniqueName: \"kubernetes.io/projected/b3409087-fe02-4ea0-9662-f682cd4fda20-kube-api-access-6sgxx\") pod \"b3409087-fe02-4ea0-9662-f682cd4fda20\" (UID: \"b3409087-fe02-4ea0-9662-f682cd4fda20\") "
Sep 15 12:27:19.243206 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.243121    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c2b1801-220f-4b66-a232-09aa568820f9-home\") pod \"3c2b1801-220f-4b66-a232-09aa568820f9\" (UID: \"3c2b1801-220f-4b66-a232-09aa568820f9\") "
Sep 15 12:27:19.243206 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.243184    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3409087-fe02-4ea0-9662-f682cd4fda20-model-cache\") pod \"b3409087-fe02-4ea0-9662-f682cd4fda20\" (UID: \"b3409087-fe02-4ea0-9662-f682cd4fda20\") "
Sep 15 12:27:19.243404 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.243215    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c2b1801-220f-4b66-a232-09aa568820f9-tmp-dir\") pod \"3c2b1801-220f-4b66-a232-09aa568820f9\" (UID: \"3c2b1801-220f-4b66-a232-09aa568820f9\") "
Sep 15 12:27:19.243404 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.243235    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c2b1801-220f-4b66-a232-09aa568820f9-dshm\") pod \"3c2b1801-220f-4b66-a232-09aa568820f9\" (UID: \"3c2b1801-220f-4b66-a232-09aa568820f9\") "
Sep 15 12:27:19.243404 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.243308    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3409087-fe02-4ea0-9662-f682cd4fda20-tmp-dir\") pod \"b3409087-fe02-4ea0-9662-f682cd4fda20\" (UID: \"b3409087-fe02-4ea0-9662-f682cd4fda20\") "
Sep 15 12:27:19.243404 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.243353    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3409087-fe02-4ea0-9662-f682cd4fda20-home\") pod \"b3409087-fe02-4ea0-9662-f682cd4fda20\" (UID: \"b3409087-fe02-4ea0-9662-f682cd4fda20\") "
Sep 15 12:27:19.243404 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.243385    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c2b1801-220f-4b66-a232-09aa568820f9-tls-certs\") pod \"3c2b1801-220f-4b66-a232-09aa568820f9\" (UID: \"3c2b1801-220f-4b66-a232-09aa568820f9\") "
Sep 15 12:27:19.243650 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.243413    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3409087-fe02-4ea0-9662-f682cd4fda20-dshm\") pod \"b3409087-fe02-4ea0-9662-f682cd4fda20\" (UID: \"b3409087-fe02-4ea0-9662-f682cd4fda20\") "
Sep 15 12:27:19.243650 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.243451    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c2b1801-220f-4b66-a232-09aa568820f9-model-cache\") pod \"3c2b1801-220f-4b66-a232-09aa568820f9\" (UID: \"3c2b1801-220f-4b66-a232-09aa568820f9\") "
Sep 15 12:27:19.243650 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.243525    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c5bkb\" (UniqueName: \"kubernetes.io/projected/3c2b1801-220f-4b66-a232-09aa568820f9-kube-api-access-c5bkb\") pod \"3c2b1801-220f-4b66-a232-09aa568820f9\" (UID: \"3c2b1801-220f-4b66-a232-09aa568820f9\") "
Sep 15 12:27:19.243802 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.243663    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c2b1801-220f-4b66-a232-09aa568820f9-home" (OuterVolumeSpecName: "home") pod "3c2b1801-220f-4b66-a232-09aa568820f9" (UID: "3c2b1801-220f-4b66-a232-09aa568820f9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:27:19.243893 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.243864    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3409087-fe02-4ea0-9662-f682cd4fda20-model-cache" (OuterVolumeSpecName: "model-cache") pod "b3409087-fe02-4ea0-9662-f682cd4fda20" (UID: "b3409087-fe02-4ea0-9662-f682cd4fda20"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:27:19.243962 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.243936    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-680229/premium-w0-680229-kserve-67cfbb7659-xt66w"
Sep 15 12:27:19.244044 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.244025    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3409087-fe02-4ea0-9662-f682cd4fda20-model-cache\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:27:19.244094 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.244050    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c2b1801-220f-4b66-a232-09aa568820f9-home\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:27:19.244094 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.244046    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c2b1801-220f-4b66-a232-09aa568820f9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3c2b1801-220f-4b66-a232-09aa568820f9" (UID: "3c2b1801-220f-4b66-a232-09aa568820f9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:27:19.244253 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.244200    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3409087-fe02-4ea0-9662-f682cd4fda20-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b3409087-fe02-4ea0-9662-f682cd4fda20" (UID: "b3409087-fe02-4ea0-9662-f682cd4fda20"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:27:19.244475 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.244454    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3409087-fe02-4ea0-9662-f682cd4fda20-home" (OuterVolumeSpecName: "home") pod "b3409087-fe02-4ea0-9662-f682cd4fda20" (UID: "b3409087-fe02-4ea0-9662-f682cd4fda20"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:27:19.246308 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.244726    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c2b1801-220f-4b66-a232-09aa568820f9-model-cache" (OuterVolumeSpecName: "model-cache") pod "3c2b1801-220f-4b66-a232-09aa568820f9" (UID: "3c2b1801-220f-4b66-a232-09aa568820f9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:27:19.247736 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.247629    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3409087-fe02-4ea0-9662-f682cd4fda20-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b3409087-fe02-4ea0-9662-f682cd4fda20" (UID: "b3409087-fe02-4ea0-9662-f682cd4fda20"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:27:19.247851 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.247761    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c2b1801-220f-4b66-a232-09aa568820f9-dshm" (OuterVolumeSpecName: "dshm") pod "3c2b1801-220f-4b66-a232-09aa568820f9" (UID: "3c2b1801-220f-4b66-a232-09aa568820f9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:27:19.248184 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.247855    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c2b1801-220f-4b66-a232-09aa568820f9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3c2b1801-220f-4b66-a232-09aa568820f9" (UID: "3c2b1801-220f-4b66-a232-09aa568820f9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:27:19.248184 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.247898    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3409087-fe02-4ea0-9662-f682cd4fda20-dshm" (OuterVolumeSpecName: "dshm") pod "b3409087-fe02-4ea0-9662-f682cd4fda20" (UID: "b3409087-fe02-4ea0-9662-f682cd4fda20"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:27:19.248450 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.248353    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c2b1801-220f-4b66-a232-09aa568820f9-kube-api-access-c5bkb" (OuterVolumeSpecName: "kube-api-access-c5bkb") pod "3c2b1801-220f-4b66-a232-09aa568820f9" (UID: "3c2b1801-220f-4b66-a232-09aa568820f9"). InnerVolumeSpecName "kube-api-access-c5bkb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:27:19.248450 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.248408    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3409087-fe02-4ea0-9662-f682cd4fda20-kube-api-access-6sgxx" (OuterVolumeSpecName: "kube-api-access-6sgxx") pod "b3409087-fe02-4ea0-9662-f682cd4fda20" (UID: "b3409087-fe02-4ea0-9662-f682cd4fda20"). InnerVolumeSpecName "kube-api-access-6sgxx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:27:19.255561 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.255528    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-680229/embedding-w0-680229-kserve-5ff6df78d7-tbscw"
Sep 15 12:27:19.258747 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.258726    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-680229/distinct-2-w0-680229-kserve-858ddb69cc-pfn5m"
Sep 15 12:27:19.344474 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.344442    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4759dbe9-eeff-4ef1-aefc-59d9fa2ee19f-home\") pod \"4759dbe9-eeff-4ef1-aefc-59d9fa2ee19f\" (UID: \"4759dbe9-eeff-4ef1-aefc-59d9fa2ee19f\") "
Sep 15 12:27:19.344661 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.344480    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4759dbe9-eeff-4ef1-aefc-59d9fa2ee19f-dshm\") pod \"4759dbe9-eeff-4ef1-aefc-59d9fa2ee19f\" (UID: \"4759dbe9-eeff-4ef1-aefc-59d9fa2ee19f\") "
Sep 15 12:27:19.344661 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.344511    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b03ebf3-5e70-423a-8373-fa6af05da2d8-dshm\") pod \"4b03ebf3-5e70-423a-8373-fa6af05da2d8\" (UID: \"4b03ebf3-5e70-423a-8373-fa6af05da2d8\") "
Sep 15 12:27:19.344661 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.344560    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7970f806-b3df-4c68-9388-db776b5fc8fb-tls-certs\") pod \"7970f806-b3df-4c68-9388-db776b5fc8fb\" (UID: \"7970f806-b3df-4c68-9388-db776b5fc8fb\") "
Sep 15 12:27:19.344661 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.344585    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4759dbe9-eeff-4ef1-aefc-59d9fa2ee19f-tmp-dir\") pod \"4759dbe9-eeff-4ef1-aefc-59d9fa2ee19f\" (UID: \"4759dbe9-eeff-4ef1-aefc-59d9fa2ee19f\") "
Sep 15 12:27:19.344661 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.344609    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b03ebf3-5e70-423a-8373-fa6af05da2d8-home\") pod \"4b03ebf3-5e70-423a-8373-fa6af05da2d8\" (UID: \"4b03ebf3-5e70-423a-8373-fa6af05da2d8\") "
Sep 15 12:27:19.344661 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.344640    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b03ebf3-5e70-423a-8373-fa6af05da2d8-model-cache\") pod \"4b03ebf3-5e70-423a-8373-fa6af05da2d8\" (UID: \"4b03ebf3-5e70-423a-8373-fa6af05da2d8\") "
Sep 15 12:27:19.345009 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.344669    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4759dbe9-eeff-4ef1-aefc-59d9fa2ee19f-model-cache\") pod \"4759dbe9-eeff-4ef1-aefc-59d9fa2ee19f\" (UID: \"4759dbe9-eeff-4ef1-aefc-59d9fa2ee19f\") "
Sep 15 12:27:19.345009 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.344719    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7970f806-b3df-4c68-9388-db776b5fc8fb-home\") pod \"7970f806-b3df-4c68-9388-db776b5fc8fb\" (UID: \"7970f806-b3df-4c68-9388-db776b5fc8fb\") "
Sep 15 12:27:19.345009 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.344752    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4759dbe9-eeff-4ef1-aefc-59d9fa2ee19f-home" (OuterVolumeSpecName: "home") pod "4759dbe9-eeff-4ef1-aefc-59d9fa2ee19f" (UID: "4759dbe9-eeff-4ef1-aefc-59d9fa2ee19f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:27:19.345009 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.344780    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p4966\" (UniqueName: \"kubernetes.io/projected/4759dbe9-eeff-4ef1-aefc-59d9fa2ee19f-kube-api-access-p4966\") pod \"4759dbe9-eeff-4ef1-aefc-59d9fa2ee19f\" (UID: \"4759dbe9-eeff-4ef1-aefc-59d9fa2ee19f\") "
Sep 15 12:27:19.345009 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.344839    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7970f806-b3df-4c68-9388-db776b5fc8fb-model-cache\") pod \"7970f806-b3df-4c68-9388-db776b5fc8fb\" (UID: \"7970f806-b3df-4c68-9388-db776b5fc8fb\") "
Sep 15 12:27:19.345009 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.344865    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hz2sp\" (UniqueName: \"kubernetes.io/projected/7970f806-b3df-4c68-9388-db776b5fc8fb-kube-api-access-hz2sp\") pod \"7970f806-b3df-4c68-9388-db776b5fc8fb\" (UID: \"7970f806-b3df-4c68-9388-db776b5fc8fb\") "
Sep 15 12:27:19.345009 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.344896    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b03ebf3-5e70-423a-8373-fa6af05da2d8-tls-certs\") pod \"4b03ebf3-5e70-423a-8373-fa6af05da2d8\" (UID: \"4b03ebf3-5e70-423a-8373-fa6af05da2d8\") "
Sep 15 12:27:19.345009 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.344918    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6wdvd\" (UniqueName: \"kubernetes.io/projected/4b03ebf3-5e70-423a-8373-fa6af05da2d8-kube-api-access-6wdvd\") pod \"4b03ebf3-5e70-423a-8373-fa6af05da2d8\" (UID: \"4b03ebf3-5e70-423a-8373-fa6af05da2d8\") "
Sep 15 12:27:19.345470 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.345183    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b03ebf3-5e70-423a-8373-fa6af05da2d8-home" (OuterVolumeSpecName: "home") pod "4b03ebf3-5e70-423a-8373-fa6af05da2d8" (UID: "4b03ebf3-5e70-423a-8373-fa6af05da2d8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:27:19.346887 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.346134    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4759dbe9-eeff-4ef1-aefc-59d9fa2ee19f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4759dbe9-eeff-4ef1-aefc-59d9fa2ee19f" (UID: "4759dbe9-eeff-4ef1-aefc-59d9fa2ee19f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:27:19.346887 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.346331    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4759dbe9-eeff-4ef1-aefc-59d9fa2ee19f-model-cache" (OuterVolumeSpecName: "model-cache") pod "4759dbe9-eeff-4ef1-aefc-59d9fa2ee19f" (UID: "4759dbe9-eeff-4ef1-aefc-59d9fa2ee19f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:27:19.346887 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.346429    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7970f806-b3df-4c68-9388-db776b5fc8fb-home" (OuterVolumeSpecName: "home") pod "7970f806-b3df-4c68-9388-db776b5fc8fb" (UID: "7970f806-b3df-4c68-9388-db776b5fc8fb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:27:19.346887 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.346471    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7970f806-b3df-4c68-9388-db776b5fc8fb-dshm\") pod \"7970f806-b3df-4c68-9388-db776b5fc8fb\" (UID: \"7970f806-b3df-4c68-9388-db776b5fc8fb\") "
Sep 15 12:27:19.346887 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.346501    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4759dbe9-eeff-4ef1-aefc-59d9fa2ee19f-tls-certs\") pod \"4759dbe9-eeff-4ef1-aefc-59d9fa2ee19f\" (UID: \"4759dbe9-eeff-4ef1-aefc-59d9fa2ee19f\") "
Sep 15 12:27:19.346887 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.346686    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b03ebf3-5e70-423a-8373-fa6af05da2d8-model-cache" (OuterVolumeSpecName: "model-cache") pod "4b03ebf3-5e70-423a-8373-fa6af05da2d8" (UID: "4b03ebf3-5e70-423a-8373-fa6af05da2d8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:27:19.346887 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.346845    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b03ebf3-5e70-423a-8373-fa6af05da2d8-tmp-dir\") pod \"4b03ebf3-5e70-423a-8373-fa6af05da2d8\" (UID: \"4b03ebf3-5e70-423a-8373-fa6af05da2d8\") "
Sep 15 12:27:19.346887 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.346881    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7970f806-b3df-4c68-9388-db776b5fc8fb-tmp-dir\") pod \"7970f806-b3df-4c68-9388-db776b5fc8fb\" (UID: \"7970f806-b3df-4c68-9388-db776b5fc8fb\") "
Sep 15 12:27:19.347468 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.347127    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4759dbe9-eeff-4ef1-aefc-59d9fa2ee19f-dshm" (OuterVolumeSpecName: "dshm") pod "4759dbe9-eeff-4ef1-aefc-59d9fa2ee19f" (UID: "4759dbe9-eeff-4ef1-aefc-59d9fa2ee19f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:27:19.347468 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.347300    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3409087-fe02-4ea0-9662-f682cd4fda20-tmp-dir\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:27:19.347468 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.347320    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3409087-fe02-4ea0-9662-f682cd4fda20-home\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:27:19.347468 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.347333    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c2b1801-220f-4b66-a232-09aa568820f9-tls-certs\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:27:19.347468 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.347347    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3409087-fe02-4ea0-9662-f682cd4fda20-dshm\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:27:19.347468 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.347359    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c2b1801-220f-4b66-a232-09aa568820f9-model-cache\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:27:19.347468 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.347371    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4759dbe9-eeff-4ef1-aefc-59d9fa2ee19f-home\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:27:19.347468 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.347383    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4759dbe9-eeff-4ef1-aefc-59d9fa2ee19f-dshm\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:27:19.347468 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.347398    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5bkb\" (UniqueName: \"kubernetes.io/projected/3c2b1801-220f-4b66-a232-09aa568820f9-kube-api-access-c5bkb\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:27:19.347468 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.347411    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4759dbe9-eeff-4ef1-aefc-59d9fa2ee19f-tmp-dir\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:27:19.347468 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.347423    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b03ebf3-5e70-423a-8373-fa6af05da2d8-home\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:27:19.347468 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.347435    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b03ebf3-5e70-423a-8373-fa6af05da2d8-model-cache\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:27:19.347468 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.347447    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4759dbe9-eeff-4ef1-aefc-59d9fa2ee19f-model-cache\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:27:19.347468 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.347458    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7970f806-b3df-4c68-9388-db776b5fc8fb-home\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:27:19.347468 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.347470    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3409087-fe02-4ea0-9662-f682cd4fda20-tls-certs\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:27:19.348357 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.347483    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sgxx\" (UniqueName: \"kubernetes.io/projected/b3409087-fe02-4ea0-9662-f682cd4fda20-kube-api-access-6sgxx\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:27:19.348357 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.347495    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c2b1801-220f-4b66-a232-09aa568820f9-tmp-dir\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:27:19.348357 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.347508    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c2b1801-220f-4b66-a232-09aa568820f9-dshm\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:27:19.348357 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.347999    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7970f806-b3df-4c68-9388-db776b5fc8fb-model-cache" (OuterVolumeSpecName: "model-cache") pod "7970f806-b3df-4c68-9388-db776b5fc8fb" (UID: "7970f806-b3df-4c68-9388-db776b5fc8fb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:27:19.348357 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.348212    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7970f806-b3df-4c68-9388-db776b5fc8fb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7970f806-b3df-4c68-9388-db776b5fc8fb" (UID: "7970f806-b3df-4c68-9388-db776b5fc8fb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:27:19.348902 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.348866    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b03ebf3-5e70-423a-8373-fa6af05da2d8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4b03ebf3-5e70-423a-8373-fa6af05da2d8" (UID: "4b03ebf3-5e70-423a-8373-fa6af05da2d8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:27:19.349225 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.349197    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b03ebf3-5e70-423a-8373-fa6af05da2d8-dshm" (OuterVolumeSpecName: "dshm") pod "4b03ebf3-5e70-423a-8373-fa6af05da2d8" (UID: "4b03ebf3-5e70-423a-8373-fa6af05da2d8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:27:19.349686 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.349652    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4759dbe9-eeff-4ef1-aefc-59d9fa2ee19f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4759dbe9-eeff-4ef1-aefc-59d9fa2ee19f" (UID: "4759dbe9-eeff-4ef1-aefc-59d9fa2ee19f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:27:19.349790 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.349741    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b03ebf3-5e70-423a-8373-fa6af05da2d8-kube-api-access-6wdvd" (OuterVolumeSpecName: "kube-api-access-6wdvd") pod "4b03ebf3-5e70-423a-8373-fa6af05da2d8" (UID: "4b03ebf3-5e70-423a-8373-fa6af05da2d8"). InnerVolumeSpecName "kube-api-access-6wdvd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:27:19.350045 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.350019    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4759dbe9-eeff-4ef1-aefc-59d9fa2ee19f-kube-api-access-p4966" (OuterVolumeSpecName: "kube-api-access-p4966") pod "4759dbe9-eeff-4ef1-aefc-59d9fa2ee19f" (UID: "4759dbe9-eeff-4ef1-aefc-59d9fa2ee19f"). InnerVolumeSpecName "kube-api-access-p4966". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:27:19.350147 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.350086    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7970f806-b3df-4c68-9388-db776b5fc8fb-kube-api-access-hz2sp" (OuterVolumeSpecName: "kube-api-access-hz2sp") pod "7970f806-b3df-4c68-9388-db776b5fc8fb" (UID: "7970f806-b3df-4c68-9388-db776b5fc8fb"). InnerVolumeSpecName "kube-api-access-hz2sp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:27:19.350464 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.350431    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7970f806-b3df-4c68-9388-db776b5fc8fb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7970f806-b3df-4c68-9388-db776b5fc8fb" (UID: "7970f806-b3df-4c68-9388-db776b5fc8fb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:27:19.350464 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.350437    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b03ebf3-5e70-423a-8373-fa6af05da2d8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4b03ebf3-5e70-423a-8373-fa6af05da2d8" (UID: "4b03ebf3-5e70-423a-8373-fa6af05da2d8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:27:19.351309 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.351253    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7970f806-b3df-4c68-9388-db776b5fc8fb-dshm" (OuterVolumeSpecName: "dshm") pod "7970f806-b3df-4c68-9388-db776b5fc8fb" (UID: "7970f806-b3df-4c68-9388-db776b5fc8fb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:27:19.448167 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.448077    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b03ebf3-5e70-423a-8373-fa6af05da2d8-tmp-dir\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:27:19.448167 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.448117    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7970f806-b3df-4c68-9388-db776b5fc8fb-tmp-dir\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:27:19.448167 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.448131    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b03ebf3-5e70-423a-8373-fa6af05da2d8-dshm\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:27:19.448167 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.448143    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7970f806-b3df-4c68-9388-db776b5fc8fb-tls-certs\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:27:19.448167 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.448158    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4966\" (UniqueName: \"kubernetes.io/projected/4759dbe9-eeff-4ef1-aefc-59d9fa2ee19f-kube-api-access-p4966\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:27:19.448484 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.448172    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7970f806-b3df-4c68-9388-db776b5fc8fb-model-cache\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:27:19.448484 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.448187    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hz2sp\" (UniqueName: \"kubernetes.io/projected/7970f806-b3df-4c68-9388-db776b5fc8fb-kube-api-access-hz2sp\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:27:19.448484 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.448203    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b03ebf3-5e70-423a-8373-fa6af05da2d8-tls-certs\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:27:19.448484 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.448215    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wdvd\" (UniqueName: \"kubernetes.io/projected/4b03ebf3-5e70-423a-8373-fa6af05da2d8-kube-api-access-6wdvd\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:27:19.448484 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.448225    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7970f806-b3df-4c68-9388-db776b5fc8fb-dshm\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:27:19.448484 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.448237    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4759dbe9-eeff-4ef1-aefc-59d9fa2ee19f-tls-certs\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:27:19.819312 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.819216    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a21bea94-6968-4905-b70a-57a19fd56052" path="/var/lib/kubelet/pods/a21bea94-6968-4905-b70a-57a19fd56052/volumes"
Sep 15 12:27:19.964248 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.964208    2555 generic.go:356] "Generic (PLEG): container finished" podID="7970f806-b3df-4c68-9388-db776b5fc8fb" containerID="2de83679c23c5e9dd32f45a27aa476d97fd75235dd1b6956679646691d65fe80" exitCode=0
Sep 15 12:27:19.964709 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.964283    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-680229/distinct-2-w0-680229-kserve-858ddb69cc-pfn5m"
Sep 15 12:27:19.964709 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.964297    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-680229/distinct-2-w0-680229-kserve-858ddb69cc-pfn5m" event={"ID":"7970f806-b3df-4c68-9388-db776b5fc8fb","Type":"ContainerDied","Data":"2de83679c23c5e9dd32f45a27aa476d97fd75235dd1b6956679646691d65fe80"}
Sep 15 12:27:19.964709 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.964343    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-680229/distinct-2-w0-680229-kserve-858ddb69cc-pfn5m" event={"ID":"7970f806-b3df-4c68-9388-db776b5fc8fb","Type":"ContainerDied","Data":"b5b8ba9c8b7f31b131898d038f04238264a48ff71abb1d54b69f7f368842fbdc"}
Sep 15 12:27:19.964709 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.964365    2555 scope.go:117] "RemoveContainer" containerID="2de83679c23c5e9dd32f45a27aa476d97fd75235dd1b6956679646691d65fe80"
Sep 15 12:27:19.965961 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.965924    2555 generic.go:356] "Generic (PLEG): container finished" podID="b3409087-fe02-4ea0-9662-f682cd4fda20" containerID="743d8529096a17348a805e86b88a62169f6358809573d5e9eece6ef1d1bc0a3a" exitCode=0
Sep 15 12:27:19.966057 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.965965    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-680229/simulator-w0-680229-kserve-6bbc79c558-r5j9c" event={"ID":"b3409087-fe02-4ea0-9662-f682cd4fda20","Type":"ContainerDied","Data":"743d8529096a17348a805e86b88a62169f6358809573d5e9eece6ef1d1bc0a3a"}
Sep 15 12:27:19.966057 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.965985    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-680229/simulator-w0-680229-kserve-6bbc79c558-r5j9c"
Sep 15 12:27:19.966177 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.966006    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-680229/simulator-w0-680229-kserve-6bbc79c558-r5j9c" event={"ID":"b3409087-fe02-4ea0-9662-f682cd4fda20","Type":"ContainerDied","Data":"fe589709f37b19d4178e720b5953dbdd4a971e379b7400fd7e3220cb11840ac5"}
Sep 15 12:27:19.967606 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.967583    2555 generic.go:356] "Generic (PLEG): container finished" podID="4759dbe9-eeff-4ef1-aefc-59d9fa2ee19f" containerID="4b485a3aba5d2d4cb91d8db2c65d7af0346e540c6a534563e049a071f3d6f75a" exitCode=0
Sep 15 12:27:19.967727 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.967635    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-680229/premium-w0-680229-kserve-67cfbb7659-xt66w"
Sep 15 12:27:19.967727 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.967670    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-680229/premium-w0-680229-kserve-67cfbb7659-xt66w" event={"ID":"4759dbe9-eeff-4ef1-aefc-59d9fa2ee19f","Type":"ContainerDied","Data":"4b485a3aba5d2d4cb91d8db2c65d7af0346e540c6a534563e049a071f3d6f75a"}
Sep 15 12:27:19.967727 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.967710    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-680229/premium-w0-680229-kserve-67cfbb7659-xt66w" event={"ID":"4759dbe9-eeff-4ef1-aefc-59d9fa2ee19f","Type":"ContainerDied","Data":"3b493510c8651e4a4db327726c571b83efab68a3609373d3a9388590a5bef7c4"}
Sep 15 12:27:19.969279 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.969230    2555 generic.go:356] "Generic (PLEG): container finished" podID="3c2b1801-220f-4b66-a232-09aa568820f9" containerID="3a9b08fc7ecfe6a4ed3b26fc7a8b779c83b0768b2a68b70ab57c1d86033e596e" exitCode=0
Sep 15 12:27:19.969401 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.969300    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-680229/distinct-w0-680229-kserve-5fcc68849d-nmsc7"
Sep 15 12:27:19.969401 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.969296    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-680229/distinct-w0-680229-kserve-5fcc68849d-nmsc7" event={"ID":"3c2b1801-220f-4b66-a232-09aa568820f9","Type":"ContainerDied","Data":"3a9b08fc7ecfe6a4ed3b26fc7a8b779c83b0768b2a68b70ab57c1d86033e596e"}
Sep 15 12:27:19.969524 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.969403    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-680229/distinct-w0-680229-kserve-5fcc68849d-nmsc7" event={"ID":"3c2b1801-220f-4b66-a232-09aa568820f9","Type":"ContainerDied","Data":"dfb06f4cce080c5504355cfe36544d0fcd71807847746d924ac8222c1eb3a5c6"}
Sep 15 12:27:19.970801 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.970751    2555 generic.go:356] "Generic (PLEG): container finished" podID="4b03ebf3-5e70-423a-8373-fa6af05da2d8" containerID="433d2244a11de198587b96967b996ea2778bbbee52754408f65f5209f182276a" exitCode=0
Sep 15 12:27:19.970801 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.970795    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-680229/embedding-w0-680229-kserve-5ff6df78d7-tbscw" event={"ID":"4b03ebf3-5e70-423a-8373-fa6af05da2d8","Type":"ContainerDied","Data":"433d2244a11de198587b96967b996ea2778bbbee52754408f65f5209f182276a"}
Sep 15 12:27:19.970946 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.970784    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-680229/embedding-w0-680229-kserve-5ff6df78d7-tbscw"
Sep 15 12:27:19.970946 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.970822    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-680229/embedding-w0-680229-kserve-5ff6df78d7-tbscw" event={"ID":"4b03ebf3-5e70-423a-8373-fa6af05da2d8","Type":"ContainerDied","Data":"a1ec0ee209b684517b81e30945fc719b1a23e5fd54349006eb95fc6522891386"}
Sep 15 12:27:19.976211 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.976194    2555 scope.go:117] "RemoveContainer" containerID="2de83679c23c5e9dd32f45a27aa476d97fd75235dd1b6956679646691d65fe80"
Sep 15 12:27:19.976545 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:27:19.976525    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2de83679c23c5e9dd32f45a27aa476d97fd75235dd1b6956679646691d65fe80\": container with ID starting with 2de83679c23c5e9dd32f45a27aa476d97fd75235dd1b6956679646691d65fe80 not found: ID does not exist" containerID="2de83679c23c5e9dd32f45a27aa476d97fd75235dd1b6956679646691d65fe80"
Sep 15 12:27:19.976594 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.976553    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2de83679c23c5e9dd32f45a27aa476d97fd75235dd1b6956679646691d65fe80"} err="failed to get container status \"2de83679c23c5e9dd32f45a27aa476d97fd75235dd1b6956679646691d65fe80\": rpc error: code = NotFound desc = could not find container \"2de83679c23c5e9dd32f45a27aa476d97fd75235dd1b6956679646691d65fe80\": container with ID starting with 2de83679c23c5e9dd32f45a27aa476d97fd75235dd1b6956679646691d65fe80 not found: ID does not exist"
Sep 15 12:27:19.976594 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.976570    2555 scope.go:117] "RemoveContainer" containerID="743d8529096a17348a805e86b88a62169f6358809573d5e9eece6ef1d1bc0a3a"
Sep 15 12:27:19.984955 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.984939    2555 scope.go:117] "RemoveContainer" containerID="743d8529096a17348a805e86b88a62169f6358809573d5e9eece6ef1d1bc0a3a"
Sep 15 12:27:19.985211 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:27:19.985194    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"743d8529096a17348a805e86b88a62169f6358809573d5e9eece6ef1d1bc0a3a\": container with ID starting with 743d8529096a17348a805e86b88a62169f6358809573d5e9eece6ef1d1bc0a3a not found: ID does not exist" containerID="743d8529096a17348a805e86b88a62169f6358809573d5e9eece6ef1d1bc0a3a"
Sep 15 12:27:19.985251 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.985226    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"743d8529096a17348a805e86b88a62169f6358809573d5e9eece6ef1d1bc0a3a"} err="failed to get container status \"743d8529096a17348a805e86b88a62169f6358809573d5e9eece6ef1d1bc0a3a\": rpc error: code = NotFound desc = could not find container \"743d8529096a17348a805e86b88a62169f6358809573d5e9eece6ef1d1bc0a3a\": container with ID starting with 743d8529096a17348a805e86b88a62169f6358809573d5e9eece6ef1d1bc0a3a not found: ID does not exist"
Sep 15 12:27:19.985251 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.985242    2555 scope.go:117] "RemoveContainer" containerID="4b485a3aba5d2d4cb91d8db2c65d7af0346e540c6a534563e049a071f3d6f75a"
Sep 15 12:27:19.993535 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.993519    2555 scope.go:117] "RemoveContainer" containerID="4b485a3aba5d2d4cb91d8db2c65d7af0346e540c6a534563e049a071f3d6f75a"
Sep 15 12:27:19.993851 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:27:19.993829    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b485a3aba5d2d4cb91d8db2c65d7af0346e540c6a534563e049a071f3d6f75a\": container with ID starting with 4b485a3aba5d2d4cb91d8db2c65d7af0346e540c6a534563e049a071f3d6f75a not found: ID does not exist" containerID="4b485a3aba5d2d4cb91d8db2c65d7af0346e540c6a534563e049a071f3d6f75a"
Sep 15 12:27:19.993943 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.993861    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b485a3aba5d2d4cb91d8db2c65d7af0346e540c6a534563e049a071f3d6f75a"} err="failed to get container status \"4b485a3aba5d2d4cb91d8db2c65d7af0346e540c6a534563e049a071f3d6f75a\": rpc error: code = NotFound desc = could not find container \"4b485a3aba5d2d4cb91d8db2c65d7af0346e540c6a534563e049a071f3d6f75a\": container with ID starting with 4b485a3aba5d2d4cb91d8db2c65d7af0346e540c6a534563e049a071f3d6f75a not found: ID does not exist"
Sep 15 12:27:19.993943 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.993884    2555 scope.go:117] "RemoveContainer" containerID="3a9b08fc7ecfe6a4ed3b26fc7a8b779c83b0768b2a68b70ab57c1d86033e596e"
Sep 15 12:27:19.998995 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:19.998969    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-680229/simulator-w0-680229-kserve-6bbc79c558-r5j9c"]
Sep 15 12:27:20.002441 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:20.002425    2555 scope.go:117] "RemoveContainer" containerID="3a9b08fc7ecfe6a4ed3b26fc7a8b779c83b0768b2a68b70ab57c1d86033e596e"
Sep 15 12:27:20.002795 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:27:20.002735    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a9b08fc7ecfe6a4ed3b26fc7a8b779c83b0768b2a68b70ab57c1d86033e596e\": container with ID starting with 3a9b08fc7ecfe6a4ed3b26fc7a8b779c83b0768b2a68b70ab57c1d86033e596e not found: ID does not exist" containerID="3a9b08fc7ecfe6a4ed3b26fc7a8b779c83b0768b2a68b70ab57c1d86033e596e"
Sep 15 12:27:20.002795 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:20.002764    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a9b08fc7ecfe6a4ed3b26fc7a8b779c83b0768b2a68b70ab57c1d86033e596e"} err="failed to get container status \"3a9b08fc7ecfe6a4ed3b26fc7a8b779c83b0768b2a68b70ab57c1d86033e596e\": rpc error: code = NotFound desc = could not find container \"3a9b08fc7ecfe6a4ed3b26fc7a8b779c83b0768b2a68b70ab57c1d86033e596e\": container with ID starting with 3a9b08fc7ecfe6a4ed3b26fc7a8b779c83b0768b2a68b70ab57c1d86033e596e not found: ID does not exist"
Sep 15 12:27:20.002795 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:20.002786    2555 scope.go:117] "RemoveContainer" containerID="433d2244a11de198587b96967b996ea2778bbbee52754408f65f5209f182276a"
Sep 15 12:27:20.006494 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:20.006473    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-680229/simulator-w0-680229-kserve-6bbc79c558-r5j9c"]
Sep 15 12:27:20.011200 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:20.011182    2555 scope.go:117] "RemoveContainer" containerID="433d2244a11de198587b96967b996ea2778bbbee52754408f65f5209f182276a"
Sep 15 12:27:20.011572 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:27:20.011553    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"433d2244a11de198587b96967b996ea2778bbbee52754408f65f5209f182276a\": container with ID starting with 433d2244a11de198587b96967b996ea2778bbbee52754408f65f5209f182276a not found: ID does not exist" containerID="433d2244a11de198587b96967b996ea2778bbbee52754408f65f5209f182276a"
Sep 15 12:27:20.011644 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:20.011580    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"433d2244a11de198587b96967b996ea2778bbbee52754408f65f5209f182276a"} err="failed to get container status \"433d2244a11de198587b96967b996ea2778bbbee52754408f65f5209f182276a\": rpc error: code = NotFound desc = could not find container \"433d2244a11de198587b96967b996ea2778bbbee52754408f65f5209f182276a\": container with ID starting with 433d2244a11de198587b96967b996ea2778bbbee52754408f65f5209f182276a not found: ID does not exist"
Sep 15 12:27:20.021967 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:20.021944    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-680229/distinct-w0-680229-kserve-5fcc68849d-nmsc7"]
Sep 15 12:27:20.036541 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:20.036516    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-680229/distinct-w0-680229-kserve-5fcc68849d-nmsc7"]
Sep 15 12:27:20.063911 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:20.063874    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-680229/premium-w0-680229-kserve-67cfbb7659-xt66w"]
Sep 15 12:27:20.069301 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:20.069257    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-680229/premium-w0-680229-kserve-67cfbb7659-xt66w"]
Sep 15 12:27:20.088304 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:20.088206    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-680229/embedding-w0-680229-kserve-5ff6df78d7-tbscw"]
Sep 15 12:27:20.104854 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:20.104820    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-680229/embedding-w0-680229-kserve-5ff6df78d7-tbscw"]
Sep 15 12:27:20.133740 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:20.133705    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-680229/distinct-2-w0-680229-kserve-858ddb69cc-pfn5m"]
Sep 15 12:27:20.140530 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:20.140499    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-680229/distinct-2-w0-680229-kserve-858ddb69cc-pfn5m"]
Sep 15 12:27:20.900455 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:20.900424    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-eb9bbe-65578df9fd-ltsvl"
Sep 15 12:27:21.817366 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:21.817337    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c2b1801-220f-4b66-a232-09aa568820f9" path="/var/lib/kubelet/pods/3c2b1801-220f-4b66-a232-09aa568820f9/volumes"
Sep 15 12:27:21.817760 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:21.817741    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4759dbe9-eeff-4ef1-aefc-59d9fa2ee19f" path="/var/lib/kubelet/pods/4759dbe9-eeff-4ef1-aefc-59d9fa2ee19f/volumes"
Sep 15 12:27:21.818086 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:21.818074    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b03ebf3-5e70-423a-8373-fa6af05da2d8" path="/var/lib/kubelet/pods/4b03ebf3-5e70-423a-8373-fa6af05da2d8/volumes"
Sep 15 12:27:21.818748 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:21.818733    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7970f806-b3df-4c68-9388-db776b5fc8fb" path="/var/lib/kubelet/pods/7970f806-b3df-4c68-9388-db776b5fc8fb/volumes"
Sep 15 12:27:21.819112 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:21.819099    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3409087-fe02-4ea0-9662-f682cd4fda20" path="/var/lib/kubelet/pods/b3409087-fe02-4ea0-9662-f682cd4fda20/volumes"
Sep 15 12:27:26.959115 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:26.959081    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f1e118-d5lbg"]
Sep 15 12:27:26.959508 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:26.959456    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c2b1801-220f-4b66-a232-09aa568820f9" containerName="main"
Sep 15 12:27:26.959508 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:26.959469    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c2b1801-220f-4b66-a232-09aa568820f9" containerName="main"
Sep 15 12:27:26.959508 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:26.959479    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3409087-fe02-4ea0-9662-f682cd4fda20" containerName="main"
Sep 15 12:27:26.959508 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:26.959484    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3409087-fe02-4ea0-9662-f682cd4fda20" containerName="main"
Sep 15 12:27:26.959508 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:26.959493    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b03ebf3-5e70-423a-8373-fa6af05da2d8" containerName="main"
Sep 15 12:27:26.959508 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:26.959499    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b03ebf3-5e70-423a-8373-fa6af05da2d8" containerName="main"
Sep 15 12:27:26.959508 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:26.959511    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4759dbe9-eeff-4ef1-aefc-59d9fa2ee19f" containerName="main"
Sep 15 12:27:26.959750 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:26.959517    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="4759dbe9-eeff-4ef1-aefc-59d9fa2ee19f" containerName="main"
Sep 15 12:27:26.959750 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:26.959523    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7970f806-b3df-4c68-9388-db776b5fc8fb" containerName="main"
Sep 15 12:27:26.959750 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:26.959529    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="7970f806-b3df-4c68-9388-db776b5fc8fb" containerName="main"
Sep 15 12:27:26.959750 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:26.959588    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c2b1801-220f-4b66-a232-09aa568820f9" containerName="main"
Sep 15 12:27:26.959750 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:26.959595    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="7970f806-b3df-4c68-9388-db776b5fc8fb" containerName="main"
Sep 15 12:27:26.959750 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:26.959599    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="4759dbe9-eeff-4ef1-aefc-59d9fa2ee19f" containerName="main"
Sep 15 12:27:26.959750 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:26.959605    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3409087-fe02-4ea0-9662-f682cd4fda20" containerName="main"
Sep 15 12:27:26.959750 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:26.959611    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b03ebf3-5e70-423a-8373-fa6af05da2d8" containerName="main"
Sep 15 12:27:26.964063 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:26.964045    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f1e118-d5lbg"
Sep 15 12:27:26.967582 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:26.967549    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 12:27:26.967734 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:26.967646    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tbvkt\""
Sep 15 12:27:26.974412 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:26.974388    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f1e118-d5lbg"]
Sep 15 12:27:27.023027 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:27.022997    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e726604-47ed-4564-9dc5-3f5cde9daac6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f1e118-d5lbg\" (UID: \"0e726604-47ed-4564-9dc5-3f5cde9daac6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f1e118-d5lbg"
Sep 15 12:27:27.124644 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:27.124607    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e726604-47ed-4564-9dc5-3f5cde9daac6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f1e118-d5lbg\" (UID: \"0e726604-47ed-4564-9dc5-3f5cde9daac6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f1e118-d5lbg"
Sep 15 12:27:27.125294 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:27.125240    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e726604-47ed-4564-9dc5-3f5cde9daac6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f1e118-d5lbg\" (UID: \"0e726604-47ed-4564-9dc5-3f5cde9daac6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f1e118-d5lbg"
Sep 15 12:27:27.276185 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:27.276095    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f1e118-d5lbg"
Sep 15 12:27:27.555020 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:27.554937    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f1e118-d5lbg"]
Sep 15 12:27:27.556889 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:27:27.556847    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e726604_47ed_4564_9dc5_3f5cde9daac6.slice/crio-c665083b316fa9aed72b2545eff7131c1b6f5da2e0bff477bd0f69ec658a9565 WatchSource:0}: Error finding container c665083b316fa9aed72b2545eff7131c1b6f5da2e0bff477bd0f69ec658a9565: Status 404 returned error can't find the container with id c665083b316fa9aed72b2545eff7131c1b6f5da2e0bff477bd0f69ec658a9565
Sep 15 12:27:28.019599 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:28.019555    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f1e118-d5lbg" event={"ID":"0e726604-47ed-4564-9dc5-3f5cde9daac6","Type":"ContainerStarted","Data":"94520d89d2200c947d3e99cbace6fe48869476d396d8c529bb09436288d01adf"}
Sep 15 12:27:28.020005 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:28.019613    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f1e118-d5lbg" event={"ID":"0e726604-47ed-4564-9dc5-3f5cde9daac6","Type":"ContainerStarted","Data":"c665083b316fa9aed72b2545eff7131c1b6f5da2e0bff477bd0f69ec658a9565"}
Sep 15 12:27:28.037091 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:28.037037    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f1e118-d5lbg" podStartSLOduration=2.037022963 podStartE2EDuration="2.037022963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:27:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:27:28.03560793 +0000 UTC m=+1504.826530467" watchObservedRunningTime="2026-09-15 12:27:28.037022963 +0000 UTC m=+1504.827945474"
Sep 15 12:27:29.028076 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:29.028041    2555 generic.go:356] "Generic (PLEG): container finished" podID="0e726604-47ed-4564-9dc5-3f5cde9daac6" containerID="94520d89d2200c947d3e99cbace6fe48869476d396d8c529bb09436288d01adf" exitCode=0
Sep 15 12:27:29.028488 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:29.028131    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f1e118-d5lbg" event={"ID":"0e726604-47ed-4564-9dc5-3f5cde9daac6","Type":"ContainerDied","Data":"94520d89d2200c947d3e99cbace6fe48869476d396d8c529bb09436288d01adf"}
Sep 15 12:27:30.167229 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:30.167204    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f1e118-d5lbg"
Sep 15 12:27:30.256996 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:30.256964    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e726604-47ed-4564-9dc5-3f5cde9daac6-maas-api-service-ca\") pod \"0e726604-47ed-4564-9dc5-3f5cde9daac6\" (UID: \"0e726604-47ed-4564-9dc5-3f5cde9daac6\") "
Sep 15 12:27:30.257455 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:30.257427    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e726604-47ed-4564-9dc5-3f5cde9daac6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0e726604-47ed-4564-9dc5-3f5cde9daac6" (UID: "0e726604-47ed-4564-9dc5-3f5cde9daac6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:27:30.358294 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:30.358191    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e726604-47ed-4564-9dc5-3f5cde9daac6-maas-api-service-ca\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:27:31.042095 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:31.042062    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f1e118-d5lbg"
Sep 15 12:27:31.042095 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:31.042075    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f1e118-d5lbg" event={"ID":"0e726604-47ed-4564-9dc5-3f5cde9daac6","Type":"ContainerDied","Data":"c665083b316fa9aed72b2545eff7131c1b6f5da2e0bff477bd0f69ec658a9565"}
Sep 15 12:27:31.042359 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:31.042119    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c665083b316fa9aed72b2545eff7131c1b6f5da2e0bff477bd0f69ec658a9565"
Sep 15 12:27:42.969043 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:42.969000    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f1e118-d5lbg"]
Sep 15 12:27:42.971682 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:42.971655    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f1e118-d5lbg"]
Sep 15 12:27:43.817646 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:43.817612    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e726604-47ed-4564-9dc5-3f5cde9daac6" path="/var/lib/kubelet/pods/0e726604-47ed-4564-9dc5-3f5cde9daac6/volumes"
Sep 15 12:27:50.772135 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:50.772034    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-eb9bbe-2cmq8"]
Sep 15 12:27:50.772659 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:50.772636    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e726604-47ed-4564-9dc5-3f5cde9daac6" containerName="revoke-keys"
Sep 15 12:27:50.772733 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:50.772664    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e726604-47ed-4564-9dc5-3f5cde9daac6" containerName="revoke-keys"
Sep 15 12:27:50.772820 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:50.772807    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e726604-47ed-4564-9dc5-3f5cde9daac6" containerName="revoke-keys"
Sep 15 12:27:50.777628 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:50.777599    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-eb9bbe-2cmq8"
Sep 15 12:27:50.780349 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:50.780326    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 12:27:50.780807 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:50.780334    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tbvkt\""
Sep 15 12:27:50.785210 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:50.785176    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-eb9bbe-2cmq8"]
Sep 15 12:27:50.958949 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:50.958905    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d68b1309-de3b-4e26-aa04-de16d7322ab2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-eb9bbe-2cmq8\" (UID: \"d68b1309-de3b-4e26-aa04-de16d7322ab2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-eb9bbe-2cmq8"
Sep 15 12:27:51.060616 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:51.060524    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d68b1309-de3b-4e26-aa04-de16d7322ab2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-eb9bbe-2cmq8\" (UID: \"d68b1309-de3b-4e26-aa04-de16d7322ab2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-eb9bbe-2cmq8"
Sep 15 12:27:51.061177 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:51.061142    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d68b1309-de3b-4e26-aa04-de16d7322ab2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-eb9bbe-2cmq8\" (UID: \"d68b1309-de3b-4e26-aa04-de16d7322ab2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-eb9bbe-2cmq8"
Sep 15 12:27:51.091518 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:51.091486    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-eb9bbe-2cmq8"
Sep 15 12:27:51.371866 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:51.371838    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-eb9bbe-2cmq8"]
Sep 15 12:27:51.374028 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:27:51.373994    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd68b1309_de3b_4e26_aa04_de16d7322ab2.slice/crio-871a2a53ed58303ad8149b912576e840ee7140eff0622397ac4ce47540c14eb8 WatchSource:0}: Error finding container 871a2a53ed58303ad8149b912576e840ee7140eff0622397ac4ce47540c14eb8: Status 404 returned error can't find the container with id 871a2a53ed58303ad8149b912576e840ee7140eff0622397ac4ce47540c14eb8
Sep 15 12:27:52.177904 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:52.177868    2555 generic.go:356] "Generic (PLEG): container finished" podID="d68b1309-de3b-4e26-aa04-de16d7322ab2" containerID="dd56b88652d30f43d3fdef34113dbec5ed98ed059427e6fd32de1fe86ebfcc68" exitCode=0
Sep 15 12:27:52.178322 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:52.177982    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-eb9bbe-2cmq8" event={"ID":"d68b1309-de3b-4e26-aa04-de16d7322ab2","Type":"ContainerDied","Data":"dd56b88652d30f43d3fdef34113dbec5ed98ed059427e6fd32de1fe86ebfcc68"}
Sep 15 12:27:52.178322 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:52.178035    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-eb9bbe-2cmq8" event={"ID":"d68b1309-de3b-4e26-aa04-de16d7322ab2","Type":"ContainerStarted","Data":"871a2a53ed58303ad8149b912576e840ee7140eff0622397ac4ce47540c14eb8"}
Sep 15 12:27:53.309304 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:53.309274    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-eb9bbe-2cmq8"
Sep 15 12:27:53.484255 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:53.484159    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d68b1309-de3b-4e26-aa04-de16d7322ab2-maas-api-service-ca\") pod \"d68b1309-de3b-4e26-aa04-de16d7322ab2\" (UID: \"d68b1309-de3b-4e26-aa04-de16d7322ab2\") "
Sep 15 12:27:53.484559 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:53.484537    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d68b1309-de3b-4e26-aa04-de16d7322ab2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d68b1309-de3b-4e26-aa04-de16d7322ab2" (UID: "d68b1309-de3b-4e26-aa04-de16d7322ab2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:27:53.585406 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:53.585370    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d68b1309-de3b-4e26-aa04-de16d7322ab2-maas-api-service-ca\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:27:54.191318 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:54.191285    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-eb9bbe-2cmq8"
Sep 15 12:27:54.191520 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:54.191318    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-eb9bbe-2cmq8" event={"ID":"d68b1309-de3b-4e26-aa04-de16d7322ab2","Type":"ContainerDied","Data":"871a2a53ed58303ad8149b912576e840ee7140eff0622397ac4ce47540c14eb8"}
Sep 15 12:27:54.191520 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:27:54.191374    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="871a2a53ed58303ad8149b912576e840ee7140eff0622397ac4ce47540c14eb8"
Sep 15 12:28:06.629671 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:06.629638    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-eb9bbe-65578df9fd-ltsvl"]
Sep 15 12:28:06.630178 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:06.629845    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-eb9bbe-65578df9fd-ltsvl" podUID="7d68fcaa-5381-4a5d-ab48-6dce8a627458" containerName="payload-pre-processing" containerID="cri-o://fcf763912bb9642143ad385f7aeaa70e326108d1fcdf786d5d418b890cf04723" gracePeriod=30
Sep 15 12:28:06.885356 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:06.885328    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-eb9bbe-65578df9fd-ltsvl"
Sep 15 12:28:06.912296 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:06.912237    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d68fcaa-5381-4a5d-ab48-6dce8a627458-plugins-config-volume\") pod \"7d68fcaa-5381-4a5d-ab48-6dce8a627458\" (UID: \"7d68fcaa-5381-4a5d-ab48-6dce8a627458\") "
Sep 15 12:28:06.912491 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:06.912327    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vhgpt\" (UniqueName: \"kubernetes.io/projected/7d68fcaa-5381-4a5d-ab48-6dce8a627458-kube-api-access-vhgpt\") pod \"7d68fcaa-5381-4a5d-ab48-6dce8a627458\" (UID: \"7d68fcaa-5381-4a5d-ab48-6dce8a627458\") "
Sep 15 12:28:06.912914 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:06.912633    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d68fcaa-5381-4a5d-ab48-6dce8a627458-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7d68fcaa-5381-4a5d-ab48-6dce8a627458" (UID: "7d68fcaa-5381-4a5d-ab48-6dce8a627458"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:28:06.912914 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:06.912712    2555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d68fcaa-5381-4a5d-ab48-6dce8a627458-plugins-config-volume\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:28:06.914998 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:06.914955    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d68fcaa-5381-4a5d-ab48-6dce8a627458-kube-api-access-vhgpt" (OuterVolumeSpecName: "kube-api-access-vhgpt") pod "7d68fcaa-5381-4a5d-ab48-6dce8a627458" (UID: "7d68fcaa-5381-4a5d-ab48-6dce8a627458"). InnerVolumeSpecName "kube-api-access-vhgpt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:28:06.922028 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:06.922004    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-eb9bbe-2cmq8"]
Sep 15 12:28:06.925955 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:06.925931    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-eb9bbe-2cmq8"]
Sep 15 12:28:07.013488 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:07.013450    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhgpt\" (UniqueName: \"kubernetes.io/projected/7d68fcaa-5381-4a5d-ab48-6dce8a627458-kube-api-access-vhgpt\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:28:07.265003 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:07.264918    2555 generic.go:356] "Generic (PLEG): container finished" podID="7d68fcaa-5381-4a5d-ab48-6dce8a627458" containerID="fcf763912bb9642143ad385f7aeaa70e326108d1fcdf786d5d418b890cf04723" exitCode=0
Sep 15 12:28:07.265003 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:07.264955    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-eb9bbe-65578df9fd-ltsvl" event={"ID":"7d68fcaa-5381-4a5d-ab48-6dce8a627458","Type":"ContainerDied","Data":"fcf763912bb9642143ad385f7aeaa70e326108d1fcdf786d5d418b890cf04723"}
Sep 15 12:28:07.265188 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:07.265012    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-eb9bbe-65578df9fd-ltsvl" event={"ID":"7d68fcaa-5381-4a5d-ab48-6dce8a627458","Type":"ContainerDied","Data":"7a9c30ab58b244b7f69b742703e6e116ae061bc4489fcf8f0f92e3689ee3d567"}
Sep 15 12:28:07.265188 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:07.265036    2555 scope.go:117] "RemoveContainer" containerID="fcf763912bb9642143ad385f7aeaa70e326108d1fcdf786d5d418b890cf04723"
Sep 15 12:28:07.265188 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:07.264965    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-eb9bbe-65578df9fd-ltsvl"
Sep 15 12:28:07.274415 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:07.274398    2555 scope.go:117] "RemoveContainer" containerID="fcf763912bb9642143ad385f7aeaa70e326108d1fcdf786d5d418b890cf04723"
Sep 15 12:28:07.274672 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:28:07.274653    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcf763912bb9642143ad385f7aeaa70e326108d1fcdf786d5d418b890cf04723\": container with ID starting with fcf763912bb9642143ad385f7aeaa70e326108d1fcdf786d5d418b890cf04723 not found: ID does not exist" containerID="fcf763912bb9642143ad385f7aeaa70e326108d1fcdf786d5d418b890cf04723"
Sep 15 12:28:07.274724 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:07.274680    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcf763912bb9642143ad385f7aeaa70e326108d1fcdf786d5d418b890cf04723"} err="failed to get container status \"fcf763912bb9642143ad385f7aeaa70e326108d1fcdf786d5d418b890cf04723\": rpc error: code = NotFound desc = could not find container \"fcf763912bb9642143ad385f7aeaa70e326108d1fcdf786d5d418b890cf04723\": container with ID starting with fcf763912bb9642143ad385f7aeaa70e326108d1fcdf786d5d418b890cf04723 not found: ID does not exist"
Sep 15 12:28:07.287641 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:07.287614    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-eb9bbe-65578df9fd-ltsvl"]
Sep 15 12:28:07.291324 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:07.291295    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-eb9bbe-65578df9fd-ltsvl"]
Sep 15 12:28:07.818008 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:07.817972    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d68fcaa-5381-4a5d-ab48-6dce8a627458" path="/var/lib/kubelet/pods/7d68fcaa-5381-4a5d-ab48-6dce8a627458/volumes"
Sep 15 12:28:07.818403 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:07.818316    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d68b1309-de3b-4e26-aa04-de16d7322ab2" path="/var/lib/kubelet/pods/d68b1309-de3b-4e26-aa04-de16d7322ab2/volumes"
Sep 15 12:28:11.074230 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:11.074192    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5gkxs"]
Sep 15 12:28:14.541854 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:14.541818    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-58b0fd-65zfz"]
Sep 15 12:28:14.542469 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:14.542387    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d68fcaa-5381-4a5d-ab48-6dce8a627458" containerName="payload-pre-processing"
Sep 15 12:28:14.542469 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:14.542408    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d68fcaa-5381-4a5d-ab48-6dce8a627458" containerName="payload-pre-processing"
Sep 15 12:28:14.542469 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:14.542430    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d68b1309-de3b-4e26-aa04-de16d7322ab2" containerName="revoke-keys"
Sep 15 12:28:14.542469 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:14.542440    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="d68b1309-de3b-4e26-aa04-de16d7322ab2" containerName="revoke-keys"
Sep 15 12:28:14.542700 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:14.542527    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d68fcaa-5381-4a5d-ab48-6dce8a627458" containerName="payload-pre-processing"
Sep 15 12:28:14.542700 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:14.542540    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="d68b1309-de3b-4e26-aa04-de16d7322ab2" containerName="revoke-keys"
Sep 15 12:28:14.545688 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:14.545667    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-58b0fd-65zfz"
Sep 15 12:28:14.549226 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:14.549208    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 12:28:14.549455 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:14.549438    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tbvkt\""
Sep 15 12:28:14.557971 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:14.557948    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-58b0fd-65zfz"]
Sep 15 12:28:14.685987 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:14.685947    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1fdf13c-603a-44f7-b385-cf7fbfbc8632-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-58b0fd-65zfz\" (UID: \"b1fdf13c-603a-44f7-b385-cf7fbfbc8632\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-58b0fd-65zfz"
Sep 15 12:28:14.786647 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:14.786607    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1fdf13c-603a-44f7-b385-cf7fbfbc8632-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-58b0fd-65zfz\" (UID: \"b1fdf13c-603a-44f7-b385-cf7fbfbc8632\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-58b0fd-65zfz"
Sep 15 12:28:14.787257 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:14.787233    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1fdf13c-603a-44f7-b385-cf7fbfbc8632-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-58b0fd-65zfz\" (UID: \"b1fdf13c-603a-44f7-b385-cf7fbfbc8632\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-58b0fd-65zfz"
Sep 15 12:28:14.856075 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:14.855966    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-58b0fd-65zfz"
Sep 15 12:28:15.136581 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:15.136548    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-58b0fd-65zfz"]
Sep 15 12:28:15.138853 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:28:15.138812    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1fdf13c_603a_44f7_b385_cf7fbfbc8632.slice/crio-0a6d1f061da137084d9936d2a6649dcdec03189b3e8d6c810ce5d675a72cd5fb WatchSource:0}: Error finding container 0a6d1f061da137084d9936d2a6649dcdec03189b3e8d6c810ce5d675a72cd5fb: Status 404 returned error can't find the container with id 0a6d1f061da137084d9936d2a6649dcdec03189b3e8d6c810ce5d675a72cd5fb
Sep 15 12:28:15.313557 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:15.313519    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-58b0fd-65zfz" event={"ID":"b1fdf13c-603a-44f7-b385-cf7fbfbc8632","Type":"ContainerStarted","Data":"0a6d1f061da137084d9936d2a6649dcdec03189b3e8d6c810ce5d675a72cd5fb"}
Sep 15 12:28:16.321463 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:16.321429    2555 generic.go:356] "Generic (PLEG): container finished" podID="b1fdf13c-603a-44f7-b385-cf7fbfbc8632" containerID="3392816cc8eca6f95b8f3b3a3716544be0abeff5db0be62a30dafc58e8867615" exitCode=0
Sep 15 12:28:16.321854 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:16.321514    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-58b0fd-65zfz" event={"ID":"b1fdf13c-603a-44f7-b385-cf7fbfbc8632","Type":"ContainerDied","Data":"3392816cc8eca6f95b8f3b3a3716544be0abeff5db0be62a30dafc58e8867615"}
Sep 15 12:28:17.453905 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:17.453879    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-58b0fd-65zfz"
Sep 15 12:28:17.514988 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:17.514946    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1fdf13c-603a-44f7-b385-cf7fbfbc8632-maas-api-service-ca\") pod \"b1fdf13c-603a-44f7-b385-cf7fbfbc8632\" (UID: \"b1fdf13c-603a-44f7-b385-cf7fbfbc8632\") "
Sep 15 12:28:17.515339 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:17.515316    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1fdf13c-603a-44f7-b385-cf7fbfbc8632-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b1fdf13c-603a-44f7-b385-cf7fbfbc8632" (UID: "b1fdf13c-603a-44f7-b385-cf7fbfbc8632"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:28:17.616447 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:17.616361    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1fdf13c-603a-44f7-b385-cf7fbfbc8632-maas-api-service-ca\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:28:18.333795 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:18.333763    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-58b0fd-65zfz"
Sep 15 12:28:18.333992 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:18.333798    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-58b0fd-65zfz" event={"ID":"b1fdf13c-603a-44f7-b385-cf7fbfbc8632","Type":"ContainerDied","Data":"0a6d1f061da137084d9936d2a6649dcdec03189b3e8d6c810ce5d675a72cd5fb"}
Sep 15 12:28:18.333992 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:18.333854    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a6d1f061da137084d9936d2a6649dcdec03189b3e8d6c810ce5d675a72cd5fb"
Sep 15 12:28:25.520744 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:25.520699    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-58b0fd-65zfz"]
Sep 15 12:28:25.525986 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:25.525952    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-58b0fd-65zfz"]
Sep 15 12:28:25.817355 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:25.817244    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1fdf13c-603a-44f7-b385-cf7fbfbc8632" path="/var/lib/kubelet/pods/b1fdf13c-603a-44f7-b385-cf7fbfbc8632/volumes"
Sep 15 12:28:33.667089 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:33.667048    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ba366c-qtvc2"]
Sep 15 12:28:33.667555 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:33.667465    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1fdf13c-603a-44f7-b385-cf7fbfbc8632" containerName="revoke-keys"
Sep 15 12:28:33.667555 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:33.667476    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1fdf13c-603a-44f7-b385-cf7fbfbc8632" containerName="revoke-keys"
Sep 15 12:28:33.667555 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:33.667546    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1fdf13c-603a-44f7-b385-cf7fbfbc8632" containerName="revoke-keys"
Sep 15 12:28:33.670751 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:33.670735    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ba366c-qtvc2"
Sep 15 12:28:33.673710 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:33.673684    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tbvkt\""
Sep 15 12:28:33.673846 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:33.673816    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 12:28:33.683907 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:33.683874    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ba366c-qtvc2"]
Sep 15 12:28:33.776223 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:33.776188    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2669f23f-b68d-45a9-99b7-f5ee66e4042a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-ba366c-qtvc2\" (UID: \"2669f23f-b68d-45a9-99b7-f5ee66e4042a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ba366c-qtvc2"
Sep 15 12:28:33.876993 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:33.876956    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2669f23f-b68d-45a9-99b7-f5ee66e4042a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-ba366c-qtvc2\" (UID: \"2669f23f-b68d-45a9-99b7-f5ee66e4042a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ba366c-qtvc2"
Sep 15 12:28:33.877616 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:33.877595    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2669f23f-b68d-45a9-99b7-f5ee66e4042a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-ba366c-qtvc2\" (UID: \"2669f23f-b68d-45a9-99b7-f5ee66e4042a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ba366c-qtvc2"
Sep 15 12:28:33.981588 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:33.981502    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ba366c-qtvc2"
Sep 15 12:28:34.256435 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:34.256399    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ba366c-qtvc2"]
Sep 15 12:28:34.257872 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:28:34.257846    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2669f23f_b68d_45a9_99b7_f5ee66e4042a.slice/crio-4653f48a3a7e6071f2390dc57971d82ef7fcc32be21ab665ee88ef754ca38e34 WatchSource:0}: Error finding container 4653f48a3a7e6071f2390dc57971d82ef7fcc32be21ab665ee88ef754ca38e34: Status 404 returned error can't find the container with id 4653f48a3a7e6071f2390dc57971d82ef7fcc32be21ab665ee88ef754ca38e34
Sep 15 12:28:34.426738 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:34.426696    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ba366c-qtvc2" event={"ID":"2669f23f-b68d-45a9-99b7-f5ee66e4042a","Type":"ContainerStarted","Data":"4653f48a3a7e6071f2390dc57971d82ef7fcc32be21ab665ee88ef754ca38e34"}
Sep 15 12:28:35.434089 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:35.434057    2555 generic.go:356] "Generic (PLEG): container finished" podID="2669f23f-b68d-45a9-99b7-f5ee66e4042a" containerID="3f091afac70a9dbbb47f25203b2ea0edb33b1e29490b45639342986a4e734950" exitCode=0
Sep 15 12:28:35.434599 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:35.434155    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ba366c-qtvc2" event={"ID":"2669f23f-b68d-45a9-99b7-f5ee66e4042a","Type":"ContainerDied","Data":"3f091afac70a9dbbb47f25203b2ea0edb33b1e29490b45639342986a4e734950"}
Sep 15 12:28:36.565563 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:36.565533    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ba366c-qtvc2"
Sep 15 12:28:36.578365 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:36.578336    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-9ba48f/distinct-2-w4-9ba48f-kserve-7fd5987948-s54l2"]
Sep 15 12:28:36.586110 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:36.586074    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-9ba48f/distinct-w4-9ba48f-kserve-cf9cd67f-7hsfr"]
Sep 15 12:28:36.605235 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:36.605204    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-9ba48f/premium-w4-9ba48f-kserve-5746585544-8rnhs"]
Sep 15 12:28:36.614033 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:36.614002    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-9ba48f/simulator-w4-9ba48f-kserve-67d55d8789-vvwbq"]
Sep 15 12:28:36.704381 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:36.704289    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2669f23f-b68d-45a9-99b7-f5ee66e4042a-maas-api-service-ca\") pod \"2669f23f-b68d-45a9-99b7-f5ee66e4042a\" (UID: \"2669f23f-b68d-45a9-99b7-f5ee66e4042a\") "
Sep 15 12:28:36.704658 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:36.704634    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2669f23f-b68d-45a9-99b7-f5ee66e4042a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2669f23f-b68d-45a9-99b7-f5ee66e4042a" (UID: "2669f23f-b68d-45a9-99b7-f5ee66e4042a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:28:36.805472 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:36.805438    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2669f23f-b68d-45a9-99b7-f5ee66e4042a-maas-api-service-ca\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:28:36.991371 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:36.991256    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5gkxs"]
Sep 15 12:28:37.447038 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:37.447002    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ba366c-qtvc2" event={"ID":"2669f23f-b68d-45a9-99b7-f5ee66e4042a","Type":"ContainerDied","Data":"4653f48a3a7e6071f2390dc57971d82ef7fcc32be21ab665ee88ef754ca38e34"}
Sep 15 12:28:37.447038 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:37.447036    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ba366c-qtvc2"
Sep 15 12:28:37.447253 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:37.447059    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4653f48a3a7e6071f2390dc57971d82ef7fcc32be21ab665ee88ef754ca38e34"
Sep 15 12:28:44.632815 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:44.632778    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ba366c-5c8b5d7794-rkhhq"]
Sep 15 12:28:44.633300 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:44.633044    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ba366c-5c8b5d7794-rkhhq" podUID="0770ba7c-72be-4fb5-bf05-bef6114ce762" containerName="payload-pre-processing" containerID="cri-o://0bf8a778559a73a9ed1864b8237333aa4af6083d2816cd66417135d3900739bc" gracePeriod=30
Sep 15 12:28:44.883043 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:44.882984    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ba366c-5c8b5d7794-rkhhq"
Sep 15 12:28:44.898617 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:44.898582    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ba366c-qtvc2"]
Sep 15 12:28:44.903001 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:44.902969    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ba366c-qtvc2"]
Sep 15 12:28:44.978360 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:44.978315    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0770ba7c-72be-4fb5-bf05-bef6114ce762-plugins-config-volume\") pod \"0770ba7c-72be-4fb5-bf05-bef6114ce762\" (UID: \"0770ba7c-72be-4fb5-bf05-bef6114ce762\") "
Sep 15 12:28:44.978562 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:44.978447    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jcj4d\" (UniqueName: \"kubernetes.io/projected/0770ba7c-72be-4fb5-bf05-bef6114ce762-kube-api-access-jcj4d\") pod \"0770ba7c-72be-4fb5-bf05-bef6114ce762\" (UID: \"0770ba7c-72be-4fb5-bf05-bef6114ce762\") "
Sep 15 12:28:44.978687 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:44.978663    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0770ba7c-72be-4fb5-bf05-bef6114ce762-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0770ba7c-72be-4fb5-bf05-bef6114ce762" (UID: "0770ba7c-72be-4fb5-bf05-bef6114ce762"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:28:44.980511 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:44.980490    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0770ba7c-72be-4fb5-bf05-bef6114ce762-kube-api-access-jcj4d" (OuterVolumeSpecName: "kube-api-access-jcj4d") pod "0770ba7c-72be-4fb5-bf05-bef6114ce762" (UID: "0770ba7c-72be-4fb5-bf05-bef6114ce762"). InnerVolumeSpecName "kube-api-access-jcj4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:28:45.079994 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:45.079949    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcj4d\" (UniqueName: \"kubernetes.io/projected/0770ba7c-72be-4fb5-bf05-bef6114ce762-kube-api-access-jcj4d\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:28:45.079994 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:45.079986    2555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0770ba7c-72be-4fb5-bf05-bef6114ce762-plugins-config-volume\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:28:45.494485 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:45.494446    2555 generic.go:356] "Generic (PLEG): container finished" podID="0770ba7c-72be-4fb5-bf05-bef6114ce762" containerID="0bf8a778559a73a9ed1864b8237333aa4af6083d2816cd66417135d3900739bc" exitCode=0
Sep 15 12:28:45.494485 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:45.494486    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ba366c-5c8b5d7794-rkhhq"
Sep 15 12:28:45.494808 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:45.494502    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ba366c-5c8b5d7794-rkhhq" event={"ID":"0770ba7c-72be-4fb5-bf05-bef6114ce762","Type":"ContainerDied","Data":"0bf8a778559a73a9ed1864b8237333aa4af6083d2816cd66417135d3900739bc"}
Sep 15 12:28:45.494808 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:45.494550    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ba366c-5c8b5d7794-rkhhq" event={"ID":"0770ba7c-72be-4fb5-bf05-bef6114ce762","Type":"ContainerDied","Data":"e65ed4386f9977302e9eb9a6ec126e8757a4aebe596b66fc07c4e10e5b834c79"}
Sep 15 12:28:45.494808 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:45.494573    2555 scope.go:117] "RemoveContainer" containerID="0bf8a778559a73a9ed1864b8237333aa4af6083d2816cd66417135d3900739bc"
Sep 15 12:28:45.505223 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:45.505204    2555 scope.go:117] "RemoveContainer" containerID="0bf8a778559a73a9ed1864b8237333aa4af6083d2816cd66417135d3900739bc"
Sep 15 12:28:45.505590 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:28:45.505569    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bf8a778559a73a9ed1864b8237333aa4af6083d2816cd66417135d3900739bc\": container with ID starting with 0bf8a778559a73a9ed1864b8237333aa4af6083d2816cd66417135d3900739bc not found: ID does not exist" containerID="0bf8a778559a73a9ed1864b8237333aa4af6083d2816cd66417135d3900739bc"
Sep 15 12:28:45.505641 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:45.505601    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bf8a778559a73a9ed1864b8237333aa4af6083d2816cd66417135d3900739bc"} err="failed to get container status \"0bf8a778559a73a9ed1864b8237333aa4af6083d2816cd66417135d3900739bc\": rpc error: code = NotFound desc = could not find container \"0bf8a778559a73a9ed1864b8237333aa4af6083d2816cd66417135d3900739bc\": container with ID starting with 0bf8a778559a73a9ed1864b8237333aa4af6083d2816cd66417135d3900739bc not found: ID does not exist"
Sep 15 12:28:45.538847 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:45.538804    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ba366c-5c8b5d7794-rkhhq"]
Sep 15 12:28:45.550617 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:45.550582    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ba366c-5c8b5d7794-rkhhq"]
Sep 15 12:28:45.817095 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:45.817010    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0770ba7c-72be-4fb5-bf05-bef6114ce762" path="/var/lib/kubelet/pods/0770ba7c-72be-4fb5-bf05-bef6114ce762/volumes"
Sep 15 12:28:45.817478 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:45.817395    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2669f23f-b68d-45a9-99b7-f5ee66e4042a" path="/var/lib/kubelet/pods/2669f23f-b68d-45a9-99b7-f5ee66e4042a/volumes"
Sep 15 12:28:51.603364 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:51.603298    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-9ba48f/distinct-2-w4-9ba48f-kserve-7fd5987948-s54l2" podUID="49d7abed-95ea-4dc2-b066-2ea0ddfdff52" containerName="main" containerID="cri-o://4027c5e63a56bf22e8f49c36d1337fbd5d3182116180c0261a7e9fd46fbe86c8" gracePeriod=45
Sep 15 12:28:51.612745 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:51.612710    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-9ba48f/distinct-w4-9ba48f-kserve-cf9cd67f-7hsfr" podUID="fff9c4da-fbed-47c9-8f8a-b254c4489af3" containerName="main" containerID="cri-o://f1adb2977a7274f3da7731a46115b2fe0741033c3d5ac7aafddc0fd6279b3edb" gracePeriod=45
Sep 15 12:28:51.624839 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:51.624710    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-9ba48f/premium-w4-9ba48f-kserve-5746585544-8rnhs" podUID="e54c8cbe-3dad-4a5e-95d3-feeec2887d82" containerName="main" containerID="cri-o://9d20018093f0649e88ee94be1fc03948ccf0ef9f0ee73cdf95d92cbf429b4b1f" gracePeriod=45
Sep 15 12:28:51.633527 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:51.633487    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-9ba48f/simulator-w4-9ba48f-kserve-67d55d8789-vvwbq" podUID="a5485dc6-da0b-4a35-a7ae-c7047dce4faa" containerName="main" containerID="cri-o://dcc848dc24bf022799722a25b0f69d9deb970514cae6f5ae9e3d67d889185a48" gracePeriod=45
Sep 15 12:28:51.908483 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:51.908452    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9ba48f/premium-w4-9ba48f-kserve-5746585544-8rnhs"
Sep 15 12:28:51.939179 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:51.939148    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9ba48f/distinct-w4-9ba48f-kserve-cf9cd67f-7hsfr"
Sep 15 12:28:51.944898 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:51.944873    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e54c8cbe-3dad-4a5e-95d3-feeec2887d82-home\") pod \"e54c8cbe-3dad-4a5e-95d3-feeec2887d82\" (UID: \"e54c8cbe-3dad-4a5e-95d3-feeec2887d82\") "
Sep 15 12:28:51.945045 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:51.944957    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e54c8cbe-3dad-4a5e-95d3-feeec2887d82-tmp-dir\") pod \"e54c8cbe-3dad-4a5e-95d3-feeec2887d82\" (UID: \"e54c8cbe-3dad-4a5e-95d3-feeec2887d82\") "
Sep 15 12:28:51.945045 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:51.945026    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e54c8cbe-3dad-4a5e-95d3-feeec2887d82-dshm\") pod \"e54c8cbe-3dad-4a5e-95d3-feeec2887d82\" (UID: \"e54c8cbe-3dad-4a5e-95d3-feeec2887d82\") "
Sep 15 12:28:51.945164 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:51.945056    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e54c8cbe-3dad-4a5e-95d3-feeec2887d82-model-cache\") pod \"e54c8cbe-3dad-4a5e-95d3-feeec2887d82\" (UID: \"e54c8cbe-3dad-4a5e-95d3-feeec2887d82\") "
Sep 15 12:28:51.945164 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:51.945116    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lz64v\" (UniqueName: \"kubernetes.io/projected/e54c8cbe-3dad-4a5e-95d3-feeec2887d82-kube-api-access-lz64v\") pod \"e54c8cbe-3dad-4a5e-95d3-feeec2887d82\" (UID: \"e54c8cbe-3dad-4a5e-95d3-feeec2887d82\") "
Sep 15 12:28:51.945164 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:51.945145    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e54c8cbe-3dad-4a5e-95d3-feeec2887d82-tls-certs\") pod \"e54c8cbe-3dad-4a5e-95d3-feeec2887d82\" (UID: \"e54c8cbe-3dad-4a5e-95d3-feeec2887d82\") "
Sep 15 12:28:51.945668 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:51.945641    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e54c8cbe-3dad-4a5e-95d3-feeec2887d82-home" (OuterVolumeSpecName: "home") pod "e54c8cbe-3dad-4a5e-95d3-feeec2887d82" (UID: "e54c8cbe-3dad-4a5e-95d3-feeec2887d82"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:28:51.945770 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:51.945740    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e54c8cbe-3dad-4a5e-95d3-feeec2887d82-model-cache" (OuterVolumeSpecName: "model-cache") pod "e54c8cbe-3dad-4a5e-95d3-feeec2887d82" (UID: "e54c8cbe-3dad-4a5e-95d3-feeec2887d82"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:28:51.945770 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:51.945751    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e54c8cbe-3dad-4a5e-95d3-feeec2887d82-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e54c8cbe-3dad-4a5e-95d3-feeec2887d82" (UID: "e54c8cbe-3dad-4a5e-95d3-feeec2887d82"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:28:51.946217 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:51.946196    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9ba48f/simulator-w4-9ba48f-kserve-67d55d8789-vvwbq"
Sep 15 12:28:51.948207 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:51.948072    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e54c8cbe-3dad-4a5e-95d3-feeec2887d82-dshm" (OuterVolumeSpecName: "dshm") pod "e54c8cbe-3dad-4a5e-95d3-feeec2887d82" (UID: "e54c8cbe-3dad-4a5e-95d3-feeec2887d82"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:28:51.948207 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:51.948166    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e54c8cbe-3dad-4a5e-95d3-feeec2887d82-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e54c8cbe-3dad-4a5e-95d3-feeec2887d82" (UID: "e54c8cbe-3dad-4a5e-95d3-feeec2887d82"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:28:51.948397 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:51.948238    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e54c8cbe-3dad-4a5e-95d3-feeec2887d82-kube-api-access-lz64v" (OuterVolumeSpecName: "kube-api-access-lz64v") pod "e54c8cbe-3dad-4a5e-95d3-feeec2887d82" (UID: "e54c8cbe-3dad-4a5e-95d3-feeec2887d82"). InnerVolumeSpecName "kube-api-access-lz64v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:28:51.951465 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:51.951446    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9ba48f/distinct-2-w4-9ba48f-kserve-7fd5987948-s54l2"
Sep 15 12:28:52.046531 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:52.046497    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5485dc6-da0b-4a35-a7ae-c7047dce4faa-tmp-dir\") pod \"a5485dc6-da0b-4a35-a7ae-c7047dce4faa\" (UID: \"a5485dc6-da0b-4a35-a7ae-c7047dce4faa\") "
Sep 15 12:28:52.046723 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:52.046540    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fff9c4da-fbed-47c9-8f8a-b254c4489af3-model-cache\") pod \"fff9c4da-fbed-47c9-8f8a-b254c4489af3\" (UID: \"fff9c4da-fbed-47c9-8f8a-b254c4489af3\") "
Sep 15 12:28:52.046723 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:52.046560    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49d7abed-95ea-4dc2-b066-2ea0ddfdff52-home\") pod \"49d7abed-95ea-4dc2-b066-2ea0ddfdff52\" (UID: \"49d7abed-95ea-4dc2-b066-2ea0ddfdff52\") "
Sep 15 12:28:52.046723 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:52.046577    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49d7abed-95ea-4dc2-b066-2ea0ddfdff52-dshm\") pod \"49d7abed-95ea-4dc2-b066-2ea0ddfdff52\" (UID: \"49d7abed-95ea-4dc2-b066-2ea0ddfdff52\") "
Sep 15 12:28:52.046723 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:52.046602    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49d7abed-95ea-4dc2-b066-2ea0ddfdff52-tls-certs\") pod \"49d7abed-95ea-4dc2-b066-2ea0ddfdff52\" (UID: \"49d7abed-95ea-4dc2-b066-2ea0ddfdff52\") "
Sep 15 12:28:52.046723 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:52.046625    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fff9c4da-fbed-47c9-8f8a-b254c4489af3-tmp-dir\") pod \"fff9c4da-fbed-47c9-8f8a-b254c4489af3\" (UID: \"fff9c4da-fbed-47c9-8f8a-b254c4489af3\") "
Sep 15 12:28:52.046723 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:52.046665    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5485dc6-da0b-4a35-a7ae-c7047dce4faa-model-cache\") pod \"a5485dc6-da0b-4a35-a7ae-c7047dce4faa\" (UID: \"a5485dc6-da0b-4a35-a7ae-c7047dce4faa\") "
Sep 15 12:28:52.046723 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:52.046694    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fff9c4da-fbed-47c9-8f8a-b254c4489af3-dshm\") pod \"fff9c4da-fbed-47c9-8f8a-b254c4489af3\" (UID: \"fff9c4da-fbed-47c9-8f8a-b254c4489af3\") "
Sep 15 12:28:52.047099 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:52.046728    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d8t2c\" (UniqueName: \"kubernetes.io/projected/fff9c4da-fbed-47c9-8f8a-b254c4489af3-kube-api-access-d8t2c\") pod \"fff9c4da-fbed-47c9-8f8a-b254c4489af3\" (UID: \"fff9c4da-fbed-47c9-8f8a-b254c4489af3\") "
Sep 15 12:28:52.047099 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:52.046755    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pfz99\" (UniqueName: \"kubernetes.io/projected/a5485dc6-da0b-4a35-a7ae-c7047dce4faa-kube-api-access-pfz99\") pod \"a5485dc6-da0b-4a35-a7ae-c7047dce4faa\" (UID: \"a5485dc6-da0b-4a35-a7ae-c7047dce4faa\") "
Sep 15 12:28:52.047099 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:52.046804    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5485dc6-da0b-4a35-a7ae-c7047dce4faa-dshm\") pod \"a5485dc6-da0b-4a35-a7ae-c7047dce4faa\" (UID: \"a5485dc6-da0b-4a35-a7ae-c7047dce4faa\") "
Sep 15 12:28:52.047099 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:52.046829    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fff9c4da-fbed-47c9-8f8a-b254c4489af3-home\") pod \"fff9c4da-fbed-47c9-8f8a-b254c4489af3\" (UID: \"fff9c4da-fbed-47c9-8f8a-b254c4489af3\") "
Sep 15 12:28:52.047099 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:52.046838    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5485dc6-da0b-4a35-a7ae-c7047dce4faa-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a5485dc6-da0b-4a35-a7ae-c7047dce4faa" (UID: "a5485dc6-da0b-4a35-a7ae-c7047dce4faa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:28:52.047099 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:52.046858    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5485dc6-da0b-4a35-a7ae-c7047dce4faa-tls-certs\") pod \"a5485dc6-da0b-4a35-a7ae-c7047dce4faa\" (UID: \"a5485dc6-da0b-4a35-a7ae-c7047dce4faa\") "
Sep 15 12:28:52.047099 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:52.046844    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fff9c4da-fbed-47c9-8f8a-b254c4489af3-model-cache" (OuterVolumeSpecName: "model-cache") pod "fff9c4da-fbed-47c9-8f8a-b254c4489af3" (UID: "fff9c4da-fbed-47c9-8f8a-b254c4489af3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:28:52.047099 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:52.046900    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h97t4\" (UniqueName: \"kubernetes.io/projected/49d7abed-95ea-4dc2-b066-2ea0ddfdff52-kube-api-access-h97t4\") pod \"49d7abed-95ea-4dc2-b066-2ea0ddfdff52\" (UID: \"49d7abed-95ea-4dc2-b066-2ea0ddfdff52\") "
Sep 15 12:28:52.047099 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:52.046947    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49d7abed-95ea-4dc2-b066-2ea0ddfdff52-model-cache\") pod \"49d7abed-95ea-4dc2-b066-2ea0ddfdff52\" (UID: \"49d7abed-95ea-4dc2-b066-2ea0ddfdff52\") "
Sep 15 12:28:52.047099 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:52.046973    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49d7abed-95ea-4dc2-b066-2ea0ddfdff52-tmp-dir\") pod \"49d7abed-95ea-4dc2-b066-2ea0ddfdff52\" (UID: \"49d7abed-95ea-4dc2-b066-2ea0ddfdff52\") "
Sep 15 12:28:52.047099 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:52.046973    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5485dc6-da0b-4a35-a7ae-c7047dce4faa-model-cache" (OuterVolumeSpecName: "model-cache") pod "a5485dc6-da0b-4a35-a7ae-c7047dce4faa" (UID: "a5485dc6-da0b-4a35-a7ae-c7047dce4faa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:28:52.047099 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:52.047001    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fff9c4da-fbed-47c9-8f8a-b254c4489af3-tls-certs\") pod \"fff9c4da-fbed-47c9-8f8a-b254c4489af3\" (UID: \"fff9c4da-fbed-47c9-8f8a-b254c4489af3\") "
Sep 15 12:28:52.047099 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:52.047032    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5485dc6-da0b-4a35-a7ae-c7047dce4faa-home\") pod \"a5485dc6-da0b-4a35-a7ae-c7047dce4faa\" (UID: \"a5485dc6-da0b-4a35-a7ae-c7047dce4faa\") "
Sep 15 12:28:52.047804 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:52.047161    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fff9c4da-fbed-47c9-8f8a-b254c4489af3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fff9c4da-fbed-47c9-8f8a-b254c4489af3" (UID: "fff9c4da-fbed-47c9-8f8a-b254c4489af3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:28:52.047804 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:52.047310    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fff9c4da-fbed-47c9-8f8a-b254c4489af3-home" (OuterVolumeSpecName: "home") pod "fff9c4da-fbed-47c9-8f8a-b254c4489af3" (UID: "fff9c4da-fbed-47c9-8f8a-b254c4489af3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:28:52.047804 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:52.047420    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e54c8cbe-3dad-4a5e-95d3-feeec2887d82-tmp-dir\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:28:52.047804 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:52.047439    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5485dc6-da0b-4a35-a7ae-c7047dce4faa-tmp-dir\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:28:52.047804 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:52.047451    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e54c8cbe-3dad-4a5e-95d3-feeec2887d82-dshm\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:28:52.047804 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:52.047463    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fff9c4da-fbed-47c9-8f8a-b254c4489af3-model-cache\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:28:52.047804 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:52.047475    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e54c8cbe-3dad-4a5e-95d3-feeec2887d82-model-cache\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:28:52.047804 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:52.047486    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fff9c4da-fbed-47c9-8f8a-b254c4489af3-tmp-dir\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:28:52.047804 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:52.047500    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5485dc6-da0b-4a35-a7ae-c7047dce4faa-model-cache\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:28:52.047804 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:52.047503    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49d7abed-95ea-4dc2-b066-2ea0ddfdff52-model-cache" (OuterVolumeSpecName: "model-cache") pod "49d7abed-95ea-4dc2-b066-2ea0ddfdff52" (UID: "49d7abed-95ea-4dc2-b066-2ea0ddfdff52"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:28:52.047804 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:52.047514    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lz64v\" (UniqueName: \"kubernetes.io/projected/e54c8cbe-3dad-4a5e-95d3-feeec2887d82-kube-api-access-lz64v\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:28:52.047804 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:52.047529    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e54c8cbe-3dad-4a5e-95d3-feeec2887d82-tls-certs\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:28:52.047804 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:52.047545    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e54c8cbe-3dad-4a5e-95d3-feeec2887d82-home\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:28:52.047804 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:52.047555    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fff9c4da-fbed-47c9-8f8a-b254c4489af3-home\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:28:52.048681 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:52.047828    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49d7abed-95ea-4dc2-b066-2ea0ddfdff52-home" (OuterVolumeSpecName: "home") pod "49d7abed-95ea-4dc2-b066-2ea0ddfdff52" (UID: "49d7abed-95ea-4dc2-b066-2ea0ddfdff52"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:28:52.048826 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:52.048784    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49d7abed-95ea-4dc2-b066-2ea0ddfdff52-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "49d7abed-95ea-4dc2-b066-2ea0ddfdff52" (UID: "49d7abed-95ea-4dc2-b066-2ea0ddfdff52"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:28:52.049039 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:52.049017    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5485dc6-da0b-4a35-a7ae-c7047dce4faa-home" (OuterVolumeSpecName: "home") pod "a5485dc6-da0b-4a35-a7ae-c7047dce4faa" (UID: "a5485dc6-da0b-4a35-a7ae-c7047dce4faa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:28:52.050394 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:52.050367    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5485dc6-da0b-4a35-a7ae-c7047dce4faa-dshm" (OuterVolumeSpecName: "dshm") pod "a5485dc6-da0b-4a35-a7ae-c7047dce4faa" (UID: "a5485dc6-da0b-4a35-a7ae-c7047dce4faa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:28:52.050500 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:52.050437    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49d7abed-95ea-4dc2-b066-2ea0ddfdff52-dshm" (OuterVolumeSpecName: "dshm") pod "49d7abed-95ea-4dc2-b066-2ea0ddfdff52" (UID: "49d7abed-95ea-4dc2-b066-2ea0ddfdff52"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:28:52.050744 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:52.050706    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5485dc6-da0b-4a35-a7ae-c7047dce4faa-kube-api-access-pfz99" (OuterVolumeSpecName: "kube-api-access-pfz99") pod "a5485dc6-da0b-4a35-a7ae-c7047dce4faa" (UID: "a5485dc6-da0b-4a35-a7ae-c7047dce4faa"). InnerVolumeSpecName "kube-api-access-pfz99". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:28:52.051006 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:52.050982    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5485dc6-da0b-4a35-a7ae-c7047dce4faa-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a5485dc6-da0b-4a35-a7ae-c7047dce4faa" (UID: "a5485dc6-da0b-4a35-a7ae-c7047dce4faa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:28:52.051446 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:52.051339    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fff9c4da-fbed-47c9-8f8a-b254c4489af3-dshm" (OuterVolumeSpecName: "dshm") pod "fff9c4da-fbed-47c9-8f8a-b254c4489af3" (UID: "fff9c4da-fbed-47c9-8f8a-b254c4489af3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:28:52.051446 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:52.051355    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49d7abed-95ea-4dc2-b066-2ea0ddfdff52-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "49d7abed-95ea-4dc2-b066-2ea0ddfdff52" (UID: "49d7abed-95ea-4dc2-b066-2ea0ddfdff52"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:28:52.051446 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:52.051424    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fff9c4da-fbed-47c9-8f8a-b254c4489af3-kube-api-access-d8t2c" (OuterVolumeSpecName: "kube-api-access-d8t2c") pod "fff9c4da-fbed-47c9-8f8a-b254c4489af3" (UID: "fff9c4da-fbed-47c9-8f8a-b254c4489af3"). InnerVolumeSpecName "kube-api-access-d8t2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:28:52.051779 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:52.051758    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fff9c4da-fbed-47c9-8f8a-b254c4489af3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fff9c4da-fbed-47c9-8f8a-b254c4489af3" (UID: "fff9c4da-fbed-47c9-8f8a-b254c4489af3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:28:52.052246 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:52.052224    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49d7abed-95ea-4dc2-b066-2ea0ddfdff52-kube-api-access-h97t4" (OuterVolumeSpecName: "kube-api-access-h97t4") pod "49d7abed-95ea-4dc2-b066-2ea0ddfdff52" (UID: "49d7abed-95ea-4dc2-b066-2ea0ddfdff52"). InnerVolumeSpecName "kube-api-access-h97t4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:28:52.148029 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:52.147998    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5485dc6-da0b-4a35-a7ae-c7047dce4faa-dshm\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:28:52.148029 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:52.148025    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5485dc6-da0b-4a35-a7ae-c7047dce4faa-tls-certs\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:28:52.148029 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:52.148037    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h97t4\" (UniqueName: \"kubernetes.io/projected/49d7abed-95ea-4dc2-b066-2ea0ddfdff52-kube-api-access-h97t4\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:28:52.148313 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:52.148045    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49d7abed-95ea-4dc2-b066-2ea0ddfdff52-model-cache\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:28:52.148313 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:52.148055    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49d7abed-95ea-4dc2-b066-2ea0ddfdff52-tmp-dir\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:28:52.148313 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:52.148063    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fff9c4da-fbed-47c9-8f8a-b254c4489af3-tls-certs\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:28:52.148313 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:52.148070    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5485dc6-da0b-4a35-a7ae-c7047dce4faa-home\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:28:52.148313 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:52.148077    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49d7abed-95ea-4dc2-b066-2ea0ddfdff52-home\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:28:52.148313 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:52.148085    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49d7abed-95ea-4dc2-b066-2ea0ddfdff52-dshm\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:28:52.148313 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:52.148093    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49d7abed-95ea-4dc2-b066-2ea0ddfdff52-tls-certs\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:28:52.148313 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:52.148100    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fff9c4da-fbed-47c9-8f8a-b254c4489af3-dshm\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:28:52.148313 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:52.148107    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8t2c\" (UniqueName: \"kubernetes.io/projected/fff9c4da-fbed-47c9-8f8a-b254c4489af3-kube-api-access-d8t2c\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:28:52.148313 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:52.148115    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfz99\" (UniqueName: \"kubernetes.io/projected/a5485dc6-da0b-4a35-a7ae-c7047dce4faa-kube-api-access-pfz99\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:28:52.536517 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:52.536421    2555 generic.go:356] "Generic (PLEG): container finished" podID="e54c8cbe-3dad-4a5e-95d3-feeec2887d82" containerID="9d20018093f0649e88ee94be1fc03948ccf0ef9f0ee73cdf95d92cbf429b4b1f" exitCode=0
Sep 15 12:28:52.536517 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:52.536487    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9ba48f/premium-w4-9ba48f-kserve-5746585544-8rnhs"
Sep 15 12:28:52.536728 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:52.536512    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9ba48f/premium-w4-9ba48f-kserve-5746585544-8rnhs" event={"ID":"e54c8cbe-3dad-4a5e-95d3-feeec2887d82","Type":"ContainerDied","Data":"9d20018093f0649e88ee94be1fc03948ccf0ef9f0ee73cdf95d92cbf429b4b1f"}
Sep 15 12:28:52.536728 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:52.536569    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9ba48f/premium-w4-9ba48f-kserve-5746585544-8rnhs" event={"ID":"e54c8cbe-3dad-4a5e-95d3-feeec2887d82","Type":"ContainerDied","Data":"a9fcbca4856c2486126e9d64317bbdfcc228083f2e4c039390063bc39b208f16"}
Sep 15 12:28:52.536728 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:52.536596    2555 scope.go:117] "RemoveContainer" containerID="9d20018093f0649e88ee94be1fc03948ccf0ef9f0ee73cdf95d92cbf429b4b1f"
Sep 15 12:28:52.537990 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:52.537966    2555 generic.go:356] "Generic (PLEG): container finished" podID="49d7abed-95ea-4dc2-b066-2ea0ddfdff52" containerID="4027c5e63a56bf22e8f49c36d1337fbd5d3182116180c0261a7e9fd46fbe86c8" exitCode=0
Sep 15 12:28:52.538106 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:52.538034    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9ba48f/distinct-2-w4-9ba48f-kserve-7fd5987948-s54l2"
Sep 15 12:28:52.538106 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:52.538057    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9ba48f/distinct-2-w4-9ba48f-kserve-7fd5987948-s54l2" event={"ID":"49d7abed-95ea-4dc2-b066-2ea0ddfdff52","Type":"ContainerDied","Data":"4027c5e63a56bf22e8f49c36d1337fbd5d3182116180c0261a7e9fd46fbe86c8"}
Sep 15 12:28:52.538106 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:52.538096    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9ba48f/distinct-2-w4-9ba48f-kserve-7fd5987948-s54l2" event={"ID":"49d7abed-95ea-4dc2-b066-2ea0ddfdff52","Type":"ContainerDied","Data":"b0d394bb6f8701aa731c950e8bbbfcfca349c5de5a39a25c512aaa21cbd9c98e"}
Sep 15 12:28:52.539435 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:52.539411    2555 generic.go:356] "Generic (PLEG): container finished" podID="a5485dc6-da0b-4a35-a7ae-c7047dce4faa" containerID="dcc848dc24bf022799722a25b0f69d9deb970514cae6f5ae9e3d67d889185a48" exitCode=0
Sep 15 12:28:52.539838 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:52.539623    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9ba48f/simulator-w4-9ba48f-kserve-67d55d8789-vvwbq"
Sep 15 12:28:52.539838 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:52.539573    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9ba48f/simulator-w4-9ba48f-kserve-67d55d8789-vvwbq" event={"ID":"a5485dc6-da0b-4a35-a7ae-c7047dce4faa","Type":"ContainerDied","Data":"dcc848dc24bf022799722a25b0f69d9deb970514cae6f5ae9e3d67d889185a48"}
Sep 15 12:28:52.540002 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:52.539872    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9ba48f/simulator-w4-9ba48f-kserve-67d55d8789-vvwbq" event={"ID":"a5485dc6-da0b-4a35-a7ae-c7047dce4faa","Type":"ContainerDied","Data":"97c5f22bbc5bc2e2bdb47202389cf8f7e4e3aa7ec9fbe407a29962b4d3d1bffb"}
Sep 15 12:28:52.541295 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:52.541273    2555 generic.go:356] "Generic (PLEG): container finished" podID="fff9c4da-fbed-47c9-8f8a-b254c4489af3" containerID="f1adb2977a7274f3da7731a46115b2fe0741033c3d5ac7aafddc0fd6279b3edb" exitCode=0
Sep 15 12:28:52.541417 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:52.541327    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9ba48f/distinct-w4-9ba48f-kserve-cf9cd67f-7hsfr"
Sep 15 12:28:52.541417 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:52.541346    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9ba48f/distinct-w4-9ba48f-kserve-cf9cd67f-7hsfr" event={"ID":"fff9c4da-fbed-47c9-8f8a-b254c4489af3","Type":"ContainerDied","Data":"f1adb2977a7274f3da7731a46115b2fe0741033c3d5ac7aafddc0fd6279b3edb"}
Sep 15 12:28:52.541417 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:52.541373    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9ba48f/distinct-w4-9ba48f-kserve-cf9cd67f-7hsfr" event={"ID":"fff9c4da-fbed-47c9-8f8a-b254c4489af3","Type":"ContainerDied","Data":"04b3a554cef1fb6cdd6393b0b6e282e75719876d6d9130a44be8c0c1d2c5d486"}
Sep 15 12:28:52.548581 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:52.548540    2555 scope.go:117] "RemoveContainer" containerID="9d20018093f0649e88ee94be1fc03948ccf0ef9f0ee73cdf95d92cbf429b4b1f"
Sep 15 12:28:52.550098 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:28:52.550075    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d20018093f0649e88ee94be1fc03948ccf0ef9f0ee73cdf95d92cbf429b4b1f\": container with ID starting with 9d20018093f0649e88ee94be1fc03948ccf0ef9f0ee73cdf95d92cbf429b4b1f not found: ID does not exist" containerID="9d20018093f0649e88ee94be1fc03948ccf0ef9f0ee73cdf95d92cbf429b4b1f"
Sep 15 12:28:52.550201 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:52.550111    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d20018093f0649e88ee94be1fc03948ccf0ef9f0ee73cdf95d92cbf429b4b1f"} err="failed to get container status \"9d20018093f0649e88ee94be1fc03948ccf0ef9f0ee73cdf95d92cbf429b4b1f\": rpc error: code = NotFound desc = could not find container \"9d20018093f0649e88ee94be1fc03948ccf0ef9f0ee73cdf95d92cbf429b4b1f\": container with ID starting with 9d20018093f0649e88ee94be1fc03948ccf0ef9f0ee73cdf95d92cbf429b4b1f not found: ID does not exist"
Sep 15 12:28:52.550201 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:52.550135    2555 scope.go:117] "RemoveContainer" containerID="4027c5e63a56bf22e8f49c36d1337fbd5d3182116180c0261a7e9fd46fbe86c8"
Sep 15 12:28:52.559557 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:52.559538    2555 scope.go:117] "RemoveContainer" containerID="4027c5e63a56bf22e8f49c36d1337fbd5d3182116180c0261a7e9fd46fbe86c8"
Sep 15 12:28:52.559851 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:28:52.559818    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4027c5e63a56bf22e8f49c36d1337fbd5d3182116180c0261a7e9fd46fbe86c8\": container with ID starting with 4027c5e63a56bf22e8f49c36d1337fbd5d3182116180c0261a7e9fd46fbe86c8 not found: ID does not exist" containerID="4027c5e63a56bf22e8f49c36d1337fbd5d3182116180c0261a7e9fd46fbe86c8"
Sep 15 12:28:52.559916 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:52.559857    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4027c5e63a56bf22e8f49c36d1337fbd5d3182116180c0261a7e9fd46fbe86c8"} err="failed to get container status \"4027c5e63a56bf22e8f49c36d1337fbd5d3182116180c0261a7e9fd46fbe86c8\": rpc error: code = NotFound desc = could not find container \"4027c5e63a56bf22e8f49c36d1337fbd5d3182116180c0261a7e9fd46fbe86c8\": container with ID starting with 4027c5e63a56bf22e8f49c36d1337fbd5d3182116180c0261a7e9fd46fbe86c8 not found: ID does not exist"
Sep 15 12:28:52.559916 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:52.559874    2555 scope.go:117] "RemoveContainer" containerID="dcc848dc24bf022799722a25b0f69d9deb970514cae6f5ae9e3d67d889185a48"
Sep 15 12:28:52.568203 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:52.568186    2555 scope.go:117] "RemoveContainer" containerID="dcc848dc24bf022799722a25b0f69d9deb970514cae6f5ae9e3d67d889185a48"
Sep 15 12:28:52.568483 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:28:52.568462    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcc848dc24bf022799722a25b0f69d9deb970514cae6f5ae9e3d67d889185a48\": container with ID starting with dcc848dc24bf022799722a25b0f69d9deb970514cae6f5ae9e3d67d889185a48 not found: ID does not exist" containerID="dcc848dc24bf022799722a25b0f69d9deb970514cae6f5ae9e3d67d889185a48"
Sep 15 12:28:52.568542 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:52.568490    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcc848dc24bf022799722a25b0f69d9deb970514cae6f5ae9e3d67d889185a48"} err="failed to get container status \"dcc848dc24bf022799722a25b0f69d9deb970514cae6f5ae9e3d67d889185a48\": rpc error: code = NotFound desc = could not find container \"dcc848dc24bf022799722a25b0f69d9deb970514cae6f5ae9e3d67d889185a48\": container with ID starting with dcc848dc24bf022799722a25b0f69d9deb970514cae6f5ae9e3d67d889185a48 not found: ID does not exist"
Sep 15 12:28:52.568542 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:52.568508    2555 scope.go:117] "RemoveContainer" containerID="f1adb2977a7274f3da7731a46115b2fe0741033c3d5ac7aafddc0fd6279b3edb"
Sep 15 12:28:52.579385 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:52.579357    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-9ba48f/simulator-w4-9ba48f-kserve-67d55d8789-vvwbq"]
Sep 15 12:28:52.582212 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:52.582188    2555 scope.go:117] "RemoveContainer" containerID="f1adb2977a7274f3da7731a46115b2fe0741033c3d5ac7aafddc0fd6279b3edb"
Sep 15 12:28:52.582628 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:28:52.582607    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1adb2977a7274f3da7731a46115b2fe0741033c3d5ac7aafddc0fd6279b3edb\": container with ID starting with f1adb2977a7274f3da7731a46115b2fe0741033c3d5ac7aafddc0fd6279b3edb not found: ID does not exist" containerID="f1adb2977a7274f3da7731a46115b2fe0741033c3d5ac7aafddc0fd6279b3edb"
Sep 15 12:28:52.582735 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:52.582634    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1adb2977a7274f3da7731a46115b2fe0741033c3d5ac7aafddc0fd6279b3edb"} err="failed to get container status \"f1adb2977a7274f3da7731a46115b2fe0741033c3d5ac7aafddc0fd6279b3edb\": rpc error: code = NotFound desc = could not find container \"f1adb2977a7274f3da7731a46115b2fe0741033c3d5ac7aafddc0fd6279b3edb\": container with ID starting with f1adb2977a7274f3da7731a46115b2fe0741033c3d5ac7aafddc0fd6279b3edb not found: ID does not exist"
Sep 15 12:28:52.583339 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:52.583315    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-9ba48f/simulator-w4-9ba48f-kserve-67d55d8789-vvwbq"]
Sep 15 12:28:52.598543 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:52.598510    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-9ba48f/distinct-w4-9ba48f-kserve-cf9cd67f-7hsfr"]
Sep 15 12:28:52.601466 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:52.601444    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-9ba48f/distinct-w4-9ba48f-kserve-cf9cd67f-7hsfr"]
Sep 15 12:28:52.615938 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:52.615912    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-9ba48f/distinct-2-w4-9ba48f-kserve-7fd5987948-s54l2"]
Sep 15 12:28:52.628911 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:52.628885    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-9ba48f/distinct-2-w4-9ba48f-kserve-7fd5987948-s54l2"]
Sep 15 12:28:52.646788 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:52.646752    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-9ba48f/premium-w4-9ba48f-kserve-5746585544-8rnhs"]
Sep 15 12:28:52.649577 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:52.649553    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-9ba48f/premium-w4-9ba48f-kserve-5746585544-8rnhs"]
Sep 15 12:28:53.817316 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:53.817284    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49d7abed-95ea-4dc2-b066-2ea0ddfdff52" path="/var/lib/kubelet/pods/49d7abed-95ea-4dc2-b066-2ea0ddfdff52/volumes"
Sep 15 12:28:53.817690 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:53.817668    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5485dc6-da0b-4a35-a7ae-c7047dce4faa" path="/var/lib/kubelet/pods/a5485dc6-da0b-4a35-a7ae-c7047dce4faa/volumes"
Sep 15 12:28:53.818009 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:53.817996    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e54c8cbe-3dad-4a5e-95d3-feeec2887d82" path="/var/lib/kubelet/pods/e54c8cbe-3dad-4a5e-95d3-feeec2887d82/volumes"
Sep 15 12:28:53.818692 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:53.818676    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fff9c4da-fbed-47c9-8f8a-b254c4489af3" path="/var/lib/kubelet/pods/fff9c4da-fbed-47c9-8f8a-b254c4489af3/volumes"
Sep 15 12:28:59.415630 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:28:59.415595    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5gkxs"]
Sep 15 12:29:14.458748 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:14.458712    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5gkxs"]
Sep 15 12:29:24.124218 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:24.124186    2555 scope.go:117] "RemoveContainer" containerID="8471f1a08b78e7718094171000300434b72cc7f4b9c4a2ffb901fbd4c4957d6d"
Sep 15 12:29:27.558093 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:27.558048    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5gkxs"]
Sep 15 12:29:36.986247 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:36.986209    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7941-9f90d5"]
Sep 15 12:29:36.986725 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:36.986583    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0770ba7c-72be-4fb5-bf05-bef6114ce762" containerName="payload-pre-processing"
Sep 15 12:29:36.986725 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:36.986595    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="0770ba7c-72be-4fb5-bf05-bef6114ce762" containerName="payload-pre-processing"
Sep 15 12:29:36.986725 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:36.986604    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2669f23f-b68d-45a9-99b7-f5ee66e4042a" containerName="revoke-keys"
Sep 15 12:29:36.986725 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:36.986610    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="2669f23f-b68d-45a9-99b7-f5ee66e4042a" containerName="revoke-keys"
Sep 15 12:29:36.986725 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:36.986618    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5485dc6-da0b-4a35-a7ae-c7047dce4faa" containerName="main"
Sep 15 12:29:36.986725 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:36.986624    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5485dc6-da0b-4a35-a7ae-c7047dce4faa" containerName="main"
Sep 15 12:29:36.986725 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:36.986631    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49d7abed-95ea-4dc2-b066-2ea0ddfdff52" containerName="main"
Sep 15 12:29:36.986725 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:36.986636    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="49d7abed-95ea-4dc2-b066-2ea0ddfdff52" containerName="main"
Sep 15 12:29:36.986725 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:36.986646    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fff9c4da-fbed-47c9-8f8a-b254c4489af3" containerName="main"
Sep 15 12:29:36.986725 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:36.986651    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="fff9c4da-fbed-47c9-8f8a-b254c4489af3" containerName="main"
Sep 15 12:29:36.986725 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:36.986663    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e54c8cbe-3dad-4a5e-95d3-feeec2887d82" containerName="main"
Sep 15 12:29:36.986725 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:36.986668    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e54c8cbe-3dad-4a5e-95d3-feeec2887d82" containerName="main"
Sep 15 12:29:36.986725 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:36.986721    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="0770ba7c-72be-4fb5-bf05-bef6114ce762" containerName="payload-pre-processing"
Sep 15 12:29:36.986725 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:36.986730    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="2669f23f-b68d-45a9-99b7-f5ee66e4042a" containerName="revoke-keys"
Sep 15 12:29:36.987214 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:36.986736    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="e54c8cbe-3dad-4a5e-95d3-feeec2887d82" containerName="main"
Sep 15 12:29:36.987214 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:36.986744    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="fff9c4da-fbed-47c9-8f8a-b254c4489af3" containerName="main"
Sep 15 12:29:36.987214 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:36.986748    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5485dc6-da0b-4a35-a7ae-c7047dce4faa" containerName="main"
Sep 15 12:29:36.987214 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:36.986755    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="49d7abed-95ea-4dc2-b066-2ea0ddfdff52" containerName="main"
Sep 15 12:29:36.989885 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:36.989868    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7941-9f90d5"
Sep 15 12:29:36.992955 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:36.992931    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-q76n5\""
Sep 15 12:29:36.995919 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:36.995897    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7941-9f90d5"]
Sep 15 12:29:37.045400 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:37.045359    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x47dd\" (UniqueName: \"kubernetes.io/projected/54cc3f54-0c35-4485-9f1f-a6b871ee3b6c-kube-api-access-x47dd\") pod \"test-curl-7941-9f90d5\" (UID: \"54cc3f54-0c35-4485-9f1f-a6b871ee3b6c\") " pod="opendatahub/test-curl-7941-9f90d5"
Sep 15 12:29:37.146344 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:37.146300    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x47dd\" (UniqueName: \"kubernetes.io/projected/54cc3f54-0c35-4485-9f1f-a6b871ee3b6c-kube-api-access-x47dd\") pod \"test-curl-7941-9f90d5\" (UID: \"54cc3f54-0c35-4485-9f1f-a6b871ee3b6c\") " pod="opendatahub/test-curl-7941-9f90d5"
Sep 15 12:29:37.154987 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:37.154951    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x47dd\" (UniqueName: \"kubernetes.io/projected/54cc3f54-0c35-4485-9f1f-a6b871ee3b6c-kube-api-access-x47dd\") pod \"test-curl-7941-9f90d5\" (UID: \"54cc3f54-0c35-4485-9f1f-a6b871ee3b6c\") " pod="opendatahub/test-curl-7941-9f90d5"
Sep 15 12:29:37.300983 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:37.300895    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7941-9f90d5"
Sep 15 12:29:37.572224 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:37.572196    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7941-9f90d5"]
Sep 15 12:29:37.574208 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:29:37.574178    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod54cc3f54_0c35_4485_9f1f_a6b871ee3b6c.slice/crio-b8e9eff4099b34726b12ea9135dde207a9ac4267ec91e0510ce70d2bb8da029a WatchSource:0}: Error finding container b8e9eff4099b34726b12ea9135dde207a9ac4267ec91e0510ce70d2bb8da029a: Status 404 returned error can't find the container with id b8e9eff4099b34726b12ea9135dde207a9ac4267ec91e0510ce70d2bb8da029a
Sep 15 12:29:37.807993 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:37.807955    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7941-9f90d5" event={"ID":"54cc3f54-0c35-4485-9f1f-a6b871ee3b6c","Type":"ContainerStarted","Data":"53f4c88e5d9d320385006e5cc7269ee7fd395c6fa8e6fb5c874e58b14a649cd8"}
Sep 15 12:29:37.807993 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:37.807996    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7941-9f90d5" event={"ID":"54cc3f54-0c35-4485-9f1f-a6b871ee3b6c","Type":"ContainerStarted","Data":"b8e9eff4099b34726b12ea9135dde207a9ac4267ec91e0510ce70d2bb8da029a"}
Sep 15 12:29:37.824995 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:37.824895    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7941-9f90d5" podStartSLOduration=1.8248801669999999 podStartE2EDuration="1.824880167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:29:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:29:37.823185859 +0000 UTC m=+1634.614108371" watchObservedRunningTime="2026-09-15 12:29:37.824880167 +0000 UTC m=+1634.615802707"
Sep 15 12:29:38.428974 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:38.428939    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7941-9f90d5"]
Sep 15 12:29:38.435385 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:38.435352    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7941-9f90d5"]
Sep 15 12:29:38.813759 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:29:38.813728    2555 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7941-9f90d5" container="test-curl-7941-9f90d5"
Sep 15 12:29:39.064132 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:39.064044    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5gkxs"]
Sep 15 12:29:39.758006 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:39.757970    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1dc430/auth-test-model-1dc430-kserve-77cd669cc5-wc628"]
Sep 15 12:29:39.818952 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:39.818913    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7941-9f90d5" podUID="54cc3f54-0c35-4485-9f1f-a6b871ee3b6c" containerName="test-curl-7941-9f90d5" containerID="cri-o://53f4c88e5d9d320385006e5cc7269ee7fd395c6fa8e6fb5c874e58b14a649cd8" gracePeriod=2
Sep 15 12:29:41.989877 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:41.989848    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7941-9f90d5"
Sep 15 12:29:42.087294 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:42.087207    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x47dd\" (UniqueName: \"kubernetes.io/projected/54cc3f54-0c35-4485-9f1f-a6b871ee3b6c-kube-api-access-x47dd\") pod \"54cc3f54-0c35-4485-9f1f-a6b871ee3b6c\" (UID: \"54cc3f54-0c35-4485-9f1f-a6b871ee3b6c\") "
Sep 15 12:29:42.089235 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:42.089202    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54cc3f54-0c35-4485-9f1f-a6b871ee3b6c-kube-api-access-x47dd" (OuterVolumeSpecName: "kube-api-access-x47dd") pod "54cc3f54-0c35-4485-9f1f-a6b871ee3b6c" (UID: "54cc3f54-0c35-4485-9f1f-a6b871ee3b6c"). InnerVolumeSpecName "kube-api-access-x47dd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:29:42.188190 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:42.188155    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x47dd\" (UniqueName: \"kubernetes.io/projected/54cc3f54-0c35-4485-9f1f-a6b871ee3b6c-kube-api-access-x47dd\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:29:42.836795 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:42.836756    2555 generic.go:356] "Generic (PLEG): container finished" podID="54cc3f54-0c35-4485-9f1f-a6b871ee3b6c" containerID="53f4c88e5d9d320385006e5cc7269ee7fd395c6fa8e6fb5c874e58b14a649cd8" exitCode=137
Sep 15 12:29:42.836977 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:42.836800    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7941-9f90d5"
Sep 15 12:29:42.836977 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:42.836851    2555 scope.go:117] "RemoveContainer" containerID="53f4c88e5d9d320385006e5cc7269ee7fd395c6fa8e6fb5c874e58b14a649cd8"
Sep 15 12:29:42.845792 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:42.845775    2555 scope.go:117] "RemoveContainer" containerID="53f4c88e5d9d320385006e5cc7269ee7fd395c6fa8e6fb5c874e58b14a649cd8"
Sep 15 12:29:42.846043 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:29:42.846021    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53f4c88e5d9d320385006e5cc7269ee7fd395c6fa8e6fb5c874e58b14a649cd8\": container with ID starting with 53f4c88e5d9d320385006e5cc7269ee7fd395c6fa8e6fb5c874e58b14a649cd8 not found: ID does not exist" containerID="53f4c88e5d9d320385006e5cc7269ee7fd395c6fa8e6fb5c874e58b14a649cd8"
Sep 15 12:29:42.846103 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:42.846054    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53f4c88e5d9d320385006e5cc7269ee7fd395c6fa8e6fb5c874e58b14a649cd8"} err="failed to get container status \"53f4c88e5d9d320385006e5cc7269ee7fd395c6fa8e6fb5c874e58b14a649cd8\": rpc error: code = NotFound desc = could not find container \"53f4c88e5d9d320385006e5cc7269ee7fd395c6fa8e6fb5c874e58b14a649cd8\": container with ID starting with 53f4c88e5d9d320385006e5cc7269ee7fd395c6fa8e6fb5c874e58b14a649cd8 not found: ID does not exist"
Sep 15 12:29:42.846937 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:29:42.846916    2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7941-9f90d5\" is forbidden: User \"system:node:ip-10-0-140-17.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-17.ec2.internal' and this object" podUID="54cc3f54-0c35-4485-9f1f-a6b871ee3b6c" pod="opendatahub/test-curl-7941-9f90d5"
Sep 15 12:29:43.816670 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:29:43.816632    2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7941-9f90d5\" is forbidden: User \"system:node:ip-10-0-140-17.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-17.ec2.internal' and this object" podUID="54cc3f54-0c35-4485-9f1f-a6b871ee3b6c" pod="opendatahub/test-curl-7941-9f90d5"
Sep 15 12:29:43.818210 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:43.818181    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54cc3f54-0c35-4485-9f1f-a6b871ee3b6c" path="/var/lib/kubelet/pods/54cc3f54-0c35-4485-9f1f-a6b871ee3b6c/volumes"
Sep 15 12:29:47.163682 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:47.163644    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5gkxs"]
Sep 15 12:29:48.424859 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:48.424824    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-1dc430/sub-test-model-1dc430-kserve-79cccdffd9-xg97q"]
Sep 15 12:29:48.425224 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:48.425178    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54cc3f54-0c35-4485-9f1f-a6b871ee3b6c" containerName="test-curl-7941-9f90d5"
Sep 15 12:29:48.425224 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:48.425190    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="54cc3f54-0c35-4485-9f1f-a6b871ee3b6c" containerName="test-curl-7941-9f90d5"
Sep 15 12:29:48.425374 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:48.425256    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="54cc3f54-0c35-4485-9f1f-a6b871ee3b6c" containerName="test-curl-7941-9f90d5"
Sep 15 12:29:48.429803 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:48.429782    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1dc430/sub-test-model-1dc430-kserve-79cccdffd9-xg97q"
Sep 15 12:29:48.433347 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:48.433324    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-1dc430\"/\"sub-test-model-1dc430-kserve-self-signed-certs\""
Sep 15 12:29:48.454206 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:48.454174    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1dc430/sub-test-model-1dc430-kserve-79cccdffd9-xg97q"]
Sep 15 12:29:48.549451 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:48.549416    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a042f16d-ba98-47c0-8f0e-cecb58dad951-home\") pod \"sub-test-model-1dc430-kserve-79cccdffd9-xg97q\" (UID: \"a042f16d-ba98-47c0-8f0e-cecb58dad951\") " pod="ai-tenant-e2e-shared-a-w3-1dc430/sub-test-model-1dc430-kserve-79cccdffd9-xg97q"
Sep 15 12:29:48.549637 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:48.549462    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a042f16d-ba98-47c0-8f0e-cecb58dad951-model-cache\") pod \"sub-test-model-1dc430-kserve-79cccdffd9-xg97q\" (UID: \"a042f16d-ba98-47c0-8f0e-cecb58dad951\") " pod="ai-tenant-e2e-shared-a-w3-1dc430/sub-test-model-1dc430-kserve-79cccdffd9-xg97q"
Sep 15 12:29:48.549637 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:48.549484    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a042f16d-ba98-47c0-8f0e-cecb58dad951-tls-certs\") pod \"sub-test-model-1dc430-kserve-79cccdffd9-xg97q\" (UID: \"a042f16d-ba98-47c0-8f0e-cecb58dad951\") " pod="ai-tenant-e2e-shared-a-w3-1dc430/sub-test-model-1dc430-kserve-79cccdffd9-xg97q"
Sep 15 12:29:48.549637 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:48.549573    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a042f16d-ba98-47c0-8f0e-cecb58dad951-tmp-dir\") pod \"sub-test-model-1dc430-kserve-79cccdffd9-xg97q\" (UID: \"a042f16d-ba98-47c0-8f0e-cecb58dad951\") " pod="ai-tenant-e2e-shared-a-w3-1dc430/sub-test-model-1dc430-kserve-79cccdffd9-xg97q"
Sep 15 12:29:48.549637 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:48.549621    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a042f16d-ba98-47c0-8f0e-cecb58dad951-dshm\") pod \"sub-test-model-1dc430-kserve-79cccdffd9-xg97q\" (UID: \"a042f16d-ba98-47c0-8f0e-cecb58dad951\") " pod="ai-tenant-e2e-shared-a-w3-1dc430/sub-test-model-1dc430-kserve-79cccdffd9-xg97q"
Sep 15 12:29:48.549823 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:48.549654    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pj45c\" (UniqueName: \"kubernetes.io/projected/a042f16d-ba98-47c0-8f0e-cecb58dad951-kube-api-access-pj45c\") pod \"sub-test-model-1dc430-kserve-79cccdffd9-xg97q\" (UID: \"a042f16d-ba98-47c0-8f0e-cecb58dad951\") " pod="ai-tenant-e2e-shared-a-w3-1dc430/sub-test-model-1dc430-kserve-79cccdffd9-xg97q"
Sep 15 12:29:48.650868 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:48.650832    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a042f16d-ba98-47c0-8f0e-cecb58dad951-home\") pod \"sub-test-model-1dc430-kserve-79cccdffd9-xg97q\" (UID: \"a042f16d-ba98-47c0-8f0e-cecb58dad951\") " pod="ai-tenant-e2e-shared-a-w3-1dc430/sub-test-model-1dc430-kserve-79cccdffd9-xg97q"
Sep 15 12:29:48.651073 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:48.650877    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a042f16d-ba98-47c0-8f0e-cecb58dad951-model-cache\") pod \"sub-test-model-1dc430-kserve-79cccdffd9-xg97q\" (UID: \"a042f16d-ba98-47c0-8f0e-cecb58dad951\") " pod="ai-tenant-e2e-shared-a-w3-1dc430/sub-test-model-1dc430-kserve-79cccdffd9-xg97q"
Sep 15 12:29:48.651073 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:48.650897    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a042f16d-ba98-47c0-8f0e-cecb58dad951-tls-certs\") pod \"sub-test-model-1dc430-kserve-79cccdffd9-xg97q\" (UID: \"a042f16d-ba98-47c0-8f0e-cecb58dad951\") " pod="ai-tenant-e2e-shared-a-w3-1dc430/sub-test-model-1dc430-kserve-79cccdffd9-xg97q"
Sep 15 12:29:48.651073 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:48.650916    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a042f16d-ba98-47c0-8f0e-cecb58dad951-tmp-dir\") pod \"sub-test-model-1dc430-kserve-79cccdffd9-xg97q\" (UID: \"a042f16d-ba98-47c0-8f0e-cecb58dad951\") " pod="ai-tenant-e2e-shared-a-w3-1dc430/sub-test-model-1dc430-kserve-79cccdffd9-xg97q"
Sep 15 12:29:48.651073 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:48.650946    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a042f16d-ba98-47c0-8f0e-cecb58dad951-dshm\") pod \"sub-test-model-1dc430-kserve-79cccdffd9-xg97q\" (UID: \"a042f16d-ba98-47c0-8f0e-cecb58dad951\") " pod="ai-tenant-e2e-shared-a-w3-1dc430/sub-test-model-1dc430-kserve-79cccdffd9-xg97q"
Sep 15 12:29:48.651073 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:48.650986    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pj45c\" (UniqueName: \"kubernetes.io/projected/a042f16d-ba98-47c0-8f0e-cecb58dad951-kube-api-access-pj45c\") pod \"sub-test-model-1dc430-kserve-79cccdffd9-xg97q\" (UID: \"a042f16d-ba98-47c0-8f0e-cecb58dad951\") " pod="ai-tenant-e2e-shared-a-w3-1dc430/sub-test-model-1dc430-kserve-79cccdffd9-xg97q"
Sep 15 12:29:48.651391 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:48.651329    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a042f16d-ba98-47c0-8f0e-cecb58dad951-home\") pod \"sub-test-model-1dc430-kserve-79cccdffd9-xg97q\" (UID: \"a042f16d-ba98-47c0-8f0e-cecb58dad951\") " pod="ai-tenant-e2e-shared-a-w3-1dc430/sub-test-model-1dc430-kserve-79cccdffd9-xg97q"
Sep 15 12:29:48.651599 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:48.651568    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a042f16d-ba98-47c0-8f0e-cecb58dad951-tmp-dir\") pod \"sub-test-model-1dc430-kserve-79cccdffd9-xg97q\" (UID: \"a042f16d-ba98-47c0-8f0e-cecb58dad951\") " pod="ai-tenant-e2e-shared-a-w3-1dc430/sub-test-model-1dc430-kserve-79cccdffd9-xg97q"
Sep 15 12:29:48.651599 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:48.651583    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a042f16d-ba98-47c0-8f0e-cecb58dad951-model-cache\") pod \"sub-test-model-1dc430-kserve-79cccdffd9-xg97q\" (UID: \"a042f16d-ba98-47c0-8f0e-cecb58dad951\") " pod="ai-tenant-e2e-shared-a-w3-1dc430/sub-test-model-1dc430-kserve-79cccdffd9-xg97q"
Sep 15 12:29:48.653371 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:48.653348    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a042f16d-ba98-47c0-8f0e-cecb58dad951-dshm\") pod \"sub-test-model-1dc430-kserve-79cccdffd9-xg97q\" (UID: \"a042f16d-ba98-47c0-8f0e-cecb58dad951\") " pod="ai-tenant-e2e-shared-a-w3-1dc430/sub-test-model-1dc430-kserve-79cccdffd9-xg97q"
Sep 15 12:29:48.653622 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:48.653603    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a042f16d-ba98-47c0-8f0e-cecb58dad951-tls-certs\") pod \"sub-test-model-1dc430-kserve-79cccdffd9-xg97q\" (UID: \"a042f16d-ba98-47c0-8f0e-cecb58dad951\") " pod="ai-tenant-e2e-shared-a-w3-1dc430/sub-test-model-1dc430-kserve-79cccdffd9-xg97q"
Sep 15 12:29:48.661323 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:48.661301    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pj45c\" (UniqueName: \"kubernetes.io/projected/a042f16d-ba98-47c0-8f0e-cecb58dad951-kube-api-access-pj45c\") pod \"sub-test-model-1dc430-kserve-79cccdffd9-xg97q\" (UID: \"a042f16d-ba98-47c0-8f0e-cecb58dad951\") " pod="ai-tenant-e2e-shared-a-w3-1dc430/sub-test-model-1dc430-kserve-79cccdffd9-xg97q"
Sep 15 12:29:48.739460 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:48.739371    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1dc430/sub-test-model-1dc430-kserve-79cccdffd9-xg97q"
Sep 15 12:29:49.023643 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:49.023616    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1dc430/sub-test-model-1dc430-kserve-79cccdffd9-xg97q"]
Sep 15 12:29:49.025197 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:29:49.025170    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda042f16d_ba98_47c0_8f0e_cecb58dad951.slice/crio-62be2bc4cbddf4261a46f8063c38e75de6ec32289940cf5e90cddd3457bd11a1 WatchSource:0}: Error finding container 62be2bc4cbddf4261a46f8063c38e75de6ec32289940cf5e90cddd3457bd11a1: Status 404 returned error can't find the container with id 62be2bc4cbddf4261a46f8063c38e75de6ec32289940cf5e90cddd3457bd11a1
Sep 15 12:29:49.876745 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:49.876708    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1dc430/sub-test-model-1dc430-kserve-79cccdffd9-xg97q" event={"ID":"a042f16d-ba98-47c0-8f0e-cecb58dad951","Type":"ContainerStarted","Data":"064516be9cf9eb65600e2d882e09ad4f714b85e7ec269b3f3605a47856a5bdaa"}
Sep 15 12:29:49.876745 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:49.876748    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1dc430/sub-test-model-1dc430-kserve-79cccdffd9-xg97q" event={"ID":"a042f16d-ba98-47c0-8f0e-cecb58dad951","Type":"ContainerStarted","Data":"62be2bc4cbddf4261a46f8063c38e75de6ec32289940cf5e90cddd3457bd11a1"}
Sep 15 12:29:49.895939 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:49.895878    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-1dc430/sub-test-model-1dc430-kserve-79cccdffd9-xg97q" podStartSLOduration=1.895857864 podStartE2EDuration="1.895857864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:29:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:29:49.895118361 +0000 UTC m=+1646.686040887" watchObservedRunningTime="2026-09-15 12:29:49.895857864 +0000 UTC m=+1646.686780376"
Sep 15 12:29:54.777834 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:54.777776    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-1dc430/auth-test-model-1dc430-kserve-77cd669cc5-wc628" podUID="099303bf-d5c6-4c12-bc2c-1f148d4fa19f" containerName="main" containerID="cri-o://6fcd1853ed7e1b02bb97eb2c3ef34c644e4b32f2ed2b6038e40f271607dcce11" gracePeriod=45
Sep 15 12:29:54.912202 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:54.912166    2555 generic.go:356] "Generic (PLEG): container finished" podID="099303bf-d5c6-4c12-bc2c-1f148d4fa19f" containerID="6fcd1853ed7e1b02bb97eb2c3ef34c644e4b32f2ed2b6038e40f271607dcce11" exitCode=0
Sep 15 12:29:54.912395 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:54.912196    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1dc430/auth-test-model-1dc430-kserve-77cd669cc5-wc628" event={"ID":"099303bf-d5c6-4c12-bc2c-1f148d4fa19f","Type":"ContainerDied","Data":"6fcd1853ed7e1b02bb97eb2c3ef34c644e4b32f2ed2b6038e40f271607dcce11"}
Sep 15 12:29:55.027585 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:55.027558    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1dc430/auth-test-model-1dc430-kserve-77cd669cc5-wc628"
Sep 15 12:29:55.113478 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:55.113397    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l8pzx\" (UniqueName: \"kubernetes.io/projected/099303bf-d5c6-4c12-bc2c-1f148d4fa19f-kube-api-access-l8pzx\") pod \"099303bf-d5c6-4c12-bc2c-1f148d4fa19f\" (UID: \"099303bf-d5c6-4c12-bc2c-1f148d4fa19f\") "
Sep 15 12:29:55.113478 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:55.113431    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/099303bf-d5c6-4c12-bc2c-1f148d4fa19f-dshm\") pod \"099303bf-d5c6-4c12-bc2c-1f148d4fa19f\" (UID: \"099303bf-d5c6-4c12-bc2c-1f148d4fa19f\") "
Sep 15 12:29:55.113478 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:55.113470    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/099303bf-d5c6-4c12-bc2c-1f148d4fa19f-tls-certs\") pod \"099303bf-d5c6-4c12-bc2c-1f148d4fa19f\" (UID: \"099303bf-d5c6-4c12-bc2c-1f148d4fa19f\") "
Sep 15 12:29:55.113766 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:55.113562    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/099303bf-d5c6-4c12-bc2c-1f148d4fa19f-home\") pod \"099303bf-d5c6-4c12-bc2c-1f148d4fa19f\" (UID: \"099303bf-d5c6-4c12-bc2c-1f148d4fa19f\") "
Sep 15 12:29:55.113766 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:55.113598    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/099303bf-d5c6-4c12-bc2c-1f148d4fa19f-tmp-dir\") pod \"099303bf-d5c6-4c12-bc2c-1f148d4fa19f\" (UID: \"099303bf-d5c6-4c12-bc2c-1f148d4fa19f\") "
Sep 15 12:29:55.113766 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:55.113630    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/099303bf-d5c6-4c12-bc2c-1f148d4fa19f-model-cache\") pod \"099303bf-d5c6-4c12-bc2c-1f148d4fa19f\" (UID: \"099303bf-d5c6-4c12-bc2c-1f148d4fa19f\") "
Sep 15 12:29:55.113943 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:55.113843    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/099303bf-d5c6-4c12-bc2c-1f148d4fa19f-home" (OuterVolumeSpecName: "home") pod "099303bf-d5c6-4c12-bc2c-1f148d4fa19f" (UID: "099303bf-d5c6-4c12-bc2c-1f148d4fa19f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:29:55.113943 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:55.113899    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/099303bf-d5c6-4c12-bc2c-1f148d4fa19f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "099303bf-d5c6-4c12-bc2c-1f148d4fa19f" (UID: "099303bf-d5c6-4c12-bc2c-1f148d4fa19f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:29:55.113943 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:55.113928    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/099303bf-d5c6-4c12-bc2c-1f148d4fa19f-model-cache" (OuterVolumeSpecName: "model-cache") pod "099303bf-d5c6-4c12-bc2c-1f148d4fa19f" (UID: "099303bf-d5c6-4c12-bc2c-1f148d4fa19f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:29:55.114164 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:55.114042    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/099303bf-d5c6-4c12-bc2c-1f148d4fa19f-home\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:29:55.114164 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:55.114061    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/099303bf-d5c6-4c12-bc2c-1f148d4fa19f-tmp-dir\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:29:55.114164 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:55.114074    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/099303bf-d5c6-4c12-bc2c-1f148d4fa19f-model-cache\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:29:55.115836 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:55.115798    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/099303bf-d5c6-4c12-bc2c-1f148d4fa19f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "099303bf-d5c6-4c12-bc2c-1f148d4fa19f" (UID: "099303bf-d5c6-4c12-bc2c-1f148d4fa19f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:29:55.115956 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:55.115833    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/099303bf-d5c6-4c12-bc2c-1f148d4fa19f-dshm" (OuterVolumeSpecName: "dshm") pod "099303bf-d5c6-4c12-bc2c-1f148d4fa19f" (UID: "099303bf-d5c6-4c12-bc2c-1f148d4fa19f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:29:55.115956 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:55.115856    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/099303bf-d5c6-4c12-bc2c-1f148d4fa19f-kube-api-access-l8pzx" (OuterVolumeSpecName: "kube-api-access-l8pzx") pod "099303bf-d5c6-4c12-bc2c-1f148d4fa19f" (UID: "099303bf-d5c6-4c12-bc2c-1f148d4fa19f"). InnerVolumeSpecName "kube-api-access-l8pzx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:29:55.215349 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:55.215308    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8pzx\" (UniqueName: \"kubernetes.io/projected/099303bf-d5c6-4c12-bc2c-1f148d4fa19f-kube-api-access-l8pzx\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:29:55.215349 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:55.215340    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/099303bf-d5c6-4c12-bc2c-1f148d4fa19f-dshm\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:29:55.215349 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:55.215350    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/099303bf-d5c6-4c12-bc2c-1f148d4fa19f-tls-certs\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:29:55.665420 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:55.665383    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5gkxs"]
Sep 15 12:29:55.919224 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:55.919131    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1dc430/auth-test-model-1dc430-kserve-77cd669cc5-wc628" event={"ID":"099303bf-d5c6-4c12-bc2c-1f148d4fa19f","Type":"ContainerDied","Data":"94ae2a49dbee18bb125c59da9edcf8c2fe019238e7e12c7c1af46689ddb91e94"}
Sep 15 12:29:55.919224 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:55.919144    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1dc430/auth-test-model-1dc430-kserve-77cd669cc5-wc628"
Sep 15 12:29:55.919224 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:55.919184    2555 scope.go:117] "RemoveContainer" containerID="6fcd1853ed7e1b02bb97eb2c3ef34c644e4b32f2ed2b6038e40f271607dcce11"
Sep 15 12:29:55.935495 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:55.935466    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1dc430/auth-test-model-1dc430-kserve-77cd669cc5-wc628"]
Sep 15 12:29:55.939480 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:55.939456    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1dc430/auth-test-model-1dc430-kserve-77cd669cc5-wc628"]
Sep 15 12:29:57.816746 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:57.816709    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="099303bf-d5c6-4c12-bc2c-1f148d4fa19f" path="/var/lib/kubelet/pods/099303bf-d5c6-4c12-bc2c-1f148d4fa19f/volumes"
Sep 15 12:29:58.739765 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:58.739723    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-1dc430/sub-test-model-1dc430-kserve-79cccdffd9-xg97q"
Sep 15 12:29:58.739765 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:58.739770    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-1dc430/sub-test-model-1dc430-kserve-79cccdffd9-xg97q"
Sep 15 12:29:58.752340 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:58.752311    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-1dc430/sub-test-model-1dc430-kserve-79cccdffd9-xg97q"
Sep 15 12:29:58.948988 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:29:58.948960    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-1dc430/sub-test-model-1dc430-kserve-79cccdffd9-xg97q"
Sep 15 12:30:00.134606 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:00.134562    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-969130-29824590-vk95z"]
Sep 15 12:30:00.135074 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:00.135056    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="099303bf-d5c6-4c12-bc2c-1f148d4fa19f" containerName="main"
Sep 15 12:30:00.135138 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:00.135077    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="099303bf-d5c6-4c12-bc2c-1f148d4fa19f" containerName="main"
Sep 15 12:30:00.135190 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:00.135154    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="099303bf-d5c6-4c12-bc2c-1f148d4fa19f" containerName="main"
Sep 15 12:30:00.139841 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:00.139814    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-969130-29824590-vk95z"
Sep 15 12:30:00.142564 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:00.142538    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tbvkt\""
Sep 15 12:30:00.149423 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:00.149395    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-969130-29824590-vk95z"]
Sep 15 12:30:00.150975 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:00.150950    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-969130-29824590-vk95z"
Sep 15 12:30:00.424088 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:00.424063    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-969130-29824590-vk95z"]
Sep 15 12:30:00.425751 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:30:00.425726    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95baa04b_f10c_456c_9bd6_403dbf093a75.slice/crio-94c0e195051c5dae01e1017323227705d2452a6cdf2f6a453549151eb0639f1b WatchSource:0}: Error finding container 94c0e195051c5dae01e1017323227705d2452a6cdf2f6a453549151eb0639f1b: Status 404 returned error can't find the container with id 94c0e195051c5dae01e1017323227705d2452a6cdf2f6a453549151eb0639f1b
Sep 15 12:30:00.948718 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:00.948676    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-969130-29824590-vk95z" event={"ID":"95baa04b-f10c-456c-9bd6-403dbf093a75","Type":"ContainerStarted","Data":"dc51035b271e40e8ae2f41b320c74d9f9c537247354ae580590401bfe164dee6"}
Sep 15 12:30:00.948877 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:00.948735    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-969130-29824590-vk95z" event={"ID":"95baa04b-f10c-456c-9bd6-403dbf093a75","Type":"ContainerStarted","Data":"94c0e195051c5dae01e1017323227705d2452a6cdf2f6a453549151eb0639f1b"}
Sep 15 12:30:00.967276 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:00.967217    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-969130-29824590-vk95z" podStartSLOduration=0.967201952 podStartE2EDuration="967.201952ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:30:00.966292399 +0000 UTC m=+1657.757214906" watchObservedRunningTime="2026-09-15 12:30:00.967201952 +0000 UTC m=+1657.758124462"
Sep 15 12:30:01.957052 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:01.957014    2555 generic.go:356] "Generic (PLEG): container finished" podID="95baa04b-f10c-456c-9bd6-403dbf093a75" containerID="dc51035b271e40e8ae2f41b320c74d9f9c537247354ae580590401bfe164dee6" exitCode=0
Sep 15 12:30:01.957462 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:01.957105    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-969130-29824590-vk95z" event={"ID":"95baa04b-f10c-456c-9bd6-403dbf093a75","Type":"ContainerDied","Data":"dc51035b271e40e8ae2f41b320c74d9f9c537247354ae580590401bfe164dee6"}
Sep 15 12:30:03.092841 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:03.092816    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-969130-29824590-vk95z"
Sep 15 12:30:03.969229 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:03.969202    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-969130-29824590-vk95z"
Sep 15 12:30:03.969424 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:03.969235    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-969130-29824590-vk95z" event={"ID":"95baa04b-f10c-456c-9bd6-403dbf093a75","Type":"ContainerDied","Data":"94c0e195051c5dae01e1017323227705d2452a6cdf2f6a453549151eb0639f1b"}
Sep 15 12:30:03.969424 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:03.969308    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="94c0e195051c5dae01e1017323227705d2452a6cdf2f6a453549151eb0639f1b"
Sep 15 12:30:11.953036 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:11.953000    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-969130/sub-test-model-969130-kserve-84bd666c5-svkgs"]
Sep 15 12:30:11.953442 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:11.953393    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95baa04b-f10c-456c-9bd6-403dbf093a75" containerName="cleanup"
Sep 15 12:30:11.953442 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:11.953405    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="95baa04b-f10c-456c-9bd6-403dbf093a75" containerName="cleanup"
Sep 15 12:30:11.953528 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:11.953466    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="95baa04b-f10c-456c-9bd6-403dbf093a75" containerName="cleanup"
Sep 15 12:30:11.956503 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:11.956483    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-969130/sub-test-model-969130-kserve-84bd666c5-svkgs"
Sep 15 12:30:11.961897 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:11.961871    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-969130\"/\"default-dockercfg-xf252\""
Sep 15 12:30:11.962029 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:11.961872    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-969130\"/\"sub-test-model-969130-kserve-self-signed-certs\""
Sep 15 12:30:11.962029 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:11.961954    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-969130\"/\"kube-root-ca.crt\""
Sep 15 12:30:11.962282 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:11.962245    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-969130\"/\"openshift-service-ca.crt\""
Sep 15 12:30:11.980564 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:11.980532    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-969130/sub-test-model-969130-kserve-84bd666c5-svkgs"]
Sep 15 12:30:12.064650 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:12.064610    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7da4aec8-d414-44c9-a02c-53120196f900-home\") pod \"sub-test-model-969130-kserve-84bd666c5-svkgs\" (UID: \"7da4aec8-d414-44c9-a02c-53120196f900\") " pod="ai-tenant-e2e-shared-b-w3-969130/sub-test-model-969130-kserve-84bd666c5-svkgs"
Sep 15 12:30:12.064839 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:12.064674    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7da4aec8-d414-44c9-a02c-53120196f900-tls-certs\") pod \"sub-test-model-969130-kserve-84bd666c5-svkgs\" (UID: \"7da4aec8-d414-44c9-a02c-53120196f900\") " pod="ai-tenant-e2e-shared-b-w3-969130/sub-test-model-969130-kserve-84bd666c5-svkgs"
Sep 15 12:30:12.064839 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:12.064711    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7da4aec8-d414-44c9-a02c-53120196f900-model-cache\") pod \"sub-test-model-969130-kserve-84bd666c5-svkgs\" (UID: \"7da4aec8-d414-44c9-a02c-53120196f900\") " pod="ai-tenant-e2e-shared-b-w3-969130/sub-test-model-969130-kserve-84bd666c5-svkgs"
Sep 15 12:30:12.064839 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:12.064763    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7da4aec8-d414-44c9-a02c-53120196f900-tmp-dir\") pod \"sub-test-model-969130-kserve-84bd666c5-svkgs\" (UID: \"7da4aec8-d414-44c9-a02c-53120196f900\") " pod="ai-tenant-e2e-shared-b-w3-969130/sub-test-model-969130-kserve-84bd666c5-svkgs"
Sep 15 12:30:12.064839 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:12.064796    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7da4aec8-d414-44c9-a02c-53120196f900-dshm\") pod \"sub-test-model-969130-kserve-84bd666c5-svkgs\" (UID: \"7da4aec8-d414-44c9-a02c-53120196f900\") " pod="ai-tenant-e2e-shared-b-w3-969130/sub-test-model-969130-kserve-84bd666c5-svkgs"
Sep 15 12:30:12.064839 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:12.064814    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2r4pg\" (UniqueName: \"kubernetes.io/projected/7da4aec8-d414-44c9-a02c-53120196f900-kube-api-access-2r4pg\") pod \"sub-test-model-969130-kserve-84bd666c5-svkgs\" (UID: \"7da4aec8-d414-44c9-a02c-53120196f900\") " pod="ai-tenant-e2e-shared-b-w3-969130/sub-test-model-969130-kserve-84bd666c5-svkgs"
Sep 15 12:30:12.166346 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:12.166306    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7da4aec8-d414-44c9-a02c-53120196f900-tls-certs\") pod \"sub-test-model-969130-kserve-84bd666c5-svkgs\" (UID: \"7da4aec8-d414-44c9-a02c-53120196f900\") " pod="ai-tenant-e2e-shared-b-w3-969130/sub-test-model-969130-kserve-84bd666c5-svkgs"
Sep 15 12:30:12.166346 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:12.166348    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7da4aec8-d414-44c9-a02c-53120196f900-model-cache\") pod \"sub-test-model-969130-kserve-84bd666c5-svkgs\" (UID: \"7da4aec8-d414-44c9-a02c-53120196f900\") " pod="ai-tenant-e2e-shared-b-w3-969130/sub-test-model-969130-kserve-84bd666c5-svkgs"
Sep 15 12:30:12.166606 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:12.166399    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7da4aec8-d414-44c9-a02c-53120196f900-tmp-dir\") pod \"sub-test-model-969130-kserve-84bd666c5-svkgs\" (UID: \"7da4aec8-d414-44c9-a02c-53120196f900\") " pod="ai-tenant-e2e-shared-b-w3-969130/sub-test-model-969130-kserve-84bd666c5-svkgs"
Sep 15 12:30:12.166606 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:12.166420    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7da4aec8-d414-44c9-a02c-53120196f900-dshm\") pod \"sub-test-model-969130-kserve-84bd666c5-svkgs\" (UID: \"7da4aec8-d414-44c9-a02c-53120196f900\") " pod="ai-tenant-e2e-shared-b-w3-969130/sub-test-model-969130-kserve-84bd666c5-svkgs"
Sep 15 12:30:12.166606 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:12.166438    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2r4pg\" (UniqueName: \"kubernetes.io/projected/7da4aec8-d414-44c9-a02c-53120196f900-kube-api-access-2r4pg\") pod \"sub-test-model-969130-kserve-84bd666c5-svkgs\" (UID: \"7da4aec8-d414-44c9-a02c-53120196f900\") " pod="ai-tenant-e2e-shared-b-w3-969130/sub-test-model-969130-kserve-84bd666c5-svkgs"
Sep 15 12:30:12.166606 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:12.166459    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7da4aec8-d414-44c9-a02c-53120196f900-home\") pod \"sub-test-model-969130-kserve-84bd666c5-svkgs\" (UID: \"7da4aec8-d414-44c9-a02c-53120196f900\") " pod="ai-tenant-e2e-shared-b-w3-969130/sub-test-model-969130-kserve-84bd666c5-svkgs"
Sep 15 12:30:12.166895 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:12.166851    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7da4aec8-d414-44c9-a02c-53120196f900-tmp-dir\") pod \"sub-test-model-969130-kserve-84bd666c5-svkgs\" (UID: \"7da4aec8-d414-44c9-a02c-53120196f900\") " pod="ai-tenant-e2e-shared-b-w3-969130/sub-test-model-969130-kserve-84bd666c5-svkgs"
Sep 15 12:30:12.167015 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:12.166905    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7da4aec8-d414-44c9-a02c-53120196f900-home\") pod \"sub-test-model-969130-kserve-84bd666c5-svkgs\" (UID: \"7da4aec8-d414-44c9-a02c-53120196f900\") " pod="ai-tenant-e2e-shared-b-w3-969130/sub-test-model-969130-kserve-84bd666c5-svkgs"
Sep 15 12:30:12.167015 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:12.166921    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7da4aec8-d414-44c9-a02c-53120196f900-model-cache\") pod \"sub-test-model-969130-kserve-84bd666c5-svkgs\" (UID: \"7da4aec8-d414-44c9-a02c-53120196f900\") " pod="ai-tenant-e2e-shared-b-w3-969130/sub-test-model-969130-kserve-84bd666c5-svkgs"
Sep 15 12:30:12.168701 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:12.168675    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7da4aec8-d414-44c9-a02c-53120196f900-dshm\") pod \"sub-test-model-969130-kserve-84bd666c5-svkgs\" (UID: \"7da4aec8-d414-44c9-a02c-53120196f900\") " pod="ai-tenant-e2e-shared-b-w3-969130/sub-test-model-969130-kserve-84bd666c5-svkgs"
Sep 15 12:30:12.168992 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:12.168973    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7da4aec8-d414-44c9-a02c-53120196f900-tls-certs\") pod \"sub-test-model-969130-kserve-84bd666c5-svkgs\" (UID: \"7da4aec8-d414-44c9-a02c-53120196f900\") " pod="ai-tenant-e2e-shared-b-w3-969130/sub-test-model-969130-kserve-84bd666c5-svkgs"
Sep 15 12:30:12.176355 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:12.176326    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2r4pg\" (UniqueName: \"kubernetes.io/projected/7da4aec8-d414-44c9-a02c-53120196f900-kube-api-access-2r4pg\") pod \"sub-test-model-969130-kserve-84bd666c5-svkgs\" (UID: \"7da4aec8-d414-44c9-a02c-53120196f900\") " pod="ai-tenant-e2e-shared-b-w3-969130/sub-test-model-969130-kserve-84bd666c5-svkgs"
Sep 15 12:30:12.265177 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:12.265094    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-969130/sub-test-model-969130-kserve-84bd666c5-svkgs"
Sep 15 12:30:12.607095 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:30:12.607062    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7da4aec8_d414_44c9_a02c_53120196f900.slice/crio-357b6e842b8f5dee962c7d1f058941a78fca056e27906b1c8a78de0dd0fc8a73 WatchSource:0}: Error finding container 357b6e842b8f5dee962c7d1f058941a78fca056e27906b1c8a78de0dd0fc8a73: Status 404 returned error can't find the container with id 357b6e842b8f5dee962c7d1f058941a78fca056e27906b1c8a78de0dd0fc8a73
Sep 15 12:30:12.607561 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:12.607538    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-969130/sub-test-model-969130-kserve-84bd666c5-svkgs"]
Sep 15 12:30:13.020506 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:13.020471    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-969130/sub-test-model-969130-kserve-84bd666c5-svkgs" event={"ID":"7da4aec8-d414-44c9-a02c-53120196f900","Type":"ContainerStarted","Data":"3033a73d6bf758ea3b7b7430bb3cb885170227df45e2719dcc2ed910a93a1b49"}
Sep 15 12:30:13.020977 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:13.020517    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-969130/sub-test-model-969130-kserve-84bd666c5-svkgs" event={"ID":"7da4aec8-d414-44c9-a02c-53120196f900","Type":"ContainerStarted","Data":"357b6e842b8f5dee962c7d1f058941a78fca056e27906b1c8a78de0dd0fc8a73"}
Sep 15 12:30:13.049802 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:13.049737    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-969130/sub-test-model-969130-kserve-84bd666c5-svkgs" podStartSLOduration=2.049713296 podStartE2EDuration="2.049713296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:30:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:30:13.047022123 +0000 UTC m=+1669.837944655" watchObservedRunningTime="2026-09-15 12:30:13.049713296 +0000 UTC m=+1669.840635810"
Sep 15 12:30:15.699099 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:15.699061    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5gkxs"]
Sep 15 12:30:22.265232 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:22.265195    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-969130/sub-test-model-969130-kserve-84bd666c5-svkgs"
Sep 15 12:30:22.265685 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:22.265420    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-969130/sub-test-model-969130-kserve-84bd666c5-svkgs"
Sep 15 12:30:22.278360 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:22.278331    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-969130/sub-test-model-969130-kserve-84bd666c5-svkgs"
Sep 15 12:30:23.087075 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:23.087043    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-969130/sub-test-model-969130-kserve-84bd666c5-svkgs"
Sep 15 12:30:33.771974 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:33.771934    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5gkxs"]
Sep 15 12:30:45.190883 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:45.190847    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7941-b4e8e4"]
Sep 15 12:30:45.194453 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:45.194435    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7941-b4e8e4"
Sep 15 12:30:45.197296 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:45.197255    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-q76n5\""
Sep 15 12:30:45.205131 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:45.205105    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7941-b4e8e4"]
Sep 15 12:30:45.265953 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:45.265907    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjnlf\" (UniqueName: \"kubernetes.io/projected/8c9f1da7-7d9e-4273-a8f6-d4ba8c5aff55-kube-api-access-xjnlf\") pod \"test-curl-7941-b4e8e4\" (UID: \"8c9f1da7-7d9e-4273-a8f6-d4ba8c5aff55\") " pod="opendatahub/test-curl-7941-b4e8e4"
Sep 15 12:30:45.366879 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:45.366834    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjnlf\" (UniqueName: \"kubernetes.io/projected/8c9f1da7-7d9e-4273-a8f6-d4ba8c5aff55-kube-api-access-xjnlf\") pod \"test-curl-7941-b4e8e4\" (UID: \"8c9f1da7-7d9e-4273-a8f6-d4ba8c5aff55\") " pod="opendatahub/test-curl-7941-b4e8e4"
Sep 15 12:30:45.376192 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:45.376161    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjnlf\" (UniqueName: \"kubernetes.io/projected/8c9f1da7-7d9e-4273-a8f6-d4ba8c5aff55-kube-api-access-xjnlf\") pod \"test-curl-7941-b4e8e4\" (UID: \"8c9f1da7-7d9e-4273-a8f6-d4ba8c5aff55\") " pod="opendatahub/test-curl-7941-b4e8e4"
Sep 15 12:30:45.504740 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:45.504650    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7941-b4e8e4"
Sep 15 12:30:45.773338 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:45.773312    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7941-b4e8e4"]
Sep 15 12:30:45.775090 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:30:45.775062    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8c9f1da7_7d9e_4273_a8f6_d4ba8c5aff55.slice/crio-bd1784ed8adb0cc7ba2e5b733bb3c78b896e05f733cd982de2aa1252c7fde698 WatchSource:0}: Error finding container bd1784ed8adb0cc7ba2e5b733bb3c78b896e05f733cd982de2aa1252c7fde698: Status 404 returned error can't find the container with id bd1784ed8adb0cc7ba2e5b733bb3c78b896e05f733cd982de2aa1252c7fde698
Sep 15 12:30:46.205080 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:46.205045    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7941-b4e8e4" event={"ID":"8c9f1da7-7d9e-4273-a8f6-d4ba8c5aff55","Type":"ContainerStarted","Data":"2e12ae883374d05e68b301a180dc297af55a63024ea72d21af9b0a40aec0da83"}
Sep 15 12:30:46.205512 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:46.205094    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7941-b4e8e4" event={"ID":"8c9f1da7-7d9e-4273-a8f6-d4ba8c5aff55","Type":"ContainerStarted","Data":"bd1784ed8adb0cc7ba2e5b733bb3c78b896e05f733cd982de2aa1252c7fde698"}
Sep 15 12:30:46.220739 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:46.220485    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7941-b4e8e4" podStartSLOduration=1.22046622 podStartE2EDuration="1.22046622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:30:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:30:46.220408672 +0000 UTC m=+1703.011331183" watchObservedRunningTime="2026-09-15 12:30:46.22046622 +0000 UTC m=+1703.011388730"
Sep 15 12:30:46.855894 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:46.855854    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7941-b4e8e4"]
Sep 15 12:30:46.861221 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:46.861188    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7941-b4e8e4"]
Sep 15 12:30:47.007314 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:47.007276    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7941-5890b7"]
Sep 15 12:30:47.009969 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:47.008108    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c9f1da7-7d9e-4273-a8f6-d4ba8c5aff55" containerName="test-curl-7941-b4e8e4"
Sep 15 12:30:47.009969 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:47.008132    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c9f1da7-7d9e-4273-a8f6-d4ba8c5aff55" containerName="test-curl-7941-b4e8e4"
Sep 15 12:30:47.009969 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:47.008327    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c9f1da7-7d9e-4273-a8f6-d4ba8c5aff55" containerName="test-curl-7941-b4e8e4"
Sep 15 12:30:47.014006 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:47.013978    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7941-5890b7"
Sep 15 12:30:47.017880 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:47.017852    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7941-5890b7"]
Sep 15 12:30:47.082474 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:47.082438    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c9lj\" (UniqueName: \"kubernetes.io/projected/dc7d897a-c79d-43cf-9999-9c9600a56e83-kube-api-access-6c9lj\") pod \"test-curl-7941-5890b7\" (UID: \"dc7d897a-c79d-43cf-9999-9c9600a56e83\") " pod="opendatahub/test-curl-7941-5890b7"
Sep 15 12:30:47.183468 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:47.183429    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c9lj\" (UniqueName: \"kubernetes.io/projected/dc7d897a-c79d-43cf-9999-9c9600a56e83-kube-api-access-6c9lj\") pod \"test-curl-7941-5890b7\" (UID: \"dc7d897a-c79d-43cf-9999-9c9600a56e83\") " pod="opendatahub/test-curl-7941-5890b7"
Sep 15 12:30:47.193609 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:47.193577    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c9lj\" (UniqueName: \"kubernetes.io/projected/dc7d897a-c79d-43cf-9999-9c9600a56e83-kube-api-access-6c9lj\") pod \"test-curl-7941-5890b7\" (UID: \"dc7d897a-c79d-43cf-9999-9c9600a56e83\") " pod="opendatahub/test-curl-7941-5890b7"
Sep 15 12:30:47.210242 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:30:47.210217    2555 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7941-b4e8e4" container="test-curl-7941-b4e8e4"
Sep 15 12:30:47.325368 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:47.325255    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7941-5890b7"
Sep 15 12:30:47.595332 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:47.595275    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7941-5890b7"]
Sep 15 12:30:47.597251 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:30:47.597213    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddc7d897a_c79d_43cf_9999_9c9600a56e83.slice/crio-6fdefb096eab40b962e771b63f6b1d40e92ef8aa6f14df2496de0c1545553e02 WatchSource:0}: Error finding container 6fdefb096eab40b962e771b63f6b1d40e92ef8aa6f14df2496de0c1545553e02: Status 404 returned error can't find the container with id 6fdefb096eab40b962e771b63f6b1d40e92ef8aa6f14df2496de0c1545553e02
Sep 15 12:30:48.216984 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:48.216944    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7941-5890b7" event={"ID":"dc7d897a-c79d-43cf-9999-9c9600a56e83","Type":"ContainerStarted","Data":"86ed874a5218838d1a470c67c00ab128fd6f97761bd617764ef7d502fb627ca4"}
Sep 15 12:30:48.216984 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:48.216987    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7941-5890b7" event={"ID":"dc7d897a-c79d-43cf-9999-9c9600a56e83","Type":"ContainerStarted","Data":"6fdefb096eab40b962e771b63f6b1d40e92ef8aa6f14df2496de0c1545553e02"}
Sep 15 12:30:48.217515 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:48.217016    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7941-b4e8e4" podUID="8c9f1da7-7d9e-4273-a8f6-d4ba8c5aff55" containerName="test-curl-7941-b4e8e4" containerID="cri-o://2e12ae883374d05e68b301a180dc297af55a63024ea72d21af9b0a40aec0da83" gracePeriod=2
Sep 15 12:30:48.234193 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:48.234135    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7941-5890b7" podStartSLOduration=2.23411926 podStartE2EDuration="2.23411926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:30:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:30:48.231408045 +0000 UTC m=+1705.022330567" watchObservedRunningTime="2026-09-15 12:30:48.23411926 +0000 UTC m=+1705.025041771"
Sep 15 12:30:48.884518 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:48.884473    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7941-5890b7"]
Sep 15 12:30:48.891508 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:48.891477    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7941-5890b7"]
Sep 15 12:30:49.063258 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:49.063172    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5gkxs"]
Sep 15 12:30:49.222772 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:30:49.222744    2555 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7941-5890b7" container="test-curl-7941-5890b7"
Sep 15 12:30:50.227839 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:50.227807    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7941-5890b7" podUID="dc7d897a-c79d-43cf-9999-9c9600a56e83" containerName="test-curl-7941-5890b7" containerID="cri-o://86ed874a5218838d1a470c67c00ab128fd6f97761bd617764ef7d502fb627ca4" gracePeriod=2
Sep 15 12:30:50.409072 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:50.409049    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7941-b4e8e4"
Sep 15 12:30:50.512190 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:50.512101    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xjnlf\" (UniqueName: \"kubernetes.io/projected/8c9f1da7-7d9e-4273-a8f6-d4ba8c5aff55-kube-api-access-xjnlf\") pod \"8c9f1da7-7d9e-4273-a8f6-d4ba8c5aff55\" (UID: \"8c9f1da7-7d9e-4273-a8f6-d4ba8c5aff55\") "
Sep 15 12:30:50.514218 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:50.514193    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c9f1da7-7d9e-4273-a8f6-d4ba8c5aff55-kube-api-access-xjnlf" (OuterVolumeSpecName: "kube-api-access-xjnlf") pod "8c9f1da7-7d9e-4273-a8f6-d4ba8c5aff55" (UID: "8c9f1da7-7d9e-4273-a8f6-d4ba8c5aff55"). InnerVolumeSpecName "kube-api-access-xjnlf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:30:50.613646 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:50.613607    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjnlf\" (UniqueName: \"kubernetes.io/projected/8c9f1da7-7d9e-4273-a8f6-d4ba8c5aff55-kube-api-access-xjnlf\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:30:51.080207 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:51.080172    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1dc430/sub-test-model-1dc430-kserve-79cccdffd9-xg97q"]
Sep 15 12:30:51.234040 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:51.234006    2555 generic.go:356] "Generic (PLEG): container finished" podID="8c9f1da7-7d9e-4273-a8f6-d4ba8c5aff55" containerID="2e12ae883374d05e68b301a180dc297af55a63024ea72d21af9b0a40aec0da83" exitCode=137
Sep 15 12:30:51.234469 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:51.234058    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7941-b4e8e4"
Sep 15 12:30:51.234469 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:51.234062    2555 scope.go:117] "RemoveContainer" containerID="2e12ae883374d05e68b301a180dc297af55a63024ea72d21af9b0a40aec0da83"
Sep 15 12:30:51.243845 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:51.243827    2555 scope.go:117] "RemoveContainer" containerID="2e12ae883374d05e68b301a180dc297af55a63024ea72d21af9b0a40aec0da83"
Sep 15 12:30:51.244149 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:30:51.244124    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e12ae883374d05e68b301a180dc297af55a63024ea72d21af9b0a40aec0da83\": container with ID starting with 2e12ae883374d05e68b301a180dc297af55a63024ea72d21af9b0a40aec0da83 not found: ID does not exist" containerID="2e12ae883374d05e68b301a180dc297af55a63024ea72d21af9b0a40aec0da83"
Sep 15 12:30:51.244197 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:51.244165    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e12ae883374d05e68b301a180dc297af55a63024ea72d21af9b0a40aec0da83"} err="failed to get container status \"2e12ae883374d05e68b301a180dc297af55a63024ea72d21af9b0a40aec0da83\": rpc error: code = NotFound desc = could not find container \"2e12ae883374d05e68b301a180dc297af55a63024ea72d21af9b0a40aec0da83\": container with ID starting with 2e12ae883374d05e68b301a180dc297af55a63024ea72d21af9b0a40aec0da83 not found: ID does not exist"
Sep 15 12:30:51.244344 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:30:51.244323    2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7941-b4e8e4\" is forbidden: User \"system:node:ip-10-0-140-17.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-17.ec2.internal' and this object" podUID="8c9f1da7-7d9e-4273-a8f6-d4ba8c5aff55" pod="opendatahub/test-curl-7941-b4e8e4"
Sep 15 12:30:51.817327 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:51.817290    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c9f1da7-7d9e-4273-a8f6-d4ba8c5aff55" path="/var/lib/kubelet/pods/8c9f1da7-7d9e-4273-a8f6-d4ba8c5aff55/volumes"
Sep 15 12:30:51.940371 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:51.940337    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-969130/sub-test-model-969130-kserve-84bd666c5-svkgs"]
Sep 15 12:30:52.408732 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:52.408700    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7941-5890b7"
Sep 15 12:30:52.530612 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:52.530492    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6c9lj\" (UniqueName: \"kubernetes.io/projected/dc7d897a-c79d-43cf-9999-9c9600a56e83-kube-api-access-6c9lj\") pod \"dc7d897a-c79d-43cf-9999-9c9600a56e83\" (UID: \"dc7d897a-c79d-43cf-9999-9c9600a56e83\") "
Sep 15 12:30:52.532620 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:52.532594    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc7d897a-c79d-43cf-9999-9c9600a56e83-kube-api-access-6c9lj" (OuterVolumeSpecName: "kube-api-access-6c9lj") pod "dc7d897a-c79d-43cf-9999-9c9600a56e83" (UID: "dc7d897a-c79d-43cf-9999-9c9600a56e83"). InnerVolumeSpecName "kube-api-access-6c9lj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:30:52.631284 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:52.631219    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6c9lj\" (UniqueName: \"kubernetes.io/projected/dc7d897a-c79d-43cf-9999-9c9600a56e83-kube-api-access-6c9lj\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:30:53.262894 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:53.262858    2555 generic.go:356] "Generic (PLEG): container finished" podID="dc7d897a-c79d-43cf-9999-9c9600a56e83" containerID="86ed874a5218838d1a470c67c00ab128fd6f97761bd617764ef7d502fb627ca4" exitCode=137
Sep 15 12:30:53.263083 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:53.262902    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7941-5890b7"
Sep 15 12:30:53.263083 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:53.262967    2555 scope.go:117] "RemoveContainer" containerID="86ed874a5218838d1a470c67c00ab128fd6f97761bd617764ef7d502fb627ca4"
Sep 15 12:30:53.272653 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:53.272490    2555 scope.go:117] "RemoveContainer" containerID="86ed874a5218838d1a470c67c00ab128fd6f97761bd617764ef7d502fb627ca4"
Sep 15 12:30:53.272846 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:30:53.272827    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86ed874a5218838d1a470c67c00ab128fd6f97761bd617764ef7d502fb627ca4\": container with ID starting with 86ed874a5218838d1a470c67c00ab128fd6f97761bd617764ef7d502fb627ca4 not found: ID does not exist" containerID="86ed874a5218838d1a470c67c00ab128fd6f97761bd617764ef7d502fb627ca4"
Sep 15 12:30:53.272902 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:53.272855    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86ed874a5218838d1a470c67c00ab128fd6f97761bd617764ef7d502fb627ca4"} err="failed to get container status \"86ed874a5218838d1a470c67c00ab128fd6f97761bd617764ef7d502fb627ca4\": rpc error: code = NotFound desc = could not find container \"86ed874a5218838d1a470c67c00ab128fd6f97761bd617764ef7d502fb627ca4\": container with ID starting with 86ed874a5218838d1a470c67c00ab128fd6f97761bd617764ef7d502fb627ca4 not found: ID does not exist"
Sep 15 12:30:53.274281 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:30:53.274244    2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7941-5890b7\" is forbidden: User \"system:node:ip-10-0-140-17.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-17.ec2.internal' and this object" podUID="dc7d897a-c79d-43cf-9999-9c9600a56e83" pod="opendatahub/test-curl-7941-5890b7"
Sep 15 12:30:53.816785 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:30:53.816746    2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7941-5890b7\" is forbidden: User \"system:node:ip-10-0-140-17.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-17.ec2.internal' and this object" podUID="dc7d897a-c79d-43cf-9999-9c9600a56e83" pod="opendatahub/test-curl-7941-5890b7"
Sep 15 12:30:53.817227 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:53.817207    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc7d897a-c79d-43cf-9999-9c9600a56e83" path="/var/lib/kubelet/pods/dc7d897a-c79d-43cf-9999-9c9600a56e83/volumes"
Sep 15 12:30:57.952523 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:57.952490    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-1dc430/rate-test-model-1dc430-kserve-75c596b759-g6xch"]
Sep 15 12:30:57.952904 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:57.952844    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc7d897a-c79d-43cf-9999-9c9600a56e83" containerName="test-curl-7941-5890b7"
Sep 15 12:30:57.952904 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:57.952857    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc7d897a-c79d-43cf-9999-9c9600a56e83" containerName="test-curl-7941-5890b7"
Sep 15 12:30:57.952979 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:57.952936    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc7d897a-c79d-43cf-9999-9c9600a56e83" containerName="test-curl-7941-5890b7"
Sep 15 12:30:57.955886 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:57.955871    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1dc430/rate-test-model-1dc430-kserve-75c596b759-g6xch"
Sep 15 12:30:57.962905 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:57.962882    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-1dc430\"/\"rate-test-model-1dc430-kserve-self-signed-certs\""
Sep 15 12:30:57.983920 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:57.983891    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1dc430/rate-test-model-1dc430-kserve-75c596b759-g6xch"]
Sep 15 12:30:58.084536 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:58.084495    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ec07ee0-5376-4887-bff7-a0d76804939d-dshm\") pod \"rate-test-model-1dc430-kserve-75c596b759-g6xch\" (UID: \"7ec07ee0-5376-4887-bff7-a0d76804939d\") " pod="ai-tenant-e2e-shared-a-w3-1dc430/rate-test-model-1dc430-kserve-75c596b759-g6xch"
Sep 15 12:30:58.084735 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:58.084565    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ec07ee0-5376-4887-bff7-a0d76804939d-model-cache\") pod \"rate-test-model-1dc430-kserve-75c596b759-g6xch\" (UID: \"7ec07ee0-5376-4887-bff7-a0d76804939d\") " pod="ai-tenant-e2e-shared-a-w3-1dc430/rate-test-model-1dc430-kserve-75c596b759-g6xch"
Sep 15 12:30:58.084735 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:58.084613    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxlvh\" (UniqueName: \"kubernetes.io/projected/7ec07ee0-5376-4887-bff7-a0d76804939d-kube-api-access-kxlvh\") pod \"rate-test-model-1dc430-kserve-75c596b759-g6xch\" (UID: \"7ec07ee0-5376-4887-bff7-a0d76804939d\") " pod="ai-tenant-e2e-shared-a-w3-1dc430/rate-test-model-1dc430-kserve-75c596b759-g6xch"
Sep 15 12:30:58.084735 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:58.084649    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ec07ee0-5376-4887-bff7-a0d76804939d-tls-certs\") pod \"rate-test-model-1dc430-kserve-75c596b759-g6xch\" (UID: \"7ec07ee0-5376-4887-bff7-a0d76804939d\") " pod="ai-tenant-e2e-shared-a-w3-1dc430/rate-test-model-1dc430-kserve-75c596b759-g6xch"
Sep 15 12:30:58.084735 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:58.084670    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ec07ee0-5376-4887-bff7-a0d76804939d-tmp-dir\") pod \"rate-test-model-1dc430-kserve-75c596b759-g6xch\" (UID: \"7ec07ee0-5376-4887-bff7-a0d76804939d\") " pod="ai-tenant-e2e-shared-a-w3-1dc430/rate-test-model-1dc430-kserve-75c596b759-g6xch"
Sep 15 12:30:58.084887 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:58.084749    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ec07ee0-5376-4887-bff7-a0d76804939d-home\") pod \"rate-test-model-1dc430-kserve-75c596b759-g6xch\" (UID: \"7ec07ee0-5376-4887-bff7-a0d76804939d\") " pod="ai-tenant-e2e-shared-a-w3-1dc430/rate-test-model-1dc430-kserve-75c596b759-g6xch"
Sep 15 12:30:58.186192 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:58.186153    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ec07ee0-5376-4887-bff7-a0d76804939d-home\") pod \"rate-test-model-1dc430-kserve-75c596b759-g6xch\" (UID: \"7ec07ee0-5376-4887-bff7-a0d76804939d\") " pod="ai-tenant-e2e-shared-a-w3-1dc430/rate-test-model-1dc430-kserve-75c596b759-g6xch"
Sep 15 12:30:58.186419 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:58.186229    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ec07ee0-5376-4887-bff7-a0d76804939d-dshm\") pod \"rate-test-model-1dc430-kserve-75c596b759-g6xch\" (UID: \"7ec07ee0-5376-4887-bff7-a0d76804939d\") " pod="ai-tenant-e2e-shared-a-w3-1dc430/rate-test-model-1dc430-kserve-75c596b759-g6xch"
Sep 15 12:30:58.186419 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:58.186298    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ec07ee0-5376-4887-bff7-a0d76804939d-model-cache\") pod \"rate-test-model-1dc430-kserve-75c596b759-g6xch\" (UID: \"7ec07ee0-5376-4887-bff7-a0d76804939d\") " pod="ai-tenant-e2e-shared-a-w3-1dc430/rate-test-model-1dc430-kserve-75c596b759-g6xch"
Sep 15 12:30:58.186419 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:58.186317    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxlvh\" (UniqueName: \"kubernetes.io/projected/7ec07ee0-5376-4887-bff7-a0d76804939d-kube-api-access-kxlvh\") pod \"rate-test-model-1dc430-kserve-75c596b759-g6xch\" (UID: \"7ec07ee0-5376-4887-bff7-a0d76804939d\") " pod="ai-tenant-e2e-shared-a-w3-1dc430/rate-test-model-1dc430-kserve-75c596b759-g6xch"
Sep 15 12:30:58.186419 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:58.186334    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ec07ee0-5376-4887-bff7-a0d76804939d-tls-certs\") pod \"rate-test-model-1dc430-kserve-75c596b759-g6xch\" (UID: \"7ec07ee0-5376-4887-bff7-a0d76804939d\") " pod="ai-tenant-e2e-shared-a-w3-1dc430/rate-test-model-1dc430-kserve-75c596b759-g6xch"
Sep 15 12:30:58.186419 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:58.186352    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ec07ee0-5376-4887-bff7-a0d76804939d-tmp-dir\") pod \"rate-test-model-1dc430-kserve-75c596b759-g6xch\" (UID: \"7ec07ee0-5376-4887-bff7-a0d76804939d\") " pod="ai-tenant-e2e-shared-a-w3-1dc430/rate-test-model-1dc430-kserve-75c596b759-g6xch"
Sep 15 12:30:58.186690 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:58.186626    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ec07ee0-5376-4887-bff7-a0d76804939d-home\") pod \"rate-test-model-1dc430-kserve-75c596b759-g6xch\" (UID: \"7ec07ee0-5376-4887-bff7-a0d76804939d\") " pod="ai-tenant-e2e-shared-a-w3-1dc430/rate-test-model-1dc430-kserve-75c596b759-g6xch"
Sep 15 12:30:58.186690 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:58.186652    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ec07ee0-5376-4887-bff7-a0d76804939d-tmp-dir\") pod \"rate-test-model-1dc430-kserve-75c596b759-g6xch\" (UID: \"7ec07ee0-5376-4887-bff7-a0d76804939d\") " pod="ai-tenant-e2e-shared-a-w3-1dc430/rate-test-model-1dc430-kserve-75c596b759-g6xch"
Sep 15 12:30:58.186764 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:58.186721    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ec07ee0-5376-4887-bff7-a0d76804939d-model-cache\") pod \"rate-test-model-1dc430-kserve-75c596b759-g6xch\" (UID: \"7ec07ee0-5376-4887-bff7-a0d76804939d\") " pod="ai-tenant-e2e-shared-a-w3-1dc430/rate-test-model-1dc430-kserve-75c596b759-g6xch"
Sep 15 12:30:58.188410 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:58.188388    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ec07ee0-5376-4887-bff7-a0d76804939d-dshm\") pod \"rate-test-model-1dc430-kserve-75c596b759-g6xch\" (UID: \"7ec07ee0-5376-4887-bff7-a0d76804939d\") " pod="ai-tenant-e2e-shared-a-w3-1dc430/rate-test-model-1dc430-kserve-75c596b759-g6xch"
Sep 15 12:30:58.188747 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:58.188730    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ec07ee0-5376-4887-bff7-a0d76804939d-tls-certs\") pod \"rate-test-model-1dc430-kserve-75c596b759-g6xch\" (UID: \"7ec07ee0-5376-4887-bff7-a0d76804939d\") " pod="ai-tenant-e2e-shared-a-w3-1dc430/rate-test-model-1dc430-kserve-75c596b759-g6xch"
Sep 15 12:30:58.197182 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:58.197159    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxlvh\" (UniqueName: \"kubernetes.io/projected/7ec07ee0-5376-4887-bff7-a0d76804939d-kube-api-access-kxlvh\") pod \"rate-test-model-1dc430-kserve-75c596b759-g6xch\" (UID: \"7ec07ee0-5376-4887-bff7-a0d76804939d\") " pod="ai-tenant-e2e-shared-a-w3-1dc430/rate-test-model-1dc430-kserve-75c596b759-g6xch"
Sep 15 12:30:58.265356 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:58.265253    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1dc430/rate-test-model-1dc430-kserve-75c596b759-g6xch"
Sep 15 12:30:58.566577 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:58.566547    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1dc430/rate-test-model-1dc430-kserve-75c596b759-g6xch"]
Sep 15 12:30:58.567362 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:30:58.567334    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7ec07ee0_5376_4887_bff7_a0d76804939d.slice/crio-981f2b26afd7723d13e15ffe05bdbb956534fb3ca3b40a000d23584abf8c1231 WatchSource:0}: Error finding container 981f2b26afd7723d13e15ffe05bdbb956534fb3ca3b40a000d23584abf8c1231: Status 404 returned error can't find the container with id 981f2b26afd7723d13e15ffe05bdbb956534fb3ca3b40a000d23584abf8c1231
Sep 15 12:30:59.303817 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:59.303781    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1dc430/rate-test-model-1dc430-kserve-75c596b759-g6xch" event={"ID":"7ec07ee0-5376-4887-bff7-a0d76804939d","Type":"ContainerStarted","Data":"0073268a18f5c2ee3719d034a3d6c3fdd8ee9327017a1657749e1b58229287d7"}
Sep 15 12:30:59.303817 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:59.303823    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1dc430/rate-test-model-1dc430-kserve-75c596b759-g6xch" event={"ID":"7ec07ee0-5376-4887-bff7-a0d76804939d","Type":"ContainerStarted","Data":"981f2b26afd7723d13e15ffe05bdbb956534fb3ca3b40a000d23584abf8c1231"}
Sep 15 12:30:59.336249 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:30:59.336191    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-1dc430/rate-test-model-1dc430-kserve-75c596b759-g6xch" podStartSLOduration=2.336176777 podStartE2EDuration="2.336176777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:30:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:30:59.335615218 +0000 UTC m=+1716.126537729" watchObservedRunningTime="2026-09-15 12:30:59.336176777 +0000 UTC m=+1716.127099287"
Sep 15 12:31:00.909197 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:00.909147    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5gkxs"]
Sep 15 12:31:06.099034 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:06.098990    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-1dc430/sub-test-model-1dc430-kserve-79cccdffd9-xg97q" podUID="a042f16d-ba98-47c0-8f0e-cecb58dad951" containerName="main" containerID="cri-o://064516be9cf9eb65600e2d882e09ad4f714b85e7ec269b3f3605a47856a5bdaa" gracePeriod=45
Sep 15 12:31:06.351302 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:06.351199    2555 generic.go:356] "Generic (PLEG): container finished" podID="a042f16d-ba98-47c0-8f0e-cecb58dad951" containerID="064516be9cf9eb65600e2d882e09ad4f714b85e7ec269b3f3605a47856a5bdaa" exitCode=0
Sep 15 12:31:06.351456 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:06.351291    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1dc430/sub-test-model-1dc430-kserve-79cccdffd9-xg97q" event={"ID":"a042f16d-ba98-47c0-8f0e-cecb58dad951","Type":"ContainerDied","Data":"064516be9cf9eb65600e2d882e09ad4f714b85e7ec269b3f3605a47856a5bdaa"}
Sep 15 12:31:06.351456 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:06.351341    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1dc430/sub-test-model-1dc430-kserve-79cccdffd9-xg97q" event={"ID":"a042f16d-ba98-47c0-8f0e-cecb58dad951","Type":"ContainerDied","Data":"62be2bc4cbddf4261a46f8063c38e75de6ec32289940cf5e90cddd3457bd11a1"}
Sep 15 12:31:06.351456 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:06.351356    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="62be2bc4cbddf4261a46f8063c38e75de6ec32289940cf5e90cddd3457bd11a1"
Sep 15 12:31:06.357846 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:06.357822    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1dc430/sub-test-model-1dc430-kserve-79cccdffd9-xg97q"
Sep 15 12:31:06.463220 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:06.463163    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a042f16d-ba98-47c0-8f0e-cecb58dad951-model-cache\") pod \"a042f16d-ba98-47c0-8f0e-cecb58dad951\" (UID: \"a042f16d-ba98-47c0-8f0e-cecb58dad951\") "
Sep 15 12:31:06.463220 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:06.463233    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a042f16d-ba98-47c0-8f0e-cecb58dad951-tmp-dir\") pod \"a042f16d-ba98-47c0-8f0e-cecb58dad951\" (UID: \"a042f16d-ba98-47c0-8f0e-cecb58dad951\") "
Sep 15 12:31:06.463530 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:06.463304    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pj45c\" (UniqueName: \"kubernetes.io/projected/a042f16d-ba98-47c0-8f0e-cecb58dad951-kube-api-access-pj45c\") pod \"a042f16d-ba98-47c0-8f0e-cecb58dad951\" (UID: \"a042f16d-ba98-47c0-8f0e-cecb58dad951\") "
Sep 15 12:31:06.463530 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:06.463331    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a042f16d-ba98-47c0-8f0e-cecb58dad951-tls-certs\") pod \"a042f16d-ba98-47c0-8f0e-cecb58dad951\" (UID: \"a042f16d-ba98-47c0-8f0e-cecb58dad951\") "
Sep 15 12:31:06.463530 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:06.463434    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a042f16d-ba98-47c0-8f0e-cecb58dad951-home\") pod \"a042f16d-ba98-47c0-8f0e-cecb58dad951\" (UID: \"a042f16d-ba98-47c0-8f0e-cecb58dad951\") "
Sep 15 12:31:06.463530 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:06.463478    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a042f16d-ba98-47c0-8f0e-cecb58dad951-dshm\") pod \"a042f16d-ba98-47c0-8f0e-cecb58dad951\" (UID: \"a042f16d-ba98-47c0-8f0e-cecb58dad951\") "
Sep 15 12:31:06.463703 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:06.463554    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a042f16d-ba98-47c0-8f0e-cecb58dad951-model-cache" (OuterVolumeSpecName: "model-cache") pod "a042f16d-ba98-47c0-8f0e-cecb58dad951" (UID: "a042f16d-ba98-47c0-8f0e-cecb58dad951"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:31:06.463703 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:06.463547    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a042f16d-ba98-47c0-8f0e-cecb58dad951-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a042f16d-ba98-47c0-8f0e-cecb58dad951" (UID: "a042f16d-ba98-47c0-8f0e-cecb58dad951"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:31:06.463776 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:06.463726    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a042f16d-ba98-47c0-8f0e-cecb58dad951-home" (OuterVolumeSpecName: "home") pod "a042f16d-ba98-47c0-8f0e-cecb58dad951" (UID: "a042f16d-ba98-47c0-8f0e-cecb58dad951"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:31:06.463833 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:06.463815    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a042f16d-ba98-47c0-8f0e-cecb58dad951-model-cache\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:31:06.463870 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:06.463838    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a042f16d-ba98-47c0-8f0e-cecb58dad951-tmp-dir\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:31:06.465599 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:06.465573    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a042f16d-ba98-47c0-8f0e-cecb58dad951-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a042f16d-ba98-47c0-8f0e-cecb58dad951" (UID: "a042f16d-ba98-47c0-8f0e-cecb58dad951"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:31:06.465702 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:06.465610    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a042f16d-ba98-47c0-8f0e-cecb58dad951-kube-api-access-pj45c" (OuterVolumeSpecName: "kube-api-access-pj45c") pod "a042f16d-ba98-47c0-8f0e-cecb58dad951" (UID: "a042f16d-ba98-47c0-8f0e-cecb58dad951"). InnerVolumeSpecName "kube-api-access-pj45c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:31:06.465702 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:06.465661    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a042f16d-ba98-47c0-8f0e-cecb58dad951-dshm" (OuterVolumeSpecName: "dshm") pod "a042f16d-ba98-47c0-8f0e-cecb58dad951" (UID: "a042f16d-ba98-47c0-8f0e-cecb58dad951"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:31:06.564557 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:06.564513    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pj45c\" (UniqueName: \"kubernetes.io/projected/a042f16d-ba98-47c0-8f0e-cecb58dad951-kube-api-access-pj45c\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:31:06.564557 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:06.564547    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a042f16d-ba98-47c0-8f0e-cecb58dad951-tls-certs\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:31:06.564557 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:06.564558    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a042f16d-ba98-47c0-8f0e-cecb58dad951-home\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:31:06.564557 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:06.564566    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a042f16d-ba98-47c0-8f0e-cecb58dad951-dshm\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:31:06.959677 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:06.959639    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-969130/sub-test-model-969130-kserve-84bd666c5-svkgs" podUID="7da4aec8-d414-44c9-a02c-53120196f900" containerName="main" containerID="cri-o://3033a73d6bf758ea3b7b7430bb3cb885170227df45e2719dcc2ed910a93a1b49" gracePeriod=45
Sep 15 12:31:07.206761 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:07.206735    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-969130/sub-test-model-969130-kserve-84bd666c5-svkgs"
Sep 15 12:31:07.358975 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:07.358885    2555 generic.go:356] "Generic (PLEG): container finished" podID="7da4aec8-d414-44c9-a02c-53120196f900" containerID="3033a73d6bf758ea3b7b7430bb3cb885170227df45e2719dcc2ed910a93a1b49" exitCode=0
Sep 15 12:31:07.358975 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:07.358926    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-969130/sub-test-model-969130-kserve-84bd666c5-svkgs"
Sep 15 12:31:07.359223 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:07.358979    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-969130/sub-test-model-969130-kserve-84bd666c5-svkgs" event={"ID":"7da4aec8-d414-44c9-a02c-53120196f900","Type":"ContainerDied","Data":"3033a73d6bf758ea3b7b7430bb3cb885170227df45e2719dcc2ed910a93a1b49"}
Sep 15 12:31:07.359223 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:07.359025    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-969130/sub-test-model-969130-kserve-84bd666c5-svkgs" event={"ID":"7da4aec8-d414-44c9-a02c-53120196f900","Type":"ContainerDied","Data":"357b6e842b8f5dee962c7d1f058941a78fca056e27906b1c8a78de0dd0fc8a73"}
Sep 15 12:31:07.359223 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:07.359042    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1dc430/sub-test-model-1dc430-kserve-79cccdffd9-xg97q"
Sep 15 12:31:07.359223 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:07.359050    2555 scope.go:117] "RemoveContainer" containerID="3033a73d6bf758ea3b7b7430bb3cb885170227df45e2719dcc2ed910a93a1b49"
Sep 15 12:31:07.368370 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:07.368352    2555 scope.go:117] "RemoveContainer" containerID="3033a73d6bf758ea3b7b7430bb3cb885170227df45e2719dcc2ed910a93a1b49"
Sep 15 12:31:07.368624 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:31:07.368604    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3033a73d6bf758ea3b7b7430bb3cb885170227df45e2719dcc2ed910a93a1b49\": container with ID starting with 3033a73d6bf758ea3b7b7430bb3cb885170227df45e2719dcc2ed910a93a1b49 not found: ID does not exist" containerID="3033a73d6bf758ea3b7b7430bb3cb885170227df45e2719dcc2ed910a93a1b49"
Sep 15 12:31:07.368665 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:07.368635    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3033a73d6bf758ea3b7b7430bb3cb885170227df45e2719dcc2ed910a93a1b49"} err="failed to get container status \"3033a73d6bf758ea3b7b7430bb3cb885170227df45e2719dcc2ed910a93a1b49\": rpc error: code = NotFound desc = could not find container \"3033a73d6bf758ea3b7b7430bb3cb885170227df45e2719dcc2ed910a93a1b49\": container with ID starting with 3033a73d6bf758ea3b7b7430bb3cb885170227df45e2719dcc2ed910a93a1b49 not found: ID does not exist"
Sep 15 12:31:07.371951 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:07.371935    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7da4aec8-d414-44c9-a02c-53120196f900-tmp-dir\") pod \"7da4aec8-d414-44c9-a02c-53120196f900\" (UID: \"7da4aec8-d414-44c9-a02c-53120196f900\") "
Sep 15 12:31:07.372007 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:07.371965    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7da4aec8-d414-44c9-a02c-53120196f900-home\") pod \"7da4aec8-d414-44c9-a02c-53120196f900\" (UID: \"7da4aec8-d414-44c9-a02c-53120196f900\") "
Sep 15 12:31:07.372007 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:07.371991    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7da4aec8-d414-44c9-a02c-53120196f900-dshm\") pod \"7da4aec8-d414-44c9-a02c-53120196f900\" (UID: \"7da4aec8-d414-44c9-a02c-53120196f900\") "
Sep 15 12:31:07.372070 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:07.372012    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7da4aec8-d414-44c9-a02c-53120196f900-tls-certs\") pod \"7da4aec8-d414-44c9-a02c-53120196f900\" (UID: \"7da4aec8-d414-44c9-a02c-53120196f900\") "
Sep 15 12:31:07.372111 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:07.372073    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7da4aec8-d414-44c9-a02c-53120196f900-model-cache\") pod \"7da4aec8-d414-44c9-a02c-53120196f900\" (UID: \"7da4aec8-d414-44c9-a02c-53120196f900\") "
Sep 15 12:31:07.372111 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:07.372098    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2r4pg\" (UniqueName: \"kubernetes.io/projected/7da4aec8-d414-44c9-a02c-53120196f900-kube-api-access-2r4pg\") pod \"7da4aec8-d414-44c9-a02c-53120196f900\" (UID: \"7da4aec8-d414-44c9-a02c-53120196f900\") "
Sep 15 12:31:07.372249 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:07.372226    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7da4aec8-d414-44c9-a02c-53120196f900-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7da4aec8-d414-44c9-a02c-53120196f900" (UID: "7da4aec8-d414-44c9-a02c-53120196f900"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:31:07.372249 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:07.372239    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7da4aec8-d414-44c9-a02c-53120196f900-home" (OuterVolumeSpecName: "home") pod "7da4aec8-d414-44c9-a02c-53120196f900" (UID: "7da4aec8-d414-44c9-a02c-53120196f900"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:31:07.372392 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:07.372334    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7da4aec8-d414-44c9-a02c-53120196f900-model-cache" (OuterVolumeSpecName: "model-cache") pod "7da4aec8-d414-44c9-a02c-53120196f900" (UID: "7da4aec8-d414-44c9-a02c-53120196f900"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:31:07.372478 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:07.372463    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7da4aec8-d414-44c9-a02c-53120196f900-model-cache\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:31:07.372532 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:07.372483    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7da4aec8-d414-44c9-a02c-53120196f900-tmp-dir\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:31:07.372532 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:07.372496    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7da4aec8-d414-44c9-a02c-53120196f900-home\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:31:07.374187 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:07.374167    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7da4aec8-d414-44c9-a02c-53120196f900-dshm" (OuterVolumeSpecName: "dshm") pod "7da4aec8-d414-44c9-a02c-53120196f900" (UID: "7da4aec8-d414-44c9-a02c-53120196f900"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:31:07.374292 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:07.374209    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7da4aec8-d414-44c9-a02c-53120196f900-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7da4aec8-d414-44c9-a02c-53120196f900" (UID: "7da4aec8-d414-44c9-a02c-53120196f900"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:31:07.374292 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:07.374214    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7da4aec8-d414-44c9-a02c-53120196f900-kube-api-access-2r4pg" (OuterVolumeSpecName: "kube-api-access-2r4pg") pod "7da4aec8-d414-44c9-a02c-53120196f900" (UID: "7da4aec8-d414-44c9-a02c-53120196f900"). InnerVolumeSpecName "kube-api-access-2r4pg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:31:07.382666 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:07.382643    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1dc430/sub-test-model-1dc430-kserve-79cccdffd9-xg97q"]
Sep 15 12:31:07.386196 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:07.386171    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1dc430/sub-test-model-1dc430-kserve-79cccdffd9-xg97q"]
Sep 15 12:31:07.473376 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:07.473339    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7da4aec8-d414-44c9-a02c-53120196f900-dshm\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:31:07.473376 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:07.473366    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7da4aec8-d414-44c9-a02c-53120196f900-tls-certs\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:31:07.473376 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:07.473378    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2r4pg\" (UniqueName: \"kubernetes.io/projected/7da4aec8-d414-44c9-a02c-53120196f900-kube-api-access-2r4pg\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:31:07.684218 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:07.684190    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-969130/sub-test-model-969130-kserve-84bd666c5-svkgs"]
Sep 15 12:31:07.689912 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:07.689885    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-969130/sub-test-model-969130-kserve-84bd666c5-svkgs"]
Sep 15 12:31:07.816952 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:07.816916    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7da4aec8-d414-44c9-a02c-53120196f900" path="/var/lib/kubelet/pods/7da4aec8-d414-44c9-a02c-53120196f900/volumes"
Sep 15 12:31:07.817435 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:07.817415    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a042f16d-ba98-47c0-8f0e-cecb58dad951" path="/var/lib/kubelet/pods/a042f16d-ba98-47c0-8f0e-cecb58dad951/volumes"
Sep 15 12:31:08.266237 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:08.266197    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-1dc430/rate-test-model-1dc430-kserve-75c596b759-g6xch"
Sep 15 12:31:08.266237 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:08.266243    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-1dc430/rate-test-model-1dc430-kserve-75c596b759-g6xch"
Sep 15 12:31:08.278716 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:08.278686    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-1dc430/rate-test-model-1dc430-kserve-75c596b759-g6xch"
Sep 15 12:31:08.377257 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:08.377226    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-1dc430/rate-test-model-1dc430-kserve-75c596b759-g6xch"
Sep 15 12:31:35.213217 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:35.213176    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-969130/rate-test-model-969130-kserve-6944584587-xx8cq"]
Sep 15 12:31:35.213957 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:35.213934    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a042f16d-ba98-47c0-8f0e-cecb58dad951" containerName="main"
Sep 15 12:31:35.214044 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:35.213965    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a042f16d-ba98-47c0-8f0e-cecb58dad951" containerName="main"
Sep 15 12:31:35.214044 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:35.213983    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7da4aec8-d414-44c9-a02c-53120196f900" containerName="main"
Sep 15 12:31:35.214044 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:35.213991    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="7da4aec8-d414-44c9-a02c-53120196f900" containerName="main"
Sep 15 12:31:35.214178 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:35.214139    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="7da4aec8-d414-44c9-a02c-53120196f900" containerName="main"
Sep 15 12:31:35.214178 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:35.214153    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a042f16d-ba98-47c0-8f0e-cecb58dad951" containerName="main"
Sep 15 12:31:35.218143 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:35.218084    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-969130/rate-test-model-969130-kserve-6944584587-xx8cq"
Sep 15 12:31:35.222091 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:35.222022    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-969130\"/\"rate-test-model-969130-kserve-self-signed-certs\""
Sep 15 12:31:35.222091 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:35.222022    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-969130\"/\"openshift-service-ca.crt\""
Sep 15 12:31:35.222322 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:35.222056    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-969130\"/\"default-dockercfg-xf252\""
Sep 15 12:31:35.222478 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:35.222461    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-969130\"/\"kube-root-ca.crt\""
Sep 15 12:31:35.226007 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:35.225747    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-969130/rate-test-model-969130-kserve-6944584587-xx8cq"]
Sep 15 12:31:35.318664 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:35.318617    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fedfc000-7013-4c4c-a615-60987c5ab511-tls-certs\") pod \"rate-test-model-969130-kserve-6944584587-xx8cq\" (UID: \"fedfc000-7013-4c4c-a615-60987c5ab511\") " pod="ai-tenant-e2e-shared-b-w3-969130/rate-test-model-969130-kserve-6944584587-xx8cq"
Sep 15 12:31:35.318846 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:35.318680    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fedfc000-7013-4c4c-a615-60987c5ab511-home\") pod \"rate-test-model-969130-kserve-6944584587-xx8cq\" (UID: \"fedfc000-7013-4c4c-a615-60987c5ab511\") " pod="ai-tenant-e2e-shared-b-w3-969130/rate-test-model-969130-kserve-6944584587-xx8cq"
Sep 15 12:31:35.318846 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:35.318769    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fedfc000-7013-4c4c-a615-60987c5ab511-model-cache\") pod \"rate-test-model-969130-kserve-6944584587-xx8cq\" (UID: \"fedfc000-7013-4c4c-a615-60987c5ab511\") " pod="ai-tenant-e2e-shared-b-w3-969130/rate-test-model-969130-kserve-6944584587-xx8cq"
Sep 15 12:31:35.318846 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:35.318804    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtmw4\" (UniqueName: \"kubernetes.io/projected/fedfc000-7013-4c4c-a615-60987c5ab511-kube-api-access-jtmw4\") pod \"rate-test-model-969130-kserve-6944584587-xx8cq\" (UID: \"fedfc000-7013-4c4c-a615-60987c5ab511\") " pod="ai-tenant-e2e-shared-b-w3-969130/rate-test-model-969130-kserve-6944584587-xx8cq"
Sep 15 12:31:35.318846 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:35.318833    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fedfc000-7013-4c4c-a615-60987c5ab511-dshm\") pod \"rate-test-model-969130-kserve-6944584587-xx8cq\" (UID: \"fedfc000-7013-4c4c-a615-60987c5ab511\") " pod="ai-tenant-e2e-shared-b-w3-969130/rate-test-model-969130-kserve-6944584587-xx8cq"
Sep 15 12:31:35.318996 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:35.318868    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fedfc000-7013-4c4c-a615-60987c5ab511-tmp-dir\") pod \"rate-test-model-969130-kserve-6944584587-xx8cq\" (UID: \"fedfc000-7013-4c4c-a615-60987c5ab511\") " pod="ai-tenant-e2e-shared-b-w3-969130/rate-test-model-969130-kserve-6944584587-xx8cq"
Sep 15 12:31:35.419655 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:35.419613    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fedfc000-7013-4c4c-a615-60987c5ab511-dshm\") pod \"rate-test-model-969130-kserve-6944584587-xx8cq\" (UID: \"fedfc000-7013-4c4c-a615-60987c5ab511\") " pod="ai-tenant-e2e-shared-b-w3-969130/rate-test-model-969130-kserve-6944584587-xx8cq"
Sep 15 12:31:35.419867 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:35.419665    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fedfc000-7013-4c4c-a615-60987c5ab511-tmp-dir\") pod \"rate-test-model-969130-kserve-6944584587-xx8cq\" (UID: \"fedfc000-7013-4c4c-a615-60987c5ab511\") " pod="ai-tenant-e2e-shared-b-w3-969130/rate-test-model-969130-kserve-6944584587-xx8cq"
Sep 15 12:31:35.419867 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:35.419703    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fedfc000-7013-4c4c-a615-60987c5ab511-tls-certs\") pod \"rate-test-model-969130-kserve-6944584587-xx8cq\" (UID: \"fedfc000-7013-4c4c-a615-60987c5ab511\") " pod="ai-tenant-e2e-shared-b-w3-969130/rate-test-model-969130-kserve-6944584587-xx8cq"
Sep 15 12:31:35.419867 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:35.419735    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fedfc000-7013-4c4c-a615-60987c5ab511-home\") pod \"rate-test-model-969130-kserve-6944584587-xx8cq\" (UID: \"fedfc000-7013-4c4c-a615-60987c5ab511\") " pod="ai-tenant-e2e-shared-b-w3-969130/rate-test-model-969130-kserve-6944584587-xx8cq"
Sep 15 12:31:35.419867 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:35.419769    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fedfc000-7013-4c4c-a615-60987c5ab511-model-cache\") pod \"rate-test-model-969130-kserve-6944584587-xx8cq\" (UID: \"fedfc000-7013-4c4c-a615-60987c5ab511\") " pod="ai-tenant-e2e-shared-b-w3-969130/rate-test-model-969130-kserve-6944584587-xx8cq"
Sep 15 12:31:35.419867 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:35.419793    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtmw4\" (UniqueName: \"kubernetes.io/projected/fedfc000-7013-4c4c-a615-60987c5ab511-kube-api-access-jtmw4\") pod \"rate-test-model-969130-kserve-6944584587-xx8cq\" (UID: \"fedfc000-7013-4c4c-a615-60987c5ab511\") " pod="ai-tenant-e2e-shared-b-w3-969130/rate-test-model-969130-kserve-6944584587-xx8cq"
Sep 15 12:31:35.420177 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:35.420154    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fedfc000-7013-4c4c-a615-60987c5ab511-tmp-dir\") pod \"rate-test-model-969130-kserve-6944584587-xx8cq\" (UID: \"fedfc000-7013-4c4c-a615-60987c5ab511\") " pod="ai-tenant-e2e-shared-b-w3-969130/rate-test-model-969130-kserve-6944584587-xx8cq"
Sep 15 12:31:35.420237 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:35.420204    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fedfc000-7013-4c4c-a615-60987c5ab511-model-cache\") pod \"rate-test-model-969130-kserve-6944584587-xx8cq\" (UID: \"fedfc000-7013-4c4c-a615-60987c5ab511\") " pod="ai-tenant-e2e-shared-b-w3-969130/rate-test-model-969130-kserve-6944584587-xx8cq"
Sep 15 12:31:35.420315 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:35.420230    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fedfc000-7013-4c4c-a615-60987c5ab511-home\") pod \"rate-test-model-969130-kserve-6944584587-xx8cq\" (UID: \"fedfc000-7013-4c4c-a615-60987c5ab511\") " pod="ai-tenant-e2e-shared-b-w3-969130/rate-test-model-969130-kserve-6944584587-xx8cq"
Sep 15 12:31:35.422048 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:35.422028    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fedfc000-7013-4c4c-a615-60987c5ab511-dshm\") pod \"rate-test-model-969130-kserve-6944584587-xx8cq\" (UID: \"fedfc000-7013-4c4c-a615-60987c5ab511\") " pod="ai-tenant-e2e-shared-b-w3-969130/rate-test-model-969130-kserve-6944584587-xx8cq"
Sep 15 12:31:35.422205 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:35.422189    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fedfc000-7013-4c4c-a615-60987c5ab511-tls-certs\") pod \"rate-test-model-969130-kserve-6944584587-xx8cq\" (UID: \"fedfc000-7013-4c4c-a615-60987c5ab511\") " pod="ai-tenant-e2e-shared-b-w3-969130/rate-test-model-969130-kserve-6944584587-xx8cq"
Sep 15 12:31:35.428975 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:35.428947    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtmw4\" (UniqueName: \"kubernetes.io/projected/fedfc000-7013-4c4c-a615-60987c5ab511-kube-api-access-jtmw4\") pod \"rate-test-model-969130-kserve-6944584587-xx8cq\" (UID: \"fedfc000-7013-4c4c-a615-60987c5ab511\") " pod="ai-tenant-e2e-shared-b-w3-969130/rate-test-model-969130-kserve-6944584587-xx8cq"
Sep 15 12:31:35.529095 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:35.529010    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-969130/rate-test-model-969130-kserve-6944584587-xx8cq"
Sep 15 12:31:35.805395 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:35.805365    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-969130/rate-test-model-969130-kserve-6944584587-xx8cq"]
Sep 15 12:31:35.807159 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:31:35.807123    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfedfc000_7013_4c4c_a615_60987c5ab511.slice/crio-a4e0aab4e919562f96a84ac3b129387a2e7630895b9af51f8cc2752ca02b8828 WatchSource:0}: Error finding container a4e0aab4e919562f96a84ac3b129387a2e7630895b9af51f8cc2752ca02b8828: Status 404 returned error can't find the container with id a4e0aab4e919562f96a84ac3b129387a2e7630895b9af51f8cc2752ca02b8828
Sep 15 12:31:36.522609 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:36.522566    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-969130/rate-test-model-969130-kserve-6944584587-xx8cq" event={"ID":"fedfc000-7013-4c4c-a615-60987c5ab511","Type":"ContainerStarted","Data":"5497b323f52ba63e62c97d7814101ee2371a1f66f33d6d8e01259c12f4d0af02"}
Sep 15 12:31:36.522609 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:36.522614    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-969130/rate-test-model-969130-kserve-6944584587-xx8cq" event={"ID":"fedfc000-7013-4c4c-a615-60987c5ab511","Type":"ContainerStarted","Data":"a4e0aab4e919562f96a84ac3b129387a2e7630895b9af51f8cc2752ca02b8828"}
Sep 15 12:31:36.542185 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:36.542129    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-969130/rate-test-model-969130-kserve-6944584587-xx8cq" podStartSLOduration=1.542112522 podStartE2EDuration="1.542112522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:31:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:31:36.541203574 +0000 UTC m=+1753.332126089" watchObservedRunningTime="2026-09-15 12:31:36.542112522 +0000 UTC m=+1753.333035046"
Sep 15 12:31:37.469637 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:37.469595    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5gkxs"]
Sep 15 12:31:45.529223 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:45.529182    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-969130/rate-test-model-969130-kserve-6944584587-xx8cq"
Sep 15 12:31:45.529678 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:45.529236    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-969130/rate-test-model-969130-kserve-6944584587-xx8cq"
Sep 15 12:31:45.542039 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:45.542012    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-969130/rate-test-model-969130-kserve-6944584587-xx8cq"
Sep 15 12:31:45.585237 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:45.585209    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-969130/rate-test-model-969130-kserve-6944584587-xx8cq"
Sep 15 12:31:55.674016 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:31:55.673983    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5gkxs"]
Sep 15 12:32:06.169985 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:06.169949    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5gkxs"]
Sep 15 12:32:12.285905 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:12.285864    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1dc430/rate-test-model-1dc430-kserve-75c596b759-g6xch"]
Sep 15 12:32:13.196199 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:13.196164    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-969130/rate-test-model-969130-kserve-6944584587-xx8cq"]
Sep 15 12:32:21.576638 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:21.576594    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5gkxs"]
Sep 15 12:32:27.305108 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:27.305039    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-1dc430/rate-test-model-1dc430-kserve-75c596b759-g6xch" podUID="7ec07ee0-5376-4887-bff7-a0d76804939d" containerName="main" containerID="cri-o://0073268a18f5c2ee3719d034a3d6c3fdd8ee9327017a1657749e1b58229287d7" gracePeriod=45
Sep 15 12:32:27.565666 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:27.565596    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1dc430/rate-test-model-1dc430-kserve-75c596b759-g6xch"
Sep 15 12:32:27.706139 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:27.706101    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ec07ee0-5376-4887-bff7-a0d76804939d-model-cache\") pod \"7ec07ee0-5376-4887-bff7-a0d76804939d\" (UID: \"7ec07ee0-5376-4887-bff7-a0d76804939d\") "
Sep 15 12:32:27.706375 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:27.706176    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ec07ee0-5376-4887-bff7-a0d76804939d-tls-certs\") pod \"7ec07ee0-5376-4887-bff7-a0d76804939d\" (UID: \"7ec07ee0-5376-4887-bff7-a0d76804939d\") "
Sep 15 12:32:27.706375 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:27.706198    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kxlvh\" (UniqueName: \"kubernetes.io/projected/7ec07ee0-5376-4887-bff7-a0d76804939d-kube-api-access-kxlvh\") pod \"7ec07ee0-5376-4887-bff7-a0d76804939d\" (UID: \"7ec07ee0-5376-4887-bff7-a0d76804939d\") "
Sep 15 12:32:27.706375 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:27.706250    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ec07ee0-5376-4887-bff7-a0d76804939d-home\") pod \"7ec07ee0-5376-4887-bff7-a0d76804939d\" (UID: \"7ec07ee0-5376-4887-bff7-a0d76804939d\") "
Sep 15 12:32:27.706375 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:27.706298    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ec07ee0-5376-4887-bff7-a0d76804939d-tmp-dir\") pod \"7ec07ee0-5376-4887-bff7-a0d76804939d\" (UID: \"7ec07ee0-5376-4887-bff7-a0d76804939d\") "
Sep 15 12:32:27.706375 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:27.706371    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ec07ee0-5376-4887-bff7-a0d76804939d-dshm\") pod \"7ec07ee0-5376-4887-bff7-a0d76804939d\" (UID: \"7ec07ee0-5376-4887-bff7-a0d76804939d\") "
Sep 15 12:32:27.706662 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:27.706483    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ec07ee0-5376-4887-bff7-a0d76804939d-model-cache" (OuterVolumeSpecName: "model-cache") pod "7ec07ee0-5376-4887-bff7-a0d76804939d" (UID: "7ec07ee0-5376-4887-bff7-a0d76804939d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:32:27.706662 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:27.706585    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ec07ee0-5376-4887-bff7-a0d76804939d-home" (OuterVolumeSpecName: "home") pod "7ec07ee0-5376-4887-bff7-a0d76804939d" (UID: "7ec07ee0-5376-4887-bff7-a0d76804939d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:32:27.706662 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:27.706618    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ec07ee0-5376-4887-bff7-a0d76804939d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7ec07ee0-5376-4887-bff7-a0d76804939d" (UID: "7ec07ee0-5376-4887-bff7-a0d76804939d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:32:27.706806 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:27.706733    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ec07ee0-5376-4887-bff7-a0d76804939d-model-cache\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:32:27.706806 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:27.706752    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ec07ee0-5376-4887-bff7-a0d76804939d-home\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:32:27.706806 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:27.706766    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ec07ee0-5376-4887-bff7-a0d76804939d-tmp-dir\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:32:27.708404 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:27.708383    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ec07ee0-5376-4887-bff7-a0d76804939d-dshm" (OuterVolumeSpecName: "dshm") pod "7ec07ee0-5376-4887-bff7-a0d76804939d" (UID: "7ec07ee0-5376-4887-bff7-a0d76804939d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:32:27.708480 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:27.708444    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ec07ee0-5376-4887-bff7-a0d76804939d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7ec07ee0-5376-4887-bff7-a0d76804939d" (UID: "7ec07ee0-5376-4887-bff7-a0d76804939d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:32:27.708525 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:27.708506    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ec07ee0-5376-4887-bff7-a0d76804939d-kube-api-access-kxlvh" (OuterVolumeSpecName: "kube-api-access-kxlvh") pod "7ec07ee0-5376-4887-bff7-a0d76804939d" (UID: "7ec07ee0-5376-4887-bff7-a0d76804939d"). InnerVolumeSpecName "kube-api-access-kxlvh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:32:27.807337 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:27.807303    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ec07ee0-5376-4887-bff7-a0d76804939d-dshm\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:32:27.807337 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:27.807329    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ec07ee0-5376-4887-bff7-a0d76804939d-tls-certs\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:32:27.807337 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:27.807340    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxlvh\" (UniqueName: \"kubernetes.io/projected/7ec07ee0-5376-4887-bff7-a0d76804939d-kube-api-access-kxlvh\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:32:27.809527 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:27.809500    2555 generic.go:356] "Generic (PLEG): container finished" podID="7ec07ee0-5376-4887-bff7-a0d76804939d" containerID="0073268a18f5c2ee3719d034a3d6c3fdd8ee9327017a1657749e1b58229287d7" exitCode=0
Sep 15 12:32:27.809635 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:27.809561    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1dc430/rate-test-model-1dc430-kserve-75c596b759-g6xch"
Sep 15 12:32:27.809635 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:27.809590    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1dc430/rate-test-model-1dc430-kserve-75c596b759-g6xch" event={"ID":"7ec07ee0-5376-4887-bff7-a0d76804939d","Type":"ContainerDied","Data":"0073268a18f5c2ee3719d034a3d6c3fdd8ee9327017a1657749e1b58229287d7"}
Sep 15 12:32:27.809710 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:27.809638    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1dc430/rate-test-model-1dc430-kserve-75c596b759-g6xch" event={"ID":"7ec07ee0-5376-4887-bff7-a0d76804939d","Type":"ContainerDied","Data":"981f2b26afd7723d13e15ffe05bdbb956534fb3ca3b40a000d23584abf8c1231"}
Sep 15 12:32:27.809710 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:27.809661    2555 scope.go:117] "RemoveContainer" containerID="0073268a18f5c2ee3719d034a3d6c3fdd8ee9327017a1657749e1b58229287d7"
Sep 15 12:32:27.823352 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:27.823303    2555 scope.go:117] "RemoveContainer" containerID="0073268a18f5c2ee3719d034a3d6c3fdd8ee9327017a1657749e1b58229287d7"
Sep 15 12:32:27.823875 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:32:27.823852    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0073268a18f5c2ee3719d034a3d6c3fdd8ee9327017a1657749e1b58229287d7\": container with ID starting with 0073268a18f5c2ee3719d034a3d6c3fdd8ee9327017a1657749e1b58229287d7 not found: ID does not exist" containerID="0073268a18f5c2ee3719d034a3d6c3fdd8ee9327017a1657749e1b58229287d7"
Sep 15 12:32:27.823966 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:27.823885    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0073268a18f5c2ee3719d034a3d6c3fdd8ee9327017a1657749e1b58229287d7"} err="failed to get container status \"0073268a18f5c2ee3719d034a3d6c3fdd8ee9327017a1657749e1b58229287d7\": rpc error: code = NotFound desc = could not find container \"0073268a18f5c2ee3719d034a3d6c3fdd8ee9327017a1657749e1b58229287d7\": container with ID starting with 0073268a18f5c2ee3719d034a3d6c3fdd8ee9327017a1657749e1b58229287d7 not found: ID does not exist"
Sep 15 12:32:27.836320 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:27.836293    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1dc430/rate-test-model-1dc430-kserve-75c596b759-g6xch"]
Sep 15 12:32:27.839581 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:27.839560    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1dc430/rate-test-model-1dc430-kserve-75c596b759-g6xch"]
Sep 15 12:32:28.215563 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:28.215493    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-969130/rate-test-model-969130-kserve-6944584587-xx8cq" podUID="fedfc000-7013-4c4c-a615-60987c5ab511" containerName="main" containerID="cri-o://5497b323f52ba63e62c97d7814101ee2371a1f66f33d6d8e01259c12f4d0af02" gracePeriod=45
Sep 15 12:32:28.458862 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:28.458836    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-969130/rate-test-model-969130-kserve-6944584587-xx8cq"
Sep 15 12:32:28.615039 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:28.614960    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fedfc000-7013-4c4c-a615-60987c5ab511-tls-certs\") pod \"fedfc000-7013-4c4c-a615-60987c5ab511\" (UID: \"fedfc000-7013-4c4c-a615-60987c5ab511\") "
Sep 15 12:32:28.615039 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:28.615001    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fedfc000-7013-4c4c-a615-60987c5ab511-model-cache\") pod \"fedfc000-7013-4c4c-a615-60987c5ab511\" (UID: \"fedfc000-7013-4c4c-a615-60987c5ab511\") "
Sep 15 12:32:28.615039 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:28.615023    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jtmw4\" (UniqueName: \"kubernetes.io/projected/fedfc000-7013-4c4c-a615-60987c5ab511-kube-api-access-jtmw4\") pod \"fedfc000-7013-4c4c-a615-60987c5ab511\" (UID: \"fedfc000-7013-4c4c-a615-60987c5ab511\") "
Sep 15 12:32:28.615346 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:28.615055    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fedfc000-7013-4c4c-a615-60987c5ab511-home\") pod \"fedfc000-7013-4c4c-a615-60987c5ab511\" (UID: \"fedfc000-7013-4c4c-a615-60987c5ab511\") "
Sep 15 12:32:28.615346 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:28.615238    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fedfc000-7013-4c4c-a615-60987c5ab511-tmp-dir\") pod \"fedfc000-7013-4c4c-a615-60987c5ab511\" (UID: \"fedfc000-7013-4c4c-a615-60987c5ab511\") "
Sep 15 12:32:28.615346 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:28.615234    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fedfc000-7013-4c4c-a615-60987c5ab511-model-cache" (OuterVolumeSpecName: "model-cache") pod "fedfc000-7013-4c4c-a615-60987c5ab511" (UID: "fedfc000-7013-4c4c-a615-60987c5ab511"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:32:28.615346 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:28.615289    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fedfc000-7013-4c4c-a615-60987c5ab511-home" (OuterVolumeSpecName: "home") pod "fedfc000-7013-4c4c-a615-60987c5ab511" (UID: "fedfc000-7013-4c4c-a615-60987c5ab511"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:32:28.615346 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:28.615317    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fedfc000-7013-4c4c-a615-60987c5ab511-dshm\") pod \"fedfc000-7013-4c4c-a615-60987c5ab511\" (UID: \"fedfc000-7013-4c4c-a615-60987c5ab511\") "
Sep 15 12:32:28.615596 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:28.615422    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fedfc000-7013-4c4c-a615-60987c5ab511-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fedfc000-7013-4c4c-a615-60987c5ab511" (UID: "fedfc000-7013-4c4c-a615-60987c5ab511"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:32:28.615653 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:28.615598    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fedfc000-7013-4c4c-a615-60987c5ab511-tmp-dir\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:32:28.615653 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:28.615616    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fedfc000-7013-4c4c-a615-60987c5ab511-model-cache\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:32:28.615653 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:28.615630    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fedfc000-7013-4c4c-a615-60987c5ab511-home\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:32:28.617133 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:28.617114    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fedfc000-7013-4c4c-a615-60987c5ab511-dshm" (OuterVolumeSpecName: "dshm") pod "fedfc000-7013-4c4c-a615-60987c5ab511" (UID: "fedfc000-7013-4c4c-a615-60987c5ab511"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:32:28.617198 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:28.617152    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fedfc000-7013-4c4c-a615-60987c5ab511-kube-api-access-jtmw4" (OuterVolumeSpecName: "kube-api-access-jtmw4") pod "fedfc000-7013-4c4c-a615-60987c5ab511" (UID: "fedfc000-7013-4c4c-a615-60987c5ab511"). InnerVolumeSpecName "kube-api-access-jtmw4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:32:28.617198 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:28.617176    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fedfc000-7013-4c4c-a615-60987c5ab511-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fedfc000-7013-4c4c-a615-60987c5ab511" (UID: "fedfc000-7013-4c4c-a615-60987c5ab511"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:32:28.716114 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:28.716076    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fedfc000-7013-4c4c-a615-60987c5ab511-dshm\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:32:28.716114 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:28.716106    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fedfc000-7013-4c4c-a615-60987c5ab511-tls-certs\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:32:28.716114 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:28.716116    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtmw4\" (UniqueName: \"kubernetes.io/projected/fedfc000-7013-4c4c-a615-60987c5ab511-kube-api-access-jtmw4\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:32:28.817359 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:28.817324    2555 generic.go:356] "Generic (PLEG): container finished" podID="fedfc000-7013-4c4c-a615-60987c5ab511" containerID="5497b323f52ba63e62c97d7814101ee2371a1f66f33d6d8e01259c12f4d0af02" exitCode=0
Sep 15 12:32:28.817566 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:28.817367    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-969130/rate-test-model-969130-kserve-6944584587-xx8cq"
Sep 15 12:32:28.817566 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:28.817410    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-969130/rate-test-model-969130-kserve-6944584587-xx8cq" event={"ID":"fedfc000-7013-4c4c-a615-60987c5ab511","Type":"ContainerDied","Data":"5497b323f52ba63e62c97d7814101ee2371a1f66f33d6d8e01259c12f4d0af02"}
Sep 15 12:32:28.817566 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:28.817451    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-969130/rate-test-model-969130-kserve-6944584587-xx8cq" event={"ID":"fedfc000-7013-4c4c-a615-60987c5ab511","Type":"ContainerDied","Data":"a4e0aab4e919562f96a84ac3b129387a2e7630895b9af51f8cc2752ca02b8828"}
Sep 15 12:32:28.817566 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:28.817470    2555 scope.go:117] "RemoveContainer" containerID="5497b323f52ba63e62c97d7814101ee2371a1f66f33d6d8e01259c12f4d0af02"
Sep 15 12:32:28.827320 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:28.827304    2555 scope.go:117] "RemoveContainer" containerID="5497b323f52ba63e62c97d7814101ee2371a1f66f33d6d8e01259c12f4d0af02"
Sep 15 12:32:28.827587 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:32:28.827566    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5497b323f52ba63e62c97d7814101ee2371a1f66f33d6d8e01259c12f4d0af02\": container with ID starting with 5497b323f52ba63e62c97d7814101ee2371a1f66f33d6d8e01259c12f4d0af02 not found: ID does not exist" containerID="5497b323f52ba63e62c97d7814101ee2371a1f66f33d6d8e01259c12f4d0af02"
Sep 15 12:32:28.827667 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:28.827600    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5497b323f52ba63e62c97d7814101ee2371a1f66f33d6d8e01259c12f4d0af02"} err="failed to get container status \"5497b323f52ba63e62c97d7814101ee2371a1f66f33d6d8e01259c12f4d0af02\": rpc error: code = NotFound desc = could not find container \"5497b323f52ba63e62c97d7814101ee2371a1f66f33d6d8e01259c12f4d0af02\": container with ID starting with 5497b323f52ba63e62c97d7814101ee2371a1f66f33d6d8e01259c12f4d0af02 not found: ID does not exist"
Sep 15 12:32:28.839600 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:28.839568    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-969130/rate-test-model-969130-kserve-6944584587-xx8cq"]
Sep 15 12:32:28.843485 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:28.843460    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-969130/rate-test-model-969130-kserve-6944584587-xx8cq"]
Sep 15 12:32:29.817317 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:29.817256    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ec07ee0-5376-4887-bff7-a0d76804939d" path="/var/lib/kubelet/pods/7ec07ee0-5376-4887-bff7-a0d76804939d/volumes"
Sep 15 12:32:29.817882 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:29.817859    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fedfc000-7013-4c4c-a615-60987c5ab511" path="/var/lib/kubelet/pods/fedfc000-7013-4c4c-a615-60987c5ab511/volumes"
Sep 15 12:32:42.895749 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:42.895708    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-7b204e-openshift-default-55685c44c6-l58rv"]
Sep 15 12:32:42.896192 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:42.896170    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ec07ee0-5376-4887-bff7-a0d76804939d" containerName="main"
Sep 15 12:32:42.896192 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:42.896186    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ec07ee0-5376-4887-bff7-a0d76804939d" containerName="main"
Sep 15 12:32:42.896287 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:42.896195    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fedfc000-7013-4c4c-a615-60987c5ab511" containerName="main"
Sep 15 12:32:42.896287 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:42.896201    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="fedfc000-7013-4c4c-a615-60987c5ab511" containerName="main"
Sep 15 12:32:42.896383 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:42.896294    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="fedfc000-7013-4c4c-a615-60987c5ab511" containerName="main"
Sep 15 12:32:42.896383 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:42.896304    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ec07ee0-5376-4887-bff7-a0d76804939d" containerName="main"
Sep 15 12:32:42.900657 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:42.900639    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-7b204e-openshift-default-55685c44c6-l58rv"
Sep 15 12:32:42.903524 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:42.903500    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 15 12:32:42.903637 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:42.903526    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-7b204e-openshift-default-dockercfg-725bm\""
Sep 15 12:32:42.913062 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:42.913032    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-7b204e-openshift-default-55685c44c6-l58rv"]
Sep 15 12:32:43.053471 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:43.053426    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e2ec4b3-4578-47a6-b505-ba920b1fa4d0-workload-socket\") pod \"e2e-ipp-b-7b204e-openshift-default-55685c44c6-l58rv\" (UID: \"4e2ec4b3-4578-47a6-b505-ba920b1fa4d0\") " pod="openshift-ingress/e2e-ipp-b-7b204e-openshift-default-55685c44c6-l58rv"
Sep 15 12:32:43.053471 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:43.053470    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e2ec4b3-4578-47a6-b505-ba920b1fa4d0-credential-socket\") pod \"e2e-ipp-b-7b204e-openshift-default-55685c44c6-l58rv\" (UID: \"4e2ec4b3-4578-47a6-b505-ba920b1fa4d0\") " pod="openshift-ingress/e2e-ipp-b-7b204e-openshift-default-55685c44c6-l58rv"
Sep 15 12:32:43.053835 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:43.053498    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e2ec4b3-4578-47a6-b505-ba920b1fa4d0-istiod-ca-cert\") pod \"e2e-ipp-b-7b204e-openshift-default-55685c44c6-l58rv\" (UID: \"4e2ec4b3-4578-47a6-b505-ba920b1fa4d0\") " pod="openshift-ingress/e2e-ipp-b-7b204e-openshift-default-55685c44c6-l58rv"
Sep 15 12:32:43.053835 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:43.053537    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e2ec4b3-4578-47a6-b505-ba920b1fa4d0-istio-podinfo\") pod \"e2e-ipp-b-7b204e-openshift-default-55685c44c6-l58rv\" (UID: \"4e2ec4b3-4578-47a6-b505-ba920b1fa4d0\") " pod="openshift-ingress/e2e-ipp-b-7b204e-openshift-default-55685c44c6-l58rv"
Sep 15 12:32:43.053835 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:43.053555    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e2ec4b3-4578-47a6-b505-ba920b1fa4d0-istio-token\") pod \"e2e-ipp-b-7b204e-openshift-default-55685c44c6-l58rv\" (UID: \"4e2ec4b3-4578-47a6-b505-ba920b1fa4d0\") " pod="openshift-ingress/e2e-ipp-b-7b204e-openshift-default-55685c44c6-l58rv"
Sep 15 12:32:43.053835 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:43.053573    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e2ec4b3-4578-47a6-b505-ba920b1fa4d0-workload-certs\") pod \"e2e-ipp-b-7b204e-openshift-default-55685c44c6-l58rv\" (UID: \"4e2ec4b3-4578-47a6-b505-ba920b1fa4d0\") " pod="openshift-ingress/e2e-ipp-b-7b204e-openshift-default-55685c44c6-l58rv"
Sep 15 12:32:43.053835 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:43.053649    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hm7sj\" (UniqueName: \"kubernetes.io/projected/4e2ec4b3-4578-47a6-b505-ba920b1fa4d0-kube-api-access-hm7sj\") pod \"e2e-ipp-b-7b204e-openshift-default-55685c44c6-l58rv\" (UID: \"4e2ec4b3-4578-47a6-b505-ba920b1fa4d0\") " pod="openshift-ingress/e2e-ipp-b-7b204e-openshift-default-55685c44c6-l58rv"
Sep 15 12:32:43.053835 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:43.053720    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e2ec4b3-4578-47a6-b505-ba920b1fa4d0-istio-envoy\") pod \"e2e-ipp-b-7b204e-openshift-default-55685c44c6-l58rv\" (UID: \"4e2ec4b3-4578-47a6-b505-ba920b1fa4d0\") " pod="openshift-ingress/e2e-ipp-b-7b204e-openshift-default-55685c44c6-l58rv"
Sep 15 12:32:43.053835 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:43.053761    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e2ec4b3-4578-47a6-b505-ba920b1fa4d0-istio-data\") pod \"e2e-ipp-b-7b204e-openshift-default-55685c44c6-l58rv\" (UID: \"4e2ec4b3-4578-47a6-b505-ba920b1fa4d0\") " pod="openshift-ingress/e2e-ipp-b-7b204e-openshift-default-55685c44c6-l58rv"
Sep 15 12:32:43.154341 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:43.154237    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e2ec4b3-4578-47a6-b505-ba920b1fa4d0-istio-envoy\") pod \"e2e-ipp-b-7b204e-openshift-default-55685c44c6-l58rv\" (UID: \"4e2ec4b3-4578-47a6-b505-ba920b1fa4d0\") " pod="openshift-ingress/e2e-ipp-b-7b204e-openshift-default-55685c44c6-l58rv"
Sep 15 12:32:43.154341 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:43.154329    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e2ec4b3-4578-47a6-b505-ba920b1fa4d0-istio-data\") pod \"e2e-ipp-b-7b204e-openshift-default-55685c44c6-l58rv\" (UID: \"4e2ec4b3-4578-47a6-b505-ba920b1fa4d0\") " pod="openshift-ingress/e2e-ipp-b-7b204e-openshift-default-55685c44c6-l58rv"
Sep 15 12:32:43.154544 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:43.154371    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e2ec4b3-4578-47a6-b505-ba920b1fa4d0-workload-socket\") pod \"e2e-ipp-b-7b204e-openshift-default-55685c44c6-l58rv\" (UID: \"4e2ec4b3-4578-47a6-b505-ba920b1fa4d0\") " pod="openshift-ingress/e2e-ipp-b-7b204e-openshift-default-55685c44c6-l58rv"
Sep 15 12:32:43.154544 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:43.154404    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e2ec4b3-4578-47a6-b505-ba920b1fa4d0-credential-socket\") pod \"e2e-ipp-b-7b204e-openshift-default-55685c44c6-l58rv\" (UID: \"4e2ec4b3-4578-47a6-b505-ba920b1fa4d0\") " pod="openshift-ingress/e2e-ipp-b-7b204e-openshift-default-55685c44c6-l58rv"
Sep 15 12:32:43.154544 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:43.154439    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e2ec4b3-4578-47a6-b505-ba920b1fa4d0-istiod-ca-cert\") pod \"e2e-ipp-b-7b204e-openshift-default-55685c44c6-l58rv\" (UID: \"4e2ec4b3-4578-47a6-b505-ba920b1fa4d0\") " pod="openshift-ingress/e2e-ipp-b-7b204e-openshift-default-55685c44c6-l58rv"
Sep 15 12:32:43.154544 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:43.154475    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e2ec4b3-4578-47a6-b505-ba920b1fa4d0-istio-podinfo\") pod \"e2e-ipp-b-7b204e-openshift-default-55685c44c6-l58rv\" (UID: \"4e2ec4b3-4578-47a6-b505-ba920b1fa4d0\") " pod="openshift-ingress/e2e-ipp-b-7b204e-openshift-default-55685c44c6-l58rv"
Sep 15 12:32:43.154544 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:43.154530    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e2ec4b3-4578-47a6-b505-ba920b1fa4d0-istio-token\") pod \"e2e-ipp-b-7b204e-openshift-default-55685c44c6-l58rv\" (UID: \"4e2ec4b3-4578-47a6-b505-ba920b1fa4d0\") " pod="openshift-ingress/e2e-ipp-b-7b204e-openshift-default-55685c44c6-l58rv"
Sep 15 12:32:43.154792 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:43.154567    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e2ec4b3-4578-47a6-b505-ba920b1fa4d0-workload-certs\") pod \"e2e-ipp-b-7b204e-openshift-default-55685c44c6-l58rv\" (UID: \"4e2ec4b3-4578-47a6-b505-ba920b1fa4d0\") " pod="openshift-ingress/e2e-ipp-b-7b204e-openshift-default-55685c44c6-l58rv"
Sep 15 12:32:43.154792 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:43.154622    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hm7sj\" (UniqueName: \"kubernetes.io/projected/4e2ec4b3-4578-47a6-b505-ba920b1fa4d0-kube-api-access-hm7sj\") pod \"e2e-ipp-b-7b204e-openshift-default-55685c44c6-l58rv\" (UID: \"4e2ec4b3-4578-47a6-b505-ba920b1fa4d0\") " pod="openshift-ingress/e2e-ipp-b-7b204e-openshift-default-55685c44c6-l58rv"
Sep 15 12:32:43.154792 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:43.154734    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e2ec4b3-4578-47a6-b505-ba920b1fa4d0-workload-socket\") pod \"e2e-ipp-b-7b204e-openshift-default-55685c44c6-l58rv\" (UID: \"4e2ec4b3-4578-47a6-b505-ba920b1fa4d0\") " pod="openshift-ingress/e2e-ipp-b-7b204e-openshift-default-55685c44c6-l58rv"
Sep 15 12:32:43.154792 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:43.154766    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e2ec4b3-4578-47a6-b505-ba920b1fa4d0-istio-data\") pod \"e2e-ipp-b-7b204e-openshift-default-55685c44c6-l58rv\" (UID: \"4e2ec4b3-4578-47a6-b505-ba920b1fa4d0\") " pod="openshift-ingress/e2e-ipp-b-7b204e-openshift-default-55685c44c6-l58rv"
Sep 15 12:32:43.154993 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:43.154875    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e2ec4b3-4578-47a6-b505-ba920b1fa4d0-credential-socket\") pod \"e2e-ipp-b-7b204e-openshift-default-55685c44c6-l58rv\" (UID: \"4e2ec4b3-4578-47a6-b505-ba920b1fa4d0\") " pod="openshift-ingress/e2e-ipp-b-7b204e-openshift-default-55685c44c6-l58rv"
Sep 15 12:32:43.155052 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:43.155012    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e2ec4b3-4578-47a6-b505-ba920b1fa4d0-workload-certs\") pod \"e2e-ipp-b-7b204e-openshift-default-55685c44c6-l58rv\" (UID: \"4e2ec4b3-4578-47a6-b505-ba920b1fa4d0\") " pod="openshift-ingress/e2e-ipp-b-7b204e-openshift-default-55685c44c6-l58rv"
Sep 15 12:32:43.155436 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:43.155416    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e2ec4b3-4578-47a6-b505-ba920b1fa4d0-istiod-ca-cert\") pod \"e2e-ipp-b-7b204e-openshift-default-55685c44c6-l58rv\" (UID: \"4e2ec4b3-4578-47a6-b505-ba920b1fa4d0\") " pod="openshift-ingress/e2e-ipp-b-7b204e-openshift-default-55685c44c6-l58rv"
Sep 15 12:32:43.156598 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:43.156579    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e2ec4b3-4578-47a6-b505-ba920b1fa4d0-istio-envoy\") pod \"e2e-ipp-b-7b204e-openshift-default-55685c44c6-l58rv\" (UID: \"4e2ec4b3-4578-47a6-b505-ba920b1fa4d0\") " pod="openshift-ingress/e2e-ipp-b-7b204e-openshift-default-55685c44c6-l58rv"
Sep 15 12:32:43.157135 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:43.157118    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e2ec4b3-4578-47a6-b505-ba920b1fa4d0-istio-podinfo\") pod \"e2e-ipp-b-7b204e-openshift-default-55685c44c6-l58rv\" (UID: \"4e2ec4b3-4578-47a6-b505-ba920b1fa4d0\") " pod="openshift-ingress/e2e-ipp-b-7b204e-openshift-default-55685c44c6-l58rv"
Sep 15 12:32:43.167870 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:43.167846    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e2ec4b3-4578-47a6-b505-ba920b1fa4d0-istio-token\") pod \"e2e-ipp-b-7b204e-openshift-default-55685c44c6-l58rv\" (UID: \"4e2ec4b3-4578-47a6-b505-ba920b1fa4d0\") " pod="openshift-ingress/e2e-ipp-b-7b204e-openshift-default-55685c44c6-l58rv"
Sep 15 12:32:43.171913 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:43.171895    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hm7sj\" (UniqueName: \"kubernetes.io/projected/4e2ec4b3-4578-47a6-b505-ba920b1fa4d0-kube-api-access-hm7sj\") pod \"e2e-ipp-b-7b204e-openshift-default-55685c44c6-l58rv\" (UID: \"4e2ec4b3-4578-47a6-b505-ba920b1fa4d0\") " pod="openshift-ingress/e2e-ipp-b-7b204e-openshift-default-55685c44c6-l58rv"
Sep 15 12:32:43.211029 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:43.210987    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-7b204e-openshift-default-55685c44c6-l58rv"
Sep 15 12:32:43.493162 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:43.493134    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-7b204e-openshift-default-55685c44c6-l58rv"]
Sep 15 12:32:43.494290 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:32:43.494250    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e2ec4b3_4578_47a6_b505_ba920b1fa4d0.slice/crio-5b4984ec35488f038eff84769653f9302faf93bc13e4ba28bfb3e651df12e611 WatchSource:0}: Error finding container 5b4984ec35488f038eff84769653f9302faf93bc13e4ba28bfb3e651df12e611: Status 404 returned error can't find the container with id 5b4984ec35488f038eff84769653f9302faf93bc13e4ba28bfb3e651df12e611
Sep 15 12:32:43.496523 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:43.496486    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:32:43.496607 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:43.496556    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:32:43.496607 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:43.496600    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:32:43.899906 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:43.899868    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7b204e-openshift-default-55685c44c6-l58rv" event={"ID":"4e2ec4b3-4578-47a6-b505-ba920b1fa4d0","Type":"ContainerStarted","Data":"4ba55b6e1c1ff55fa31ad947c212618a9345e54f3a9d9bbe069b55e1d0045f6d"}
Sep 15 12:32:43.900329 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:43.899926    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7b204e-openshift-default-55685c44c6-l58rv" event={"ID":"4e2ec4b3-4578-47a6-b505-ba920b1fa4d0","Type":"ContainerStarted","Data":"5b4984ec35488f038eff84769653f9302faf93bc13e4ba28bfb3e651df12e611"}
Sep 15 12:32:43.921621 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:43.921567    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-7b204e-openshift-default-55685c44c6-l58rv" podStartSLOduration=1.921549909 podStartE2EDuration="1.921549909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:32:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:32:43.919366642 +0000 UTC m=+1820.710289187" watchObservedRunningTime="2026-09-15 12:32:43.921549909 +0000 UTC m=+1820.712472420"
Sep 15 12:32:44.211658 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:44.211551    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-7b204e-openshift-default-55685c44c6-l58rv"
Sep 15 12:32:44.217230 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:44.217203    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-7b204e-openshift-default-55685c44c6-l58rv"
Sep 15 12:32:44.905903 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:44.905869    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-7b204e-openshift-default-55685c44c6-l58rv"
Sep 15 12:32:44.906929 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:44.906909    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-7b204e-openshift-default-55685c44c6-l58rv"
Sep 15 12:32:49.322433 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:49.322392    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7b204e-67b594784b-jxsnm"]
Sep 15 12:32:49.325960 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:49.325940    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7b204e-67b594784b-jxsnm"
Sep 15 12:32:49.328613 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:49.328590    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-7b204e-dockercfg-gqgfh\""
Sep 15 12:32:49.328986 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:49.328966    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-7b204e\""
Sep 15 12:32:49.335743 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:49.335719    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7b204e-67b594784b-jxsnm"]
Sep 15 12:32:49.408619 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:49.408570    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krr4z\" (UniqueName: \"kubernetes.io/projected/95514c8a-f25f-4d43-88a3-87cc695e3bc6-kube-api-access-krr4z\") pod \"payload-pre-processing-e2e-ipp-b-7b204e-67b594784b-jxsnm\" (UID: \"95514c8a-f25f-4d43-88a3-87cc695e3bc6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7b204e-67b594784b-jxsnm"
Sep 15 12:32:49.408619 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:49.408628    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95514c8a-f25f-4d43-88a3-87cc695e3bc6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-7b204e-67b594784b-jxsnm\" (UID: \"95514c8a-f25f-4d43-88a3-87cc695e3bc6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7b204e-67b594784b-jxsnm"
Sep 15 12:32:49.509434 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:49.509390    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krr4z\" (UniqueName: \"kubernetes.io/projected/95514c8a-f25f-4d43-88a3-87cc695e3bc6-kube-api-access-krr4z\") pod \"payload-pre-processing-e2e-ipp-b-7b204e-67b594784b-jxsnm\" (UID: \"95514c8a-f25f-4d43-88a3-87cc695e3bc6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7b204e-67b594784b-jxsnm"
Sep 15 12:32:49.509654 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:49.509449    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95514c8a-f25f-4d43-88a3-87cc695e3bc6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-7b204e-67b594784b-jxsnm\" (UID: \"95514c8a-f25f-4d43-88a3-87cc695e3bc6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7b204e-67b594784b-jxsnm"
Sep 15 12:32:49.510108 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:49.510088    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95514c8a-f25f-4d43-88a3-87cc695e3bc6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-7b204e-67b594784b-jxsnm\" (UID: \"95514c8a-f25f-4d43-88a3-87cc695e3bc6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7b204e-67b594784b-jxsnm"
Sep 15 12:32:49.518379 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:49.518357    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-krr4z\" (UniqueName: \"kubernetes.io/projected/95514c8a-f25f-4d43-88a3-87cc695e3bc6-kube-api-access-krr4z\") pod \"payload-pre-processing-e2e-ipp-b-7b204e-67b594784b-jxsnm\" (UID: \"95514c8a-f25f-4d43-88a3-87cc695e3bc6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7b204e-67b594784b-jxsnm"
Sep 15 12:32:49.637913 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:49.637875    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7b204e-67b594784b-jxsnm"
Sep 15 12:32:49.912906 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:49.912870    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7b204e-67b594784b-jxsnm"]
Sep 15 12:32:49.914300 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:32:49.914270    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95514c8a_f25f_4d43_88a3_87cc695e3bc6.slice/crio-d39ac5bc15395244975c9618c517cd69a9b22162e3319e4ba80fdf992fef4cd2 WatchSource:0}: Error finding container d39ac5bc15395244975c9618c517cd69a9b22162e3319e4ba80fdf992fef4cd2: Status 404 returned error can't find the container with id d39ac5bc15395244975c9618c517cd69a9b22162e3319e4ba80fdf992fef4cd2
Sep 15 12:32:49.935643 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:49.935614    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7b204e-67b594784b-jxsnm" event={"ID":"95514c8a-f25f-4d43-88a3-87cc695e3bc6","Type":"ContainerStarted","Data":"d39ac5bc15395244975c9618c517cd69a9b22162e3319e4ba80fdf992fef4cd2"}
Sep 15 12:32:50.942788 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:50.942751    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7b204e-67b594784b-jxsnm" event={"ID":"95514c8a-f25f-4d43-88a3-87cc695e3bc6","Type":"ContainerStarted","Data":"061df28aa95afcc8d92beda098a57dd66af871ffc70f06f0862da7eaab76f93e"}
Sep 15 12:32:50.943200 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:50.942810    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7b204e-67b594784b-jxsnm"
Sep 15 12:32:50.962532 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:50.962477    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7b204e-67b594784b-jxsnm" podStartSLOduration=1.9624594260000001 podStartE2EDuration="1.962459426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:32:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:32:50.960362653 +0000 UTC m=+1827.751285183" watchObservedRunningTime="2026-09-15 12:32:50.962459426 +0000 UTC m=+1827.753381937"
Sep 15 12:32:54.167742 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:32:54.167700    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5gkxs"]
Sep 15 12:33:01.951073 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:33:01.951034    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7b204e-67b594784b-jxsnm"
Sep 15 12:33:04.023893 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:33:04.023804    2555 generic.go:356] "Generic (PLEG): container finished" podID="4e2ec4b3-4578-47a6-b505-ba920b1fa4d0" containerID="4ba55b6e1c1ff55fa31ad947c212618a9345e54f3a9d9bbe069b55e1d0045f6d" exitCode=0
Sep 15 12:33:04.023893 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:33:04.023877    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7b204e-openshift-default-55685c44c6-l58rv" event={"ID":"4e2ec4b3-4578-47a6-b505-ba920b1fa4d0","Type":"ContainerDied","Data":"4ba55b6e1c1ff55fa31ad947c212618a9345e54f3a9d9bbe069b55e1d0045f6d"}
Sep 15 12:33:04.024374 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:33:04.024311    2555 scope.go:117] "RemoveContainer" containerID="4ba55b6e1c1ff55fa31ad947c212618a9345e54f3a9d9bbe069b55e1d0045f6d"
Sep 15 12:33:04.025380 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:33:04.025347    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:33:04.025428 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:33:04.025414    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:33:04.025473 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:33:04.025444    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:33:05.031277 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:33:05.031239    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7b204e-openshift-default-55685c44c6-l58rv" event={"ID":"4e2ec4b3-4578-47a6-b505-ba920b1fa4d0","Type":"ContainerStarted","Data":"8bd42e9cd472f5b957e0442e31872a33b2dd95feffbdc008a8fb1c7c4bcfc41d"}
Sep 15 12:33:05.211959 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:33:05.211925    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-7b204e-openshift-default-55685c44c6-l58rv"
Sep 15 12:33:05.212167 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:33:05.212088    2555 patch_prober.go:28] interesting pod/e2e-ipp-b-7b204e-openshift-default-55685c44c6-l58rv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Sep 15 12:33:05.212167 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:33:05.212142    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-7b204e-openshift-default-55685c44c6-l58rv" podUID="4e2ec4b3-4578-47a6-b505-ba920b1fa4d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Sep 15 12:33:06.211476 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:33:06.211437    2555 patch_prober.go:28] interesting pod/e2e-ipp-b-7b204e-openshift-default-55685c44c6-l58rv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Sep 15 12:33:06.211922 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:33:06.211509    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-7b204e-openshift-default-55685c44c6-l58rv" podUID="4e2ec4b3-4578-47a6-b505-ba920b1fa4d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Sep 15 12:33:07.212347 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:33:07.212306    2555 patch_prober.go:28] interesting pod/e2e-ipp-b-7b204e-openshift-default-55685c44c6-l58rv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Sep 15 12:33:07.212754 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:33:07.212375    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-7b204e-openshift-default-55685c44c6-l58rv" podUID="4e2ec4b3-4578-47a6-b505-ba920b1fa4d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Sep 15 12:33:08.211940 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:33:08.211894    2555 patch_prober.go:28] interesting pod/e2e-ipp-b-7b204e-openshift-default-55685c44c6-l58rv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Sep 15 12:33:08.212132 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:33:08.211962    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-7b204e-openshift-default-55685c44c6-l58rv" podUID="4e2ec4b3-4578-47a6-b505-ba920b1fa4d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Sep 15 12:33:09.212091 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:33:09.212050    2555 patch_prober.go:28] interesting pod/e2e-ipp-b-7b204e-openshift-default-55685c44c6-l58rv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Sep 15 12:33:09.212631 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:33:09.212126    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-7b204e-openshift-default-55685c44c6-l58rv" podUID="4e2ec4b3-4578-47a6-b505-ba920b1fa4d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Sep 15 12:33:10.215873 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:33:10.215842    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-7b204e-openshift-default-55685c44c6-l58rv"
Sep 15 12:33:10.216291 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:33:10.216174    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-7b204e-openshift-default-55685c44c6-l58rv"
Sep 15 12:33:10.216804 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:33:10.216787    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-7b204e-openshift-default-55685c44c6-l58rv"
Sep 15 12:33:24.211108 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:33:24.211075    2555 scope.go:117] "RemoveContainer" containerID="bced5b9321245b06bdbec0684eab4a1c321e440b115405dc865dc8ca2aaa6b11"
Sep 15 12:33:25.505415 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:33:25.505374    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-ed21e2/ipp-route-ed21e2-kserve-75b9c49dbc-sp9fj"]
Sep 15 12:33:25.510064 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:33:25.510042    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-ed21e2/ipp-route-ed21e2-kserve-75b9c49dbc-sp9fj"
Sep 15 12:33:25.512996 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:33:25.512968    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-ed21e2\"/\"kube-root-ca.crt\""
Sep 15 12:33:25.513134 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:33:25.512977    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-ed21e2\"/\"openshift-service-ca.crt\""
Sep 15 12:33:25.514512 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:33:25.514488    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-ed21e2\"/\"ipp-route-ed21e2-kserve-self-signed-certs\""
Sep 15 12:33:25.514629 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:33:25.514518    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-ed21e2\"/\"default-dockercfg-njfxx\""
Sep 15 12:33:25.517294 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:33:25.517275    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-ed21e2/ipp-route-ed21e2-kserve-75b9c49dbc-sp9fj"]
Sep 15 12:33:25.558709 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:33:25.558667    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1adc44fa-0e09-4bf9-a8f0-e95522b6b507-tls-certs\") pod \"ipp-route-ed21e2-kserve-75b9c49dbc-sp9fj\" (UID: \"1adc44fa-0e09-4bf9-a8f0-e95522b6b507\") " pod="ai-tenant-e2e-ipp-a-ed21e2/ipp-route-ed21e2-kserve-75b9c49dbc-sp9fj"
Sep 15 12:33:25.558886 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:33:25.558734    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1adc44fa-0e09-4bf9-a8f0-e95522b6b507-home\") pod \"ipp-route-ed21e2-kserve-75b9c49dbc-sp9fj\" (UID: \"1adc44fa-0e09-4bf9-a8f0-e95522b6b507\") " pod="ai-tenant-e2e-ipp-a-ed21e2/ipp-route-ed21e2-kserve-75b9c49dbc-sp9fj"
Sep 15 12:33:25.558886 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:33:25.558762    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1adc44fa-0e09-4bf9-a8f0-e95522b6b507-dshm\") pod \"ipp-route-ed21e2-kserve-75b9c49dbc-sp9fj\" (UID: \"1adc44fa-0e09-4bf9-a8f0-e95522b6b507\") " pod="ai-tenant-e2e-ipp-a-ed21e2/ipp-route-ed21e2-kserve-75b9c49dbc-sp9fj"
Sep 15 12:33:25.558886 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:33:25.558778    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnrvw\" (UniqueName: \"kubernetes.io/projected/1adc44fa-0e09-4bf9-a8f0-e95522b6b507-kube-api-access-bnrvw\") pod \"ipp-route-ed21e2-kserve-75b9c49dbc-sp9fj\" (UID: \"1adc44fa-0e09-4bf9-a8f0-e95522b6b507\") " pod="ai-tenant-e2e-ipp-a-ed21e2/ipp-route-ed21e2-kserve-75b9c49dbc-sp9fj"
Sep 15 12:33:25.558886 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:33:25.558812    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1adc44fa-0e09-4bf9-a8f0-e95522b6b507-tmp-dir\") pod \"ipp-route-ed21e2-kserve-75b9c49dbc-sp9fj\" (UID: \"1adc44fa-0e09-4bf9-a8f0-e95522b6b507\") " pod="ai-tenant-e2e-ipp-a-ed21e2/ipp-route-ed21e2-kserve-75b9c49dbc-sp9fj"
Sep 15 12:33:25.558886 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:33:25.558836    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1adc44fa-0e09-4bf9-a8f0-e95522b6b507-model-cache\") pod \"ipp-route-ed21e2-kserve-75b9c49dbc-sp9fj\" (UID: \"1adc44fa-0e09-4bf9-a8f0-e95522b6b507\") " pod="ai-tenant-e2e-ipp-a-ed21e2/ipp-route-ed21e2-kserve-75b9c49dbc-sp9fj"
Sep 15 12:33:25.659573 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:33:25.659528    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1adc44fa-0e09-4bf9-a8f0-e95522b6b507-tmp-dir\") pod \"ipp-route-ed21e2-kserve-75b9c49dbc-sp9fj\" (UID: \"1adc44fa-0e09-4bf9-a8f0-e95522b6b507\") " pod="ai-tenant-e2e-ipp-a-ed21e2/ipp-route-ed21e2-kserve-75b9c49dbc-sp9fj"
Sep 15 12:33:25.659751 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:33:25.659595    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1adc44fa-0e09-4bf9-a8f0-e95522b6b507-model-cache\") pod \"ipp-route-ed21e2-kserve-75b9c49dbc-sp9fj\" (UID: \"1adc44fa-0e09-4bf9-a8f0-e95522b6b507\") " pod="ai-tenant-e2e-ipp-a-ed21e2/ipp-route-ed21e2-kserve-75b9c49dbc-sp9fj"
Sep 15 12:33:25.659751 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:33:25.659668    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1adc44fa-0e09-4bf9-a8f0-e95522b6b507-tls-certs\") pod \"ipp-route-ed21e2-kserve-75b9c49dbc-sp9fj\" (UID: \"1adc44fa-0e09-4bf9-a8f0-e95522b6b507\") " pod="ai-tenant-e2e-ipp-a-ed21e2/ipp-route-ed21e2-kserve-75b9c49dbc-sp9fj"
Sep 15 12:33:25.659751 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:33:25.659723    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1adc44fa-0e09-4bf9-a8f0-e95522b6b507-home\") pod \"ipp-route-ed21e2-kserve-75b9c49dbc-sp9fj\" (UID: \"1adc44fa-0e09-4bf9-a8f0-e95522b6b507\") " pod="ai-tenant-e2e-ipp-a-ed21e2/ipp-route-ed21e2-kserve-75b9c49dbc-sp9fj"
Sep 15 12:33:25.659913 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:33:25.659758    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1adc44fa-0e09-4bf9-a8f0-e95522b6b507-dshm\") pod \"ipp-route-ed21e2-kserve-75b9c49dbc-sp9fj\" (UID: \"1adc44fa-0e09-4bf9-a8f0-e95522b6b507\") " pod="ai-tenant-e2e-ipp-a-ed21e2/ipp-route-ed21e2-kserve-75b9c49dbc-sp9fj"
Sep 15 12:33:25.659913 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:33:25.659780    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnrvw\" (UniqueName: \"kubernetes.io/projected/1adc44fa-0e09-4bf9-a8f0-e95522b6b507-kube-api-access-bnrvw\") pod \"ipp-route-ed21e2-kserve-75b9c49dbc-sp9fj\" (UID: \"1adc44fa-0e09-4bf9-a8f0-e95522b6b507\") " pod="ai-tenant-e2e-ipp-a-ed21e2/ipp-route-ed21e2-kserve-75b9c49dbc-sp9fj"
Sep 15 12:33:25.660045 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:33:25.660009    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1adc44fa-0e09-4bf9-a8f0-e95522b6b507-tmp-dir\") pod \"ipp-route-ed21e2-kserve-75b9c49dbc-sp9fj\" (UID: \"1adc44fa-0e09-4bf9-a8f0-e95522b6b507\") " pod="ai-tenant-e2e-ipp-a-ed21e2/ipp-route-ed21e2-kserve-75b9c49dbc-sp9fj"
Sep 15 12:33:25.660116 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:33:25.660054    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1adc44fa-0e09-4bf9-a8f0-e95522b6b507-model-cache\") pod \"ipp-route-ed21e2-kserve-75b9c49dbc-sp9fj\" (UID: \"1adc44fa-0e09-4bf9-a8f0-e95522b6b507\") " pod="ai-tenant-e2e-ipp-a-ed21e2/ipp-route-ed21e2-kserve-75b9c49dbc-sp9fj"
Sep 15 12:33:25.660116 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:33:25.660085    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1adc44fa-0e09-4bf9-a8f0-e95522b6b507-home\") pod \"ipp-route-ed21e2-kserve-75b9c49dbc-sp9fj\" (UID: \"1adc44fa-0e09-4bf9-a8f0-e95522b6b507\") " pod="ai-tenant-e2e-ipp-a-ed21e2/ipp-route-ed21e2-kserve-75b9c49dbc-sp9fj"
Sep 15 12:33:25.662180 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:33:25.662148    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1adc44fa-0e09-4bf9-a8f0-e95522b6b507-dshm\") pod \"ipp-route-ed21e2-kserve-75b9c49dbc-sp9fj\" (UID: \"1adc44fa-0e09-4bf9-a8f0-e95522b6b507\") " pod="ai-tenant-e2e-ipp-a-ed21e2/ipp-route-ed21e2-kserve-75b9c49dbc-sp9fj"
Sep 15 12:33:25.662415 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:33:25.662398    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1adc44fa-0e09-4bf9-a8f0-e95522b6b507-tls-certs\") pod \"ipp-route-ed21e2-kserve-75b9c49dbc-sp9fj\" (UID: \"1adc44fa-0e09-4bf9-a8f0-e95522b6b507\") " pod="ai-tenant-e2e-ipp-a-ed21e2/ipp-route-ed21e2-kserve-75b9c49dbc-sp9fj"
Sep 15 12:33:25.668566 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:33:25.668541    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnrvw\" (UniqueName: \"kubernetes.io/projected/1adc44fa-0e09-4bf9-a8f0-e95522b6b507-kube-api-access-bnrvw\") pod \"ipp-route-ed21e2-kserve-75b9c49dbc-sp9fj\" (UID: \"1adc44fa-0e09-4bf9-a8f0-e95522b6b507\") " pod="ai-tenant-e2e-ipp-a-ed21e2/ipp-route-ed21e2-kserve-75b9c49dbc-sp9fj"
Sep 15 12:33:25.821831 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:33:25.821739    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-ed21e2/ipp-route-ed21e2-kserve-75b9c49dbc-sp9fj"
Sep 15 12:33:26.098488 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:33:26.098460    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-ed21e2/ipp-route-ed21e2-kserve-75b9c49dbc-sp9fj"]
Sep 15 12:33:26.099648 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:33:26.099619    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1adc44fa_0e09_4bf9_a8f0_e95522b6b507.slice/crio-48c45d498fee9822b42be9b6e9d8ba2cc78c999cff64cbbb97582d0930ffed4a WatchSource:0}: Error finding container 48c45d498fee9822b42be9b6e9d8ba2cc78c999cff64cbbb97582d0930ffed4a: Status 404 returned error can't find the container with id 48c45d498fee9822b42be9b6e9d8ba2cc78c999cff64cbbb97582d0930ffed4a
Sep 15 12:33:26.163294 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:33:26.163251    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-ed21e2/ipp-route-ed21e2-kserve-75b9c49dbc-sp9fj" event={"ID":"1adc44fa-0e09-4bf9-a8f0-e95522b6b507","Type":"ContainerStarted","Data":"48c45d498fee9822b42be9b6e9d8ba2cc78c999cff64cbbb97582d0930ffed4a"}
Sep 15 12:33:27.171534 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:33:27.171496    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-ed21e2/ipp-route-ed21e2-kserve-75b9c49dbc-sp9fj" event={"ID":"1adc44fa-0e09-4bf9-a8f0-e95522b6b507","Type":"ContainerStarted","Data":"6668f626b3fc42cc7f43532e715e908698c4cfafd390b0ae8d7c3ffdd01869af"}
Sep 15 12:33:27.190558 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:33:27.190503    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-ed21e2/ipp-route-ed21e2-kserve-75b9c49dbc-sp9fj" podStartSLOduration=2.190486972 podStartE2EDuration="2.190486972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:33:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:33:27.188425928 +0000 UTC m=+1863.979348451" watchObservedRunningTime="2026-09-15 12:33:27.190486972 +0000 UTC m=+1863.981409482"
Sep 15 12:33:29.179354 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:33:29.179312    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5gkxs"]
Sep 15 12:33:35.822335 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:33:35.822299    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-ed21e2/ipp-route-ed21e2-kserve-75b9c49dbc-sp9fj"
Sep 15 12:33:35.822335 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:33:35.822342    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-ed21e2/ipp-route-ed21e2-kserve-75b9c49dbc-sp9fj"
Sep 15 12:33:35.834831 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:33:35.834800    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-ed21e2/ipp-route-ed21e2-kserve-75b9c49dbc-sp9fj"
Sep 15 12:33:36.232049 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:33:36.232018    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-ed21e2/ipp-route-ed21e2-kserve-75b9c49dbc-sp9fj"
Sep 15 12:33:51.161502 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:33:51.161419    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5gkxs"]
Sep 15 12:34:05.686761 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:34:05.686719    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-9725ba-867b747bfd-9k682"]
Sep 15 12:34:05.692572 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:34:05.692526    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-9725ba-867b747bfd-9k682"
Sep 15 12:34:05.696065 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:34:05.696038    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-9725ba\""
Sep 15 12:34:05.696217 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:34:05.696089    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-9725ba-dockercfg-7r648\""
Sep 15 12:34:05.701489 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:34:05.701429    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-9725ba-867b747bfd-9k682"]
Sep 15 12:34:05.734704 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:34:05.734662    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0494a388-936d-4884-a916-bfee1aab24cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-9725ba-867b747bfd-9k682\" (UID: \"0494a388-936d-4884-a916-bfee1aab24cc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-9725ba-867b747bfd-9k682"
Sep 15 12:34:05.734880 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:34:05.734712    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjtkp\" (UniqueName: \"kubernetes.io/projected/0494a388-936d-4884-a916-bfee1aab24cc-kube-api-access-mjtkp\") pod \"payload-pre-processing-e2e-ipp-cleanup-9725ba-867b747bfd-9k682\" (UID: \"0494a388-936d-4884-a916-bfee1aab24cc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-9725ba-867b747bfd-9k682"
Sep 15 12:34:05.835449 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:34:05.835416    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0494a388-936d-4884-a916-bfee1aab24cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-9725ba-867b747bfd-9k682\" (UID: \"0494a388-936d-4884-a916-bfee1aab24cc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-9725ba-867b747bfd-9k682"
Sep 15 12:34:05.835681 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:34:05.835467    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjtkp\" (UniqueName: \"kubernetes.io/projected/0494a388-936d-4884-a916-bfee1aab24cc-kube-api-access-mjtkp\") pod \"payload-pre-processing-e2e-ipp-cleanup-9725ba-867b747bfd-9k682\" (UID: \"0494a388-936d-4884-a916-bfee1aab24cc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-9725ba-867b747bfd-9k682"
Sep 15 12:34:05.836180 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:34:05.836156    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0494a388-936d-4884-a916-bfee1aab24cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-9725ba-867b747bfd-9k682\" (UID: \"0494a388-936d-4884-a916-bfee1aab24cc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-9725ba-867b747bfd-9k682"
Sep 15 12:34:05.844034 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:34:05.844008    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjtkp\" (UniqueName: \"kubernetes.io/projected/0494a388-936d-4884-a916-bfee1aab24cc-kube-api-access-mjtkp\") pod \"payload-pre-processing-e2e-ipp-cleanup-9725ba-867b747bfd-9k682\" (UID: \"0494a388-936d-4884-a916-bfee1aab24cc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-9725ba-867b747bfd-9k682"
Sep 15 12:34:06.005748 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:34:06.005652    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-9725ba-867b747bfd-9k682"
Sep 15 12:34:06.278401 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:34:06.278368    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-9725ba-867b747bfd-9k682"]
Sep 15 12:34:06.279603 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:34:06.279574    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0494a388_936d_4884_a916_bfee1aab24cc.slice/crio-4f6b77b602d84dd0a975e51d980a1938a40e258fe0969774435bee098d2ac87f WatchSource:0}: Error finding container 4f6b77b602d84dd0a975e51d980a1938a40e258fe0969774435bee098d2ac87f: Status 404 returned error can't find the container with id 4f6b77b602d84dd0a975e51d980a1938a40e258fe0969774435bee098d2ac87f
Sep 15 12:34:06.383331 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:34:06.383295    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-9725ba-867b747bfd-9k682" event={"ID":"0494a388-936d-4884-a916-bfee1aab24cc","Type":"ContainerStarted","Data":"39243fc2f55ee76502ce314961df7b6c1aa930606edd78fe944623dccfac6904"}
Sep 15 12:34:06.383331 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:34:06.383351    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-9725ba-867b747bfd-9k682" event={"ID":"0494a388-936d-4884-a916-bfee1aab24cc","Type":"ContainerStarted","Data":"4f6b77b602d84dd0a975e51d980a1938a40e258fe0969774435bee098d2ac87f"}
Sep 15 12:34:06.383649 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:34:06.383540    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-9725ba-867b747bfd-9k682"
Sep 15 12:34:06.400769 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:34:06.400711    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-9725ba-867b747bfd-9k682" podStartSLOduration=1.4006947140000001 podStartE2EDuration="1.400694714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:34:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:34:06.3998328 +0000 UTC m=+1903.190755336" watchObservedRunningTime="2026-09-15 12:34:06.400694714 +0000 UTC m=+1903.191617225"
Sep 15 12:34:10.764977 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:34:10.764941    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5gkxs"]
Sep 15 12:34:17.391399 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:34:17.391366    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-9725ba-867b747bfd-9k682"
Sep 15 12:34:24.241068 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:34:24.241035    2555 scope.go:117] "RemoveContainer" containerID="dd56b88652d30f43d3fdef34113dbec5ed98ed059427e6fd32de1fe86ebfcc68"
Sep 15 12:34:24.249873 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:34:24.249852    2555 scope.go:117] "RemoveContainer" containerID="3392816cc8eca6f95b8f3b3a3716544be0abeff5db0be62a30dafc58e8867615"
Sep 15 12:34:24.258181 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:34:24.258160    2555 scope.go:117] "RemoveContainer" containerID="94520d89d2200c947d3e99cbace6fe48869476d396d8c529bb09436288d01adf"
Sep 15 12:34:32.601446 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:34:32.601409    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-9725ba-vkqf2"]
Sep 15 12:34:32.604805 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:34:32.604787    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-9725ba-vkqf2"
Sep 15 12:34:32.607596 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:34:32.607574    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 12:34:32.607705 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:34:32.607659    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tbvkt\""
Sep 15 12:34:32.613350 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:34:32.613321    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-9725ba-vkqf2"]
Sep 15 12:34:32.685518 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:34:32.685479    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0d829eb-1ca4-49f3-97c5-31e53b9db1d7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-9725ba-vkqf2\" (UID: \"f0d829eb-1ca4-49f3-97c5-31e53b9db1d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-9725ba-vkqf2"
Sep 15 12:34:32.787002 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:34:32.786964    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0d829eb-1ca4-49f3-97c5-31e53b9db1d7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-9725ba-vkqf2\" (UID: \"f0d829eb-1ca4-49f3-97c5-31e53b9db1d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-9725ba-vkqf2"
Sep 15 12:34:32.787606 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:34:32.787587    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0d829eb-1ca4-49f3-97c5-31e53b9db1d7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-9725ba-vkqf2\" (UID: \"f0d829eb-1ca4-49f3-97c5-31e53b9db1d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-9725ba-vkqf2"
Sep 15 12:34:32.916713 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:34:32.916679    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-9725ba-vkqf2"
Sep 15 12:34:33.186761 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:34:33.186734    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-9725ba-vkqf2"]
Sep 15 12:34:33.191936 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:34:33.191903    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0d829eb_1ca4_49f3_97c5_31e53b9db1d7.slice/crio-e379d685c241dee73809c11abd3e5319063e651ad636bd1ed99241f313ce0cf8 WatchSource:0}: Error finding container e379d685c241dee73809c11abd3e5319063e651ad636bd1ed99241f313ce0cf8: Status 404 returned error can't find the container with id e379d685c241dee73809c11abd3e5319063e651ad636bd1ed99241f313ce0cf8
Sep 15 12:34:33.556470 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:34:33.556364    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-9725ba-vkqf2" event={"ID":"f0d829eb-1ca4-49f3-97c5-31e53b9db1d7","Type":"ContainerStarted","Data":"ae59287b17762fc27fd4e4e63192c49d3a92e4c50a0e14701e0ffab3a7d49148"}
Sep 15 12:34:33.556470 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:34:33.556428    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-9725ba-vkqf2" event={"ID":"f0d829eb-1ca4-49f3-97c5-31e53b9db1d7","Type":"ContainerStarted","Data":"e379d685c241dee73809c11abd3e5319063e651ad636bd1ed99241f313ce0cf8"}
Sep 15 12:34:33.578254 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:34:33.578196    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-9725ba-vkqf2" podStartSLOduration=1.578179988 podStartE2EDuration="1.578179988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:34:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:34:33.576986754 +0000 UTC m=+1930.367909284" watchObservedRunningTime="2026-09-15 12:34:33.578179988 +0000 UTC m=+1930.369102498"
Sep 15 12:34:34.563720 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:34:34.563683    2555 generic.go:356] "Generic (PLEG): container finished" podID="f0d829eb-1ca4-49f3-97c5-31e53b9db1d7" containerID="ae59287b17762fc27fd4e4e63192c49d3a92e4c50a0e14701e0ffab3a7d49148" exitCode=0
Sep 15 12:34:34.563720 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:34:34.563723    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-9725ba-vkqf2" event={"ID":"f0d829eb-1ca4-49f3-97c5-31e53b9db1d7","Type":"ContainerDied","Data":"ae59287b17762fc27fd4e4e63192c49d3a92e4c50a0e14701e0ffab3a7d49148"}
Sep 15 12:34:35.703828 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:34:35.703799    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-9725ba-vkqf2"
Sep 15 12:34:35.817590 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:34:35.817555    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0d829eb-1ca4-49f3-97c5-31e53b9db1d7-maas-api-service-ca\") pod \"f0d829eb-1ca4-49f3-97c5-31e53b9db1d7\" (UID: \"f0d829eb-1ca4-49f3-97c5-31e53b9db1d7\") "
Sep 15 12:34:35.817853 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:34:35.817827    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0d829eb-1ca4-49f3-97c5-31e53b9db1d7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f0d829eb-1ca4-49f3-97c5-31e53b9db1d7" (UID: "f0d829eb-1ca4-49f3-97c5-31e53b9db1d7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:34:35.918892 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:34:35.918853    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0d829eb-1ca4-49f3-97c5-31e53b9db1d7-maas-api-service-ca\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:34:36.576114 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:34:36.576074    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-9725ba-vkqf2" event={"ID":"f0d829eb-1ca4-49f3-97c5-31e53b9db1d7","Type":"ContainerDied","Data":"e379d685c241dee73809c11abd3e5319063e651ad636bd1ed99241f313ce0cf8"}
Sep 15 12:34:36.576288 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:34:36.576133    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e379d685c241dee73809c11abd3e5319063e651ad636bd1ed99241f313ce0cf8"
Sep 15 12:34:36.576288 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:34:36.576086    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-9725ba-vkqf2"
Sep 15 12:34:43.361774 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:34:43.361739    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-9725ba-867b747bfd-9k682"]
Sep 15 12:34:43.362405 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:34:43.362021    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-9725ba-867b747bfd-9k682" podUID="0494a388-936d-4884-a916-bfee1aab24cc" containerName="payload-pre-processing" containerID="cri-o://39243fc2f55ee76502ce314961df7b6c1aa930606edd78fe944623dccfac6904" gracePeriod=30
Sep 15 12:34:43.513953 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:34:43.513919    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-9725ba-vkqf2"]
Sep 15 12:34:43.517748 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:34:43.517712    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-9725ba-vkqf2"]
Sep 15 12:34:43.617576 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:34:43.617504    2555 generic.go:356] "Generic (PLEG): container finished" podID="0494a388-936d-4884-a916-bfee1aab24cc" containerID="39243fc2f55ee76502ce314961df7b6c1aa930606edd78fe944623dccfac6904" exitCode=0
Sep 15 12:34:43.617723 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:34:43.617580    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-9725ba-867b747bfd-9k682" event={"ID":"0494a388-936d-4884-a916-bfee1aab24cc","Type":"ContainerDied","Data":"39243fc2f55ee76502ce314961df7b6c1aa930606edd78fe944623dccfac6904"}
Sep 15 12:34:43.617723 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:34:43.617651    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-9725ba-867b747bfd-9k682" event={"ID":"0494a388-936d-4884-a916-bfee1aab24cc","Type":"ContainerDied","Data":"4f6b77b602d84dd0a975e51d980a1938a40e258fe0969774435bee098d2ac87f"}
Sep 15 12:34:43.617723 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:34:43.617676    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f6b77b602d84dd0a975e51d980a1938a40e258fe0969774435bee098d2ac87f"
Sep 15 12:34:43.618897 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:34:43.618873    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-9725ba-867b747bfd-9k682"
Sep 15 12:34:43.795432 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:34:43.795387    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mjtkp\" (UniqueName: \"kubernetes.io/projected/0494a388-936d-4884-a916-bfee1aab24cc-kube-api-access-mjtkp\") pod \"0494a388-936d-4884-a916-bfee1aab24cc\" (UID: \"0494a388-936d-4884-a916-bfee1aab24cc\") "
Sep 15 12:34:43.795642 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:34:43.795463    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0494a388-936d-4884-a916-bfee1aab24cc-plugins-config-volume\") pod \"0494a388-936d-4884-a916-bfee1aab24cc\" (UID: \"0494a388-936d-4884-a916-bfee1aab24cc\") "
Sep 15 12:34:43.795805 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:34:43.795780    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0494a388-936d-4884-a916-bfee1aab24cc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0494a388-936d-4884-a916-bfee1aab24cc" (UID: "0494a388-936d-4884-a916-bfee1aab24cc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:34:43.797535 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:34:43.797511    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0494a388-936d-4884-a916-bfee1aab24cc-kube-api-access-mjtkp" (OuterVolumeSpecName: "kube-api-access-mjtkp") pod "0494a388-936d-4884-a916-bfee1aab24cc" (UID: "0494a388-936d-4884-a916-bfee1aab24cc"). InnerVolumeSpecName "kube-api-access-mjtkp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:34:43.817961 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:34:43.817927    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0d829eb-1ca4-49f3-97c5-31e53b9db1d7" path="/var/lib/kubelet/pods/f0d829eb-1ca4-49f3-97c5-31e53b9db1d7/volumes"
Sep 15 12:34:43.896308 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:34:43.896250    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjtkp\" (UniqueName: \"kubernetes.io/projected/0494a388-936d-4884-a916-bfee1aab24cc-kube-api-access-mjtkp\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:34:43.896308 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:34:43.896308    2555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0494a388-936d-4884-a916-bfee1aab24cc-plugins-config-volume\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:34:44.622816 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:34:44.622782    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-9725ba-867b747bfd-9k682"
Sep 15 12:34:44.642093 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:34:44.642057    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-9725ba-867b747bfd-9k682"]
Sep 15 12:34:44.659545 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:34:44.659511    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-9725ba-867b747bfd-9k682"]
Sep 15 12:34:45.817755 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:34:45.817722    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0494a388-936d-4884-a916-bfee1aab24cc" path="/var/lib/kubelet/pods/0494a388-936d-4884-a916-bfee1aab24cc/volumes"
Sep 15 12:34:48.174850 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:34:48.174811    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5gkxs"]
Sep 15 12:35:07.268556 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:35:07.268518    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5gkxs"]
Sep 15 12:35:15.357702 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:35:15.357663    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-ed21e2/ipp-route-ed21e2-kserve-75b9c49dbc-sp9fj"]
Sep 15 12:35:20.663846 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:35:20.663810    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5gkxs"]
Sep 15 12:35:24.285759 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:35:24.285728    2555 scope.go:117] "RemoveContainer" containerID="3f091afac70a9dbbb47f25203b2ea0edb33b1e29490b45639342986a4e734950"
Sep 15 12:35:30.376998 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:35:30.376947    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-ed21e2/ipp-route-ed21e2-kserve-75b9c49dbc-sp9fj" podUID="1adc44fa-0e09-4bf9-a8f0-e95522b6b507" containerName="main" containerID="cri-o://6668f626b3fc42cc7f43532e715e908698c4cfafd390b0ae8d7c3ffdd01869af" gracePeriod=45
Sep 15 12:35:30.635200 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:35:30.635171    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-ed21e2/ipp-route-ed21e2-kserve-75b9c49dbc-sp9fj"
Sep 15 12:35:30.732598 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:35:30.732560    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bnrvw\" (UniqueName: \"kubernetes.io/projected/1adc44fa-0e09-4bf9-a8f0-e95522b6b507-kube-api-access-bnrvw\") pod \"1adc44fa-0e09-4bf9-a8f0-e95522b6b507\" (UID: \"1adc44fa-0e09-4bf9-a8f0-e95522b6b507\") "
Sep 15 12:35:30.732790 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:35:30.732646    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1adc44fa-0e09-4bf9-a8f0-e95522b6b507-model-cache\") pod \"1adc44fa-0e09-4bf9-a8f0-e95522b6b507\" (UID: \"1adc44fa-0e09-4bf9-a8f0-e95522b6b507\") "
Sep 15 12:35:30.732790 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:35:30.732686    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1adc44fa-0e09-4bf9-a8f0-e95522b6b507-tmp-dir\") pod \"1adc44fa-0e09-4bf9-a8f0-e95522b6b507\" (UID: \"1adc44fa-0e09-4bf9-a8f0-e95522b6b507\") "
Sep 15 12:35:30.732790 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:35:30.732702    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1adc44fa-0e09-4bf9-a8f0-e95522b6b507-tls-certs\") pod \"1adc44fa-0e09-4bf9-a8f0-e95522b6b507\" (UID: \"1adc44fa-0e09-4bf9-a8f0-e95522b6b507\") "
Sep 15 12:35:30.732790 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:35:30.732728    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1adc44fa-0e09-4bf9-a8f0-e95522b6b507-home\") pod \"1adc44fa-0e09-4bf9-a8f0-e95522b6b507\" (UID: \"1adc44fa-0e09-4bf9-a8f0-e95522b6b507\") "
Sep 15 12:35:30.732790 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:35:30.732757    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1adc44fa-0e09-4bf9-a8f0-e95522b6b507-dshm\") pod \"1adc44fa-0e09-4bf9-a8f0-e95522b6b507\" (UID: \"1adc44fa-0e09-4bf9-a8f0-e95522b6b507\") "
Sep 15 12:35:30.733073 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:35:30.732920    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1adc44fa-0e09-4bf9-a8f0-e95522b6b507-model-cache" (OuterVolumeSpecName: "model-cache") pod "1adc44fa-0e09-4bf9-a8f0-e95522b6b507" (UID: "1adc44fa-0e09-4bf9-a8f0-e95522b6b507"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:35:30.733073 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:35:30.732975    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1adc44fa-0e09-4bf9-a8f0-e95522b6b507-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1adc44fa-0e09-4bf9-a8f0-e95522b6b507" (UID: "1adc44fa-0e09-4bf9-a8f0-e95522b6b507"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:35:30.733073 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:35:30.733056    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1adc44fa-0e09-4bf9-a8f0-e95522b6b507-home" (OuterVolumeSpecName: "home") pod "1adc44fa-0e09-4bf9-a8f0-e95522b6b507" (UID: "1adc44fa-0e09-4bf9-a8f0-e95522b6b507"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:35:30.733219 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:35:30.733133    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1adc44fa-0e09-4bf9-a8f0-e95522b6b507-model-cache\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:35:30.733219 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:35:30.733148    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1adc44fa-0e09-4bf9-a8f0-e95522b6b507-tmp-dir\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:35:30.734936 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:35:30.734912    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1adc44fa-0e09-4bf9-a8f0-e95522b6b507-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1adc44fa-0e09-4bf9-a8f0-e95522b6b507" (UID: "1adc44fa-0e09-4bf9-a8f0-e95522b6b507"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:35:30.735047 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:35:30.734916    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1adc44fa-0e09-4bf9-a8f0-e95522b6b507-dshm" (OuterVolumeSpecName: "dshm") pod "1adc44fa-0e09-4bf9-a8f0-e95522b6b507" (UID: "1adc44fa-0e09-4bf9-a8f0-e95522b6b507"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:35:30.735047 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:35:30.734963    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1adc44fa-0e09-4bf9-a8f0-e95522b6b507-kube-api-access-bnrvw" (OuterVolumeSpecName: "kube-api-access-bnrvw") pod "1adc44fa-0e09-4bf9-a8f0-e95522b6b507" (UID: "1adc44fa-0e09-4bf9-a8f0-e95522b6b507"). InnerVolumeSpecName "kube-api-access-bnrvw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:35:30.833962 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:35:30.833922    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1adc44fa-0e09-4bf9-a8f0-e95522b6b507-dshm\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:35:30.833962 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:35:30.833954    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnrvw\" (UniqueName: \"kubernetes.io/projected/1adc44fa-0e09-4bf9-a8f0-e95522b6b507-kube-api-access-bnrvw\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:35:30.833962 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:35:30.833968    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1adc44fa-0e09-4bf9-a8f0-e95522b6b507-tls-certs\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:35:30.834221 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:35:30.833983    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1adc44fa-0e09-4bf9-a8f0-e95522b6b507-home\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:35:30.884959 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:35:30.884926    2555 generic.go:356] "Generic (PLEG): container finished" podID="1adc44fa-0e09-4bf9-a8f0-e95522b6b507" containerID="6668f626b3fc42cc7f43532e715e908698c4cfafd390b0ae8d7c3ffdd01869af" exitCode=0
Sep 15 12:35:30.885157 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:35:30.884965    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-ed21e2/ipp-route-ed21e2-kserve-75b9c49dbc-sp9fj" event={"ID":"1adc44fa-0e09-4bf9-a8f0-e95522b6b507","Type":"ContainerDied","Data":"6668f626b3fc42cc7f43532e715e908698c4cfafd390b0ae8d7c3ffdd01869af"}
Sep 15 12:35:30.885157 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:35:30.884974    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-ed21e2/ipp-route-ed21e2-kserve-75b9c49dbc-sp9fj"
Sep 15 12:35:30.885157 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:35:30.885004    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-ed21e2/ipp-route-ed21e2-kserve-75b9c49dbc-sp9fj" event={"ID":"1adc44fa-0e09-4bf9-a8f0-e95522b6b507","Type":"ContainerDied","Data":"48c45d498fee9822b42be9b6e9d8ba2cc78c999cff64cbbb97582d0930ffed4a"}
Sep 15 12:35:30.885157 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:35:30.885026    2555 scope.go:117] "RemoveContainer" containerID="6668f626b3fc42cc7f43532e715e908698c4cfafd390b0ae8d7c3ffdd01869af"
Sep 15 12:35:30.894351 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:35:30.894332    2555 scope.go:117] "RemoveContainer" containerID="6668f626b3fc42cc7f43532e715e908698c4cfafd390b0ae8d7c3ffdd01869af"
Sep 15 12:35:30.894646 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:35:30.894626    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6668f626b3fc42cc7f43532e715e908698c4cfafd390b0ae8d7c3ffdd01869af\": container with ID starting with 6668f626b3fc42cc7f43532e715e908698c4cfafd390b0ae8d7c3ffdd01869af not found: ID does not exist" containerID="6668f626b3fc42cc7f43532e715e908698c4cfafd390b0ae8d7c3ffdd01869af"
Sep 15 12:35:30.894727 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:35:30.894659    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6668f626b3fc42cc7f43532e715e908698c4cfafd390b0ae8d7c3ffdd01869af"} err="failed to get container status \"6668f626b3fc42cc7f43532e715e908698c4cfafd390b0ae8d7c3ffdd01869af\": rpc error: code = NotFound desc = could not find container \"6668f626b3fc42cc7f43532e715e908698c4cfafd390b0ae8d7c3ffdd01869af\": container with ID starting with 6668f626b3fc42cc7f43532e715e908698c4cfafd390b0ae8d7c3ffdd01869af not found: ID does not exist"
Sep 15 12:35:30.907745 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:35:30.907715    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-ed21e2/ipp-route-ed21e2-kserve-75b9c49dbc-sp9fj"]
Sep 15 12:35:30.911987 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:35:30.911959    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-ed21e2/ipp-route-ed21e2-kserve-75b9c49dbc-sp9fj"]
Sep 15 12:35:31.817216 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:35:31.817183    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1adc44fa-0e09-4bf9-a8f0-e95522b6b507" path="/var/lib/kubelet/pods/1adc44fa-0e09-4bf9-a8f0-e95522b6b507/volumes"
Sep 15 12:36:09.568944 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:36:09.568905    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7b204e-67b594784b-jxsnm"]
Sep 15 12:36:09.569814 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:36:09.569744    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7b204e-67b594784b-jxsnm" podUID="95514c8a-f25f-4d43-88a3-87cc695e3bc6" containerName="payload-pre-processing" containerID="cri-o://061df28aa95afcc8d92beda098a57dd66af871ffc70f06f0862da7eaab76f93e" gracePeriod=30
Sep 15 12:36:09.829011 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:36:09.828936    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7b204e-67b594784b-jxsnm"
Sep 15 12:36:09.892340 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:36:09.892301    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95514c8a-f25f-4d43-88a3-87cc695e3bc6-plugins-config-volume\") pod \"95514c8a-f25f-4d43-88a3-87cc695e3bc6\" (UID: \"95514c8a-f25f-4d43-88a3-87cc695e3bc6\") "
Sep 15 12:36:09.892513 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:36:09.892355    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-krr4z\" (UniqueName: \"kubernetes.io/projected/95514c8a-f25f-4d43-88a3-87cc695e3bc6-kube-api-access-krr4z\") pod \"95514c8a-f25f-4d43-88a3-87cc695e3bc6\" (UID: \"95514c8a-f25f-4d43-88a3-87cc695e3bc6\") "
Sep 15 12:36:09.892698 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:36:09.892674    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95514c8a-f25f-4d43-88a3-87cc695e3bc6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "95514c8a-f25f-4d43-88a3-87cc695e3bc6" (UID: "95514c8a-f25f-4d43-88a3-87cc695e3bc6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:36:09.894637 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:36:09.894611    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95514c8a-f25f-4d43-88a3-87cc695e3bc6-kube-api-access-krr4z" (OuterVolumeSpecName: "kube-api-access-krr4z") pod "95514c8a-f25f-4d43-88a3-87cc695e3bc6" (UID: "95514c8a-f25f-4d43-88a3-87cc695e3bc6"). InnerVolumeSpecName "kube-api-access-krr4z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:36:09.993854 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:36:09.993812    2555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95514c8a-f25f-4d43-88a3-87cc695e3bc6-plugins-config-volume\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:36:09.993854 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:36:09.993846    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krr4z\" (UniqueName: \"kubernetes.io/projected/95514c8a-f25f-4d43-88a3-87cc695e3bc6-kube-api-access-krr4z\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:36:10.103881 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:36:10.103788    2555 generic.go:356] "Generic (PLEG): container finished" podID="95514c8a-f25f-4d43-88a3-87cc695e3bc6" containerID="061df28aa95afcc8d92beda098a57dd66af871ffc70f06f0862da7eaab76f93e" exitCode=0
Sep 15 12:36:10.103881 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:36:10.103826    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7b204e-67b594784b-jxsnm"
Sep 15 12:36:10.103881 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:36:10.103867    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7b204e-67b594784b-jxsnm" event={"ID":"95514c8a-f25f-4d43-88a3-87cc695e3bc6","Type":"ContainerDied","Data":"061df28aa95afcc8d92beda098a57dd66af871ffc70f06f0862da7eaab76f93e"}
Sep 15 12:36:10.104169 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:36:10.103927    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7b204e-67b594784b-jxsnm" event={"ID":"95514c8a-f25f-4d43-88a3-87cc695e3bc6","Type":"ContainerDied","Data":"d39ac5bc15395244975c9618c517cd69a9b22162e3319e4ba80fdf992fef4cd2"}
Sep 15 12:36:10.104169 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:36:10.103955    2555 scope.go:117] "RemoveContainer" containerID="061df28aa95afcc8d92beda098a57dd66af871ffc70f06f0862da7eaab76f93e"
Sep 15 12:36:10.113194 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:36:10.113176    2555 scope.go:117] "RemoveContainer" containerID="061df28aa95afcc8d92beda098a57dd66af871ffc70f06f0862da7eaab76f93e"
Sep 15 12:36:10.113483 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:36:10.113466    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"061df28aa95afcc8d92beda098a57dd66af871ffc70f06f0862da7eaab76f93e\": container with ID starting with 061df28aa95afcc8d92beda098a57dd66af871ffc70f06f0862da7eaab76f93e not found: ID does not exist" containerID="061df28aa95afcc8d92beda098a57dd66af871ffc70f06f0862da7eaab76f93e"
Sep 15 12:36:10.113531 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:36:10.113492    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"061df28aa95afcc8d92beda098a57dd66af871ffc70f06f0862da7eaab76f93e"} err="failed to get container status \"061df28aa95afcc8d92beda098a57dd66af871ffc70f06f0862da7eaab76f93e\": rpc error: code = NotFound desc = could not find container \"061df28aa95afcc8d92beda098a57dd66af871ffc70f06f0862da7eaab76f93e\": container with ID starting with 061df28aa95afcc8d92beda098a57dd66af871ffc70f06f0862da7eaab76f93e not found: ID does not exist"
Sep 15 12:36:10.128389 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:36:10.128351    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7b204e-67b594784b-jxsnm"]
Sep 15 12:36:10.134882 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:36:10.134849    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7b204e-67b594784b-jxsnm"]
Sep 15 12:36:11.816690 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:36:11.816647    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95514c8a-f25f-4d43-88a3-87cc695e3bc6" path="/var/lib/kubelet/pods/95514c8a-f25f-4d43-88a3-87cc695e3bc6/volumes"
Sep 15 12:36:12.973157 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:36:12.973116    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5gkxs"]
Sep 15 12:36:16.192547 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:36:16.192509    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-7b204e-openshift-default-55685c44c6-l58rv"]
Sep 15 12:36:16.193102 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:36:16.192778    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-7b204e-openshift-default-55685c44c6-l58rv" podUID="4e2ec4b3-4578-47a6-b505-ba920b1fa4d0" containerName="istio-proxy" containerID="cri-o://8bd42e9cd472f5b957e0442e31872a33b2dd95feffbdc008a8fb1c7c4bcfc41d" gracePeriod=30
Sep 15 12:36:21.451955 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:36:21.451927    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-7b204e-openshift-default-55685c44c6-l58rv"
Sep 15 12:36:21.498893 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:36:21.498787    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e2ec4b3-4578-47a6-b505-ba920b1fa4d0-istiod-ca-cert\") pod \"4e2ec4b3-4578-47a6-b505-ba920b1fa4d0\" (UID: \"4e2ec4b3-4578-47a6-b505-ba920b1fa4d0\") "
Sep 15 12:36:21.498893 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:36:21.498835    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e2ec4b3-4578-47a6-b505-ba920b1fa4d0-istio-token\") pod \"4e2ec4b3-4578-47a6-b505-ba920b1fa4d0\" (UID: \"4e2ec4b3-4578-47a6-b505-ba920b1fa4d0\") "
Sep 15 12:36:21.498893 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:36:21.498895    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e2ec4b3-4578-47a6-b505-ba920b1fa4d0-istio-data\") pod \"4e2ec4b3-4578-47a6-b505-ba920b1fa4d0\" (UID: \"4e2ec4b3-4578-47a6-b505-ba920b1fa4d0\") "
Sep 15 12:36:21.499175 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:36:21.498942    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e2ec4b3-4578-47a6-b505-ba920b1fa4d0-credential-socket\") pod \"4e2ec4b3-4578-47a6-b505-ba920b1fa4d0\" (UID: \"4e2ec4b3-4578-47a6-b505-ba920b1fa4d0\") "
Sep 15 12:36:21.499175 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:36:21.498969    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e2ec4b3-4578-47a6-b505-ba920b1fa4d0-istio-envoy\") pod \"4e2ec4b3-4578-47a6-b505-ba920b1fa4d0\" (UID: \"4e2ec4b3-4578-47a6-b505-ba920b1fa4d0\") "
Sep 15 12:36:21.499175 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:36:21.499005    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hm7sj\" (UniqueName: \"kubernetes.io/projected/4e2ec4b3-4578-47a6-b505-ba920b1fa4d0-kube-api-access-hm7sj\") pod \"4e2ec4b3-4578-47a6-b505-ba920b1fa4d0\" (UID: \"4e2ec4b3-4578-47a6-b505-ba920b1fa4d0\") "
Sep 15 12:36:21.499175 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:36:21.499039    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e2ec4b3-4578-47a6-b505-ba920b1fa4d0-workload-socket\") pod \"4e2ec4b3-4578-47a6-b505-ba920b1fa4d0\" (UID: \"4e2ec4b3-4578-47a6-b505-ba920b1fa4d0\") "
Sep 15 12:36:21.499175 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:36:21.499070    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e2ec4b3-4578-47a6-b505-ba920b1fa4d0-istio-podinfo\") pod \"4e2ec4b3-4578-47a6-b505-ba920b1fa4d0\" (UID: \"4e2ec4b3-4578-47a6-b505-ba920b1fa4d0\") "
Sep 15 12:36:21.499175 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:36:21.499105    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e2ec4b3-4578-47a6-b505-ba920b1fa4d0-workload-certs\") pod \"4e2ec4b3-4578-47a6-b505-ba920b1fa4d0\" (UID: \"4e2ec4b3-4578-47a6-b505-ba920b1fa4d0\") "
Sep 15 12:36:21.499528 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:36:21.499321    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e2ec4b3-4578-47a6-b505-ba920b1fa4d0-istio-data" (OuterVolumeSpecName: "istio-data") pod "4e2ec4b3-4578-47a6-b505-ba920b1fa4d0" (UID: "4e2ec4b3-4578-47a6-b505-ba920b1fa4d0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:36:21.499528 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:36:21.499349    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e2ec4b3-4578-47a6-b505-ba920b1fa4d0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4e2ec4b3-4578-47a6-b505-ba920b1fa4d0" (UID: "4e2ec4b3-4578-47a6-b505-ba920b1fa4d0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:36:21.499528 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:36:21.499355    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e2ec4b3-4578-47a6-b505-ba920b1fa4d0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4e2ec4b3-4578-47a6-b505-ba920b1fa4d0" (UID: "4e2ec4b3-4578-47a6-b505-ba920b1fa4d0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:36:21.499528 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:36:21.499453    2555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e2ec4b3-4578-47a6-b505-ba920b1fa4d0-istiod-ca-cert\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:36:21.499528 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:36:21.499469    2555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e2ec4b3-4578-47a6-b505-ba920b1fa4d0-istio-data\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:36:21.499777 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:36:21.499555    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e2ec4b3-4578-47a6-b505-ba920b1fa4d0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4e2ec4b3-4578-47a6-b505-ba920b1fa4d0" (UID: "4e2ec4b3-4578-47a6-b505-ba920b1fa4d0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:36:21.499777 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:36:21.499596    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e2ec4b3-4578-47a6-b505-ba920b1fa4d0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4e2ec4b3-4578-47a6-b505-ba920b1fa4d0" (UID: "4e2ec4b3-4578-47a6-b505-ba920b1fa4d0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:36:21.501172 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:36:21.501149    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e2ec4b3-4578-47a6-b505-ba920b1fa4d0-istio-token" (OuterVolumeSpecName: "istio-token") pod "4e2ec4b3-4578-47a6-b505-ba920b1fa4d0" (UID: "4e2ec4b3-4578-47a6-b505-ba920b1fa4d0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:36:21.501311 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:36:21.501225    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e2ec4b3-4578-47a6-b505-ba920b1fa4d0-kube-api-access-hm7sj" (OuterVolumeSpecName: "kube-api-access-hm7sj") pod "4e2ec4b3-4578-47a6-b505-ba920b1fa4d0" (UID: "4e2ec4b3-4578-47a6-b505-ba920b1fa4d0"). InnerVolumeSpecName "kube-api-access-hm7sj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:36:21.501771 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:36:21.501752    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e2ec4b3-4578-47a6-b505-ba920b1fa4d0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4e2ec4b3-4578-47a6-b505-ba920b1fa4d0" (UID: "4e2ec4b3-4578-47a6-b505-ba920b1fa4d0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:36:21.501771 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:36:21.501766    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4e2ec4b3-4578-47a6-b505-ba920b1fa4d0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4e2ec4b3-4578-47a6-b505-ba920b1fa4d0" (UID: "4e2ec4b3-4578-47a6-b505-ba920b1fa4d0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 12:36:21.600207 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:36:21.600171    2555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e2ec4b3-4578-47a6-b505-ba920b1fa4d0-workload-socket\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:36:21.600207 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:36:21.600200    2555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e2ec4b3-4578-47a6-b505-ba920b1fa4d0-istio-podinfo\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:36:21.600207 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:36:21.600210    2555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e2ec4b3-4578-47a6-b505-ba920b1fa4d0-workload-certs\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:36:21.600207 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:36:21.600218    2555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e2ec4b3-4578-47a6-b505-ba920b1fa4d0-istio-token\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:36:21.600520 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:36:21.600227    2555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e2ec4b3-4578-47a6-b505-ba920b1fa4d0-credential-socket\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:36:21.600520 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:36:21.600238    2555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e2ec4b3-4578-47a6-b505-ba920b1fa4d0-istio-envoy\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:36:21.600520 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:36:21.600247    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hm7sj\" (UniqueName: \"kubernetes.io/projected/4e2ec4b3-4578-47a6-b505-ba920b1fa4d0-kube-api-access-hm7sj\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:36:22.174184 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:36:22.174145    2555 generic.go:356] "Generic (PLEG): container finished" podID="4e2ec4b3-4578-47a6-b505-ba920b1fa4d0" containerID="8bd42e9cd472f5b957e0442e31872a33b2dd95feffbdc008a8fb1c7c4bcfc41d" exitCode=0
Sep 15 12:36:22.174436 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:36:22.174198    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-7b204e-openshift-default-55685c44c6-l58rv"
Sep 15 12:36:22.174436 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:36:22.174235    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7b204e-openshift-default-55685c44c6-l58rv" event={"ID":"4e2ec4b3-4578-47a6-b505-ba920b1fa4d0","Type":"ContainerDied","Data":"8bd42e9cd472f5b957e0442e31872a33b2dd95feffbdc008a8fb1c7c4bcfc41d"}
Sep 15 12:36:22.174436 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:36:22.174328    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7b204e-openshift-default-55685c44c6-l58rv" event={"ID":"4e2ec4b3-4578-47a6-b505-ba920b1fa4d0","Type":"ContainerDied","Data":"5b4984ec35488f038eff84769653f9302faf93bc13e4ba28bfb3e651df12e611"}
Sep 15 12:36:22.174436 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:36:22.174362    2555 scope.go:117] "RemoveContainer" containerID="8bd42e9cd472f5b957e0442e31872a33b2dd95feffbdc008a8fb1c7c4bcfc41d"
Sep 15 12:36:22.183798 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:36:22.183779    2555 scope.go:117] "RemoveContainer" containerID="4ba55b6e1c1ff55fa31ad947c212618a9345e54f3a9d9bbe069b55e1d0045f6d"
Sep 15 12:36:22.193108 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:36:22.193082    2555 scope.go:117] "RemoveContainer" containerID="8bd42e9cd472f5b957e0442e31872a33b2dd95feffbdc008a8fb1c7c4bcfc41d"
Sep 15 12:36:22.193473 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:36:22.193451    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bd42e9cd472f5b957e0442e31872a33b2dd95feffbdc008a8fb1c7c4bcfc41d\": container with ID starting with 8bd42e9cd472f5b957e0442e31872a33b2dd95feffbdc008a8fb1c7c4bcfc41d not found: ID does not exist" containerID="8bd42e9cd472f5b957e0442e31872a33b2dd95feffbdc008a8fb1c7c4bcfc41d"
Sep 15 12:36:22.193547 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:36:22.193487    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bd42e9cd472f5b957e0442e31872a33b2dd95feffbdc008a8fb1c7c4bcfc41d"} err="failed to get container status \"8bd42e9cd472f5b957e0442e31872a33b2dd95feffbdc008a8fb1c7c4bcfc41d\": rpc error: code = NotFound desc = could not find container \"8bd42e9cd472f5b957e0442e31872a33b2dd95feffbdc008a8fb1c7c4bcfc41d\": container with ID starting with 8bd42e9cd472f5b957e0442e31872a33b2dd95feffbdc008a8fb1c7c4bcfc41d not found: ID does not exist"
Sep 15 12:36:22.193547 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:36:22.193507    2555 scope.go:117] "RemoveContainer" containerID="4ba55b6e1c1ff55fa31ad947c212618a9345e54f3a9d9bbe069b55e1d0045f6d"
Sep 15 12:36:22.193676 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:36:22.193659    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-7b204e-openshift-default-55685c44c6-l58rv"]
Sep 15 12:36:22.193778 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:36:22.193759    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ba55b6e1c1ff55fa31ad947c212618a9345e54f3a9d9bbe069b55e1d0045f6d\": container with ID starting with 4ba55b6e1c1ff55fa31ad947c212618a9345e54f3a9d9bbe069b55e1d0045f6d not found: ID does not exist" containerID="4ba55b6e1c1ff55fa31ad947c212618a9345e54f3a9d9bbe069b55e1d0045f6d"
Sep 15 12:36:22.193818 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:36:22.193784    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ba55b6e1c1ff55fa31ad947c212618a9345e54f3a9d9bbe069b55e1d0045f6d"} err="failed to get container status \"4ba55b6e1c1ff55fa31ad947c212618a9345e54f3a9d9bbe069b55e1d0045f6d\": rpc error: code = NotFound desc = could not find container \"4ba55b6e1c1ff55fa31ad947c212618a9345e54f3a9d9bbe069b55e1d0045f6d\": container with ID starting with 4ba55b6e1c1ff55fa31ad947c212618a9345e54f3a9d9bbe069b55e1d0045f6d not found: ID does not exist"
Sep 15 12:36:22.197808 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:36:22.197783    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-7b204e-openshift-default-55685c44c6-l58rv"]
Sep 15 12:36:23.816763 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:36:23.816724    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e2ec4b3-4578-47a6-b505-ba920b1fa4d0" path="/var/lib/kubelet/pods/4e2ec4b3-4578-47a6-b505-ba920b1fa4d0/volumes"
Sep 15 12:36:24.308539 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:36:24.308506    2555 scope.go:117] "RemoveContainer" containerID="064516be9cf9eb65600e2d882e09ad4f714b85e7ec269b3f3605a47856a5bdaa"
Sep 15 12:36:27.150645 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:36:27.150612    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-1dc430-54b65bcb7c-l42f8"]
Sep 15 12:36:27.151257 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:36:27.150818    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1dc430-54b65bcb7c-l42f8" podUID="d529d4a5-2f68-419d-b739-f0e3b2abdd81" containerName="payload-processing" containerID="cri-o://48601119675edacf519d389b18eb34458cd4cbcccf2b531adf2f9669449529d6" gracePeriod=30
Sep 15 12:36:27.400098 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:36:27.400072    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1dc430-54b65bcb7c-l42f8"
Sep 15 12:36:27.451376 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:36:27.451291    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d9xfr\" (UniqueName: \"kubernetes.io/projected/d529d4a5-2f68-419d-b739-f0e3b2abdd81-kube-api-access-d9xfr\") pod \"d529d4a5-2f68-419d-b739-f0e3b2abdd81\" (UID: \"d529d4a5-2f68-419d-b739-f0e3b2abdd81\") "
Sep 15 12:36:27.451529 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:36:27.451444    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d529d4a5-2f68-419d-b739-f0e3b2abdd81-plugins-config-volume\") pod \"d529d4a5-2f68-419d-b739-f0e3b2abdd81\" (UID: \"d529d4a5-2f68-419d-b739-f0e3b2abdd81\") "
Sep 15 12:36:27.451903 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:36:27.451867    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d529d4a5-2f68-419d-b739-f0e3b2abdd81-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d529d4a5-2f68-419d-b739-f0e3b2abdd81" (UID: "d529d4a5-2f68-419d-b739-f0e3b2abdd81"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:36:27.453462 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:36:27.453437    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d529d4a5-2f68-419d-b739-f0e3b2abdd81-kube-api-access-d9xfr" (OuterVolumeSpecName: "kube-api-access-d9xfr") pod "d529d4a5-2f68-419d-b739-f0e3b2abdd81" (UID: "d529d4a5-2f68-419d-b739-f0e3b2abdd81"). InnerVolumeSpecName "kube-api-access-d9xfr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:36:27.552671 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:36:27.552637    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9xfr\" (UniqueName: \"kubernetes.io/projected/d529d4a5-2f68-419d-b739-f0e3b2abdd81-kube-api-access-d9xfr\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:36:27.552671 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:36:27.552668    2555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d529d4a5-2f68-419d-b739-f0e3b2abdd81-plugins-config-volume\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:36:28.208462 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:36:28.208426    2555 generic.go:356] "Generic (PLEG): container finished" podID="d529d4a5-2f68-419d-b739-f0e3b2abdd81" containerID="48601119675edacf519d389b18eb34458cd4cbcccf2b531adf2f9669449529d6" exitCode=0
Sep 15 12:36:28.208962 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:36:28.208480    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1dc430-54b65bcb7c-l42f8"
Sep 15 12:36:28.208962 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:36:28.208500    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1dc430-54b65bcb7c-l42f8" event={"ID":"d529d4a5-2f68-419d-b739-f0e3b2abdd81","Type":"ContainerDied","Data":"48601119675edacf519d389b18eb34458cd4cbcccf2b531adf2f9669449529d6"}
Sep 15 12:36:28.208962 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:36:28.208563    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1dc430-54b65bcb7c-l42f8" event={"ID":"d529d4a5-2f68-419d-b739-f0e3b2abdd81","Type":"ContainerDied","Data":"ce05a23ec3f54dad1ecbbbfba6e1333f4324a0f5a76da00121d0743790531143"}
Sep 15 12:36:28.208962 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:36:28.208594    2555 scope.go:117] "RemoveContainer" containerID="48601119675edacf519d389b18eb34458cd4cbcccf2b531adf2f9669449529d6"
Sep 15 12:36:28.217465 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:36:28.217445    2555 scope.go:117] "RemoveContainer" containerID="48601119675edacf519d389b18eb34458cd4cbcccf2b531adf2f9669449529d6"
Sep 15 12:36:28.217781 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:36:28.217756    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48601119675edacf519d389b18eb34458cd4cbcccf2b531adf2f9669449529d6\": container with ID starting with 48601119675edacf519d389b18eb34458cd4cbcccf2b531adf2f9669449529d6 not found: ID does not exist" containerID="48601119675edacf519d389b18eb34458cd4cbcccf2b531adf2f9669449529d6"
Sep 15 12:36:28.217866 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:36:28.217793    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48601119675edacf519d389b18eb34458cd4cbcccf2b531adf2f9669449529d6"} err="failed to get container status \"48601119675edacf519d389b18eb34458cd4cbcccf2b531adf2f9669449529d6\": rpc error: code = NotFound desc = could not find container \"48601119675edacf519d389b18eb34458cd4cbcccf2b531adf2f9669449529d6\": container with ID starting with 48601119675edacf519d389b18eb34458cd4cbcccf2b531adf2f9669449529d6 not found: ID does not exist"
Sep 15 12:36:28.225865 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:36:28.225829    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-1dc430-54b65bcb7c-l42f8"]
Sep 15 12:36:28.230491 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:36:28.230467    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-1dc430-54b65bcb7c-l42f8"]
Sep 15 12:36:29.817489 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:36:29.817442    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d529d4a5-2f68-419d-b739-f0e3b2abdd81" path="/var/lib/kubelet/pods/d529d4a5-2f68-419d-b739-f0e3b2abdd81/volumes"
Sep 15 12:36:30.064074 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:36:30.064036    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5gkxs"]
Sep 15 12:36:44.748865 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:36:44.748820    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-969130-29824590-vk95z"]
Sep 15 12:36:44.752403 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:36:44.752371    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-969130-29824590-vk95z"]
Sep 15 12:36:45.817068 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:36:45.817030    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95baa04b-f10c-456c-9bd6-403dbf093a75" path="/var/lib/kubelet/pods/95baa04b-f10c-456c-9bd6-403dbf093a75/volumes"
Sep 15 12:36:47.173623 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:36:47.173590    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5gkxs"]
Sep 15 12:36:56.275665 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:36:56.275624    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5gkxs"]
Sep 15 12:37:24.341042 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:37:24.341008    2555 scope.go:117] "RemoveContainer" containerID="dc51035b271e40e8ae2f41b320c74d9f9c537247354ae580590401bfe164dee6"
Sep 15 12:38:16.561557 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:38:16.561469    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5gkxs"]
Sep 15 12:38:40.766325 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:38:40.766282    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5gkxs"]
Sep 15 12:38:48.369679 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:38:48.369645    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5gkxs"]
Sep 15 12:38:55.670377 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:38:55.670332    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5gkxs"]
Sep 15 12:39:03.269540 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:39:03.269495    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5gkxs"]
Sep 15 12:39:17.568815 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:39:17.568774    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5gkxs"]
Sep 15 12:39:25.869493 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:39:25.869441    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5gkxs"]
Sep 15 12:40:24.380682 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:40:24.380642    2555 scope.go:117] "RemoveContainer" containerID="39243fc2f55ee76502ce314961df7b6c1aa930606edd78fe944623dccfac6904"
Sep 15 12:40:53.162449 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:40:53.162407    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5gkxs"]
Sep 15 12:41:01.173311 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:41:01.173275    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5gkxs"]
Sep 15 12:41:06.868572 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:41:06.868534    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5gkxs"]
Sep 15 12:41:08.109594 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:41:08.109562    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-4ee1883c-kserve-575874f4bc-jdkf2"]
Sep 15 12:41:08.109980 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:41:08.109928    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e2ec4b3-4578-47a6-b505-ba920b1fa4d0" containerName="istio-proxy"
Sep 15 12:41:08.109980 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:41:08.109943    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e2ec4b3-4578-47a6-b505-ba920b1fa4d0" containerName="istio-proxy"
Sep 15 12:41:08.109980 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:41:08.109955    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d529d4a5-2f68-419d-b739-f0e3b2abdd81" containerName="payload-processing"
Sep 15 12:41:08.109980 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:41:08.109961    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="d529d4a5-2f68-419d-b739-f0e3b2abdd81" containerName="payload-processing"
Sep 15 12:41:08.109980 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:41:08.109974    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0494a388-936d-4884-a916-bfee1aab24cc" containerName="payload-pre-processing"
Sep 15 12:41:08.109980 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:41:08.109980    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="0494a388-936d-4884-a916-bfee1aab24cc" containerName="payload-pre-processing"
Sep 15 12:41:08.110188 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:41:08.109991    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1adc44fa-0e09-4bf9-a8f0-e95522b6b507" containerName="main"
Sep 15 12:41:08.110188 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:41:08.110002    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="1adc44fa-0e09-4bf9-a8f0-e95522b6b507" containerName="main"
Sep 15 12:41:08.110188 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:41:08.110008    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0d829eb-1ca4-49f3-97c5-31e53b9db1d7" containerName="revoke-keys"
Sep 15 12:41:08.110188 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:41:08.110013    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0d829eb-1ca4-49f3-97c5-31e53b9db1d7" containerName="revoke-keys"
Sep 15 12:41:08.110188 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:41:08.110022    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e2ec4b3-4578-47a6-b505-ba920b1fa4d0" containerName="istio-proxy"
Sep 15 12:41:08.110188 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:41:08.110027    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e2ec4b3-4578-47a6-b505-ba920b1fa4d0" containerName="istio-proxy"
Sep 15 12:41:08.110188 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:41:08.110035    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95514c8a-f25f-4d43-88a3-87cc695e3bc6" containerName="payload-pre-processing"
Sep 15 12:41:08.110188 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:41:08.110040    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="95514c8a-f25f-4d43-88a3-87cc695e3bc6" containerName="payload-pre-processing"
Sep 15 12:41:08.110188 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:41:08.110097    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="d529d4a5-2f68-419d-b739-f0e3b2abdd81" containerName="payload-processing"
Sep 15 12:41:08.110188 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:41:08.110105    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="95514c8a-f25f-4d43-88a3-87cc695e3bc6" containerName="payload-pre-processing"
Sep 15 12:41:08.110188 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:41:08.110112    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e2ec4b3-4578-47a6-b505-ba920b1fa4d0" containerName="istio-proxy"
Sep 15 12:41:08.110188 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:41:08.110117    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0d829eb-1ca4-49f3-97c5-31e53b9db1d7" containerName="revoke-keys"
Sep 15 12:41:08.110188 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:41:08.110124    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="0494a388-936d-4884-a916-bfee1aab24cc" containerName="payload-pre-processing"
Sep 15 12:41:08.110188 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:41:08.110130    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e2ec4b3-4578-47a6-b505-ba920b1fa4d0" containerName="istio-proxy"
Sep 15 12:41:08.110188 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:41:08.110135    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="1adc44fa-0e09-4bf9-a8f0-e95522b6b507" containerName="main"
Sep 15 12:41:08.113046 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:41:08.113030    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-4ee1883c-kserve-575874f4bc-jdkf2"
Sep 15 12:41:08.115837 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:41:08.115816    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-4ee1883c-kserve-self-signed-certs\""
Sep 15 12:41:08.121130 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:41:08.121109    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-4ee1883c-kserve-575874f4bc-jdkf2"]
Sep 15 12:41:08.191205 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:41:08.191170    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/704a150b-a1cf-44c8-a03e-c9d615473943-tmp-dir\") pod \"e2e-same-id-a-4ee1883c-kserve-575874f4bc-jdkf2\" (UID: \"704a150b-a1cf-44c8-a03e-c9d615473943\") " pod="llm/e2e-same-id-a-4ee1883c-kserve-575874f4bc-jdkf2"
Sep 15 12:41:08.191435 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:41:08.191223    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/704a150b-a1cf-44c8-a03e-c9d615473943-tls-certs\") pod \"e2e-same-id-a-4ee1883c-kserve-575874f4bc-jdkf2\" (UID: \"704a150b-a1cf-44c8-a03e-c9d615473943\") " pod="llm/e2e-same-id-a-4ee1883c-kserve-575874f4bc-jdkf2"
Sep 15 12:41:08.191435 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:41:08.191280    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/704a150b-a1cf-44c8-a03e-c9d615473943-home\") pod \"e2e-same-id-a-4ee1883c-kserve-575874f4bc-jdkf2\" (UID: \"704a150b-a1cf-44c8-a03e-c9d615473943\") " pod="llm/e2e-same-id-a-4ee1883c-kserve-575874f4bc-jdkf2"
Sep 15 12:41:08.191435 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:41:08.191354    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmfdv\" (UniqueName: \"kubernetes.io/projected/704a150b-a1cf-44c8-a03e-c9d615473943-kube-api-access-lmfdv\") pod \"e2e-same-id-a-4ee1883c-kserve-575874f4bc-jdkf2\" (UID: \"704a150b-a1cf-44c8-a03e-c9d615473943\") " pod="llm/e2e-same-id-a-4ee1883c-kserve-575874f4bc-jdkf2"
Sep 15 12:41:08.191435 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:41:08.191408    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/704a150b-a1cf-44c8-a03e-c9d615473943-dshm\") pod \"e2e-same-id-a-4ee1883c-kserve-575874f4bc-jdkf2\" (UID: \"704a150b-a1cf-44c8-a03e-c9d615473943\") " pod="llm/e2e-same-id-a-4ee1883c-kserve-575874f4bc-jdkf2"
Sep 15 12:41:08.191628 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:41:08.191481    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/704a150b-a1cf-44c8-a03e-c9d615473943-model-cache\") pod \"e2e-same-id-a-4ee1883c-kserve-575874f4bc-jdkf2\" (UID: \"704a150b-a1cf-44c8-a03e-c9d615473943\") " pod="llm/e2e-same-id-a-4ee1883c-kserve-575874f4bc-jdkf2"
Sep 15 12:41:08.292560 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:41:08.292512    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/704a150b-a1cf-44c8-a03e-c9d615473943-dshm\") pod \"e2e-same-id-a-4ee1883c-kserve-575874f4bc-jdkf2\" (UID: \"704a150b-a1cf-44c8-a03e-c9d615473943\") " pod="llm/e2e-same-id-a-4ee1883c-kserve-575874f4bc-jdkf2"
Sep 15 12:41:08.292784 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:41:08.292583    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/704a150b-a1cf-44c8-a03e-c9d615473943-model-cache\") pod \"e2e-same-id-a-4ee1883c-kserve-575874f4bc-jdkf2\" (UID: \"704a150b-a1cf-44c8-a03e-c9d615473943\") " pod="llm/e2e-same-id-a-4ee1883c-kserve-575874f4bc-jdkf2"
Sep 15 12:41:08.292784 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:41:08.292640    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/704a150b-a1cf-44c8-a03e-c9d615473943-tmp-dir\") pod \"e2e-same-id-a-4ee1883c-kserve-575874f4bc-jdkf2\" (UID: \"704a150b-a1cf-44c8-a03e-c9d615473943\") " pod="llm/e2e-same-id-a-4ee1883c-kserve-575874f4bc-jdkf2"
Sep 15 12:41:08.292784 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:41:08.292686    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/704a150b-a1cf-44c8-a03e-c9d615473943-tls-certs\") pod \"e2e-same-id-a-4ee1883c-kserve-575874f4bc-jdkf2\" (UID: \"704a150b-a1cf-44c8-a03e-c9d615473943\") " pod="llm/e2e-same-id-a-4ee1883c-kserve-575874f4bc-jdkf2"
Sep 15 12:41:08.292784 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:41:08.292716    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/704a150b-a1cf-44c8-a03e-c9d615473943-home\") pod \"e2e-same-id-a-4ee1883c-kserve-575874f4bc-jdkf2\" (UID: \"704a150b-a1cf-44c8-a03e-c9d615473943\") " pod="llm/e2e-same-id-a-4ee1883c-kserve-575874f4bc-jdkf2"
Sep 15 12:41:08.292784 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:41:08.292743    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmfdv\" (UniqueName: \"kubernetes.io/projected/704a150b-a1cf-44c8-a03e-c9d615473943-kube-api-access-lmfdv\") pod \"e2e-same-id-a-4ee1883c-kserve-575874f4bc-jdkf2\" (UID: \"704a150b-a1cf-44c8-a03e-c9d615473943\") " pod="llm/e2e-same-id-a-4ee1883c-kserve-575874f4bc-jdkf2"
Sep 15 12:41:08.293106 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:41:08.293083    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/704a150b-a1cf-44c8-a03e-c9d615473943-model-cache\") pod \"e2e-same-id-a-4ee1883c-kserve-575874f4bc-jdkf2\" (UID: \"704a150b-a1cf-44c8-a03e-c9d615473943\") " pod="llm/e2e-same-id-a-4ee1883c-kserve-575874f4bc-jdkf2"
Sep 15 12:41:08.293164 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:41:08.293111    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/704a150b-a1cf-44c8-a03e-c9d615473943-tmp-dir\") pod \"e2e-same-id-a-4ee1883c-kserve-575874f4bc-jdkf2\" (UID: \"704a150b-a1cf-44c8-a03e-c9d615473943\") " pod="llm/e2e-same-id-a-4ee1883c-kserve-575874f4bc-jdkf2"
Sep 15 12:41:08.293201 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:41:08.293168    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/704a150b-a1cf-44c8-a03e-c9d615473943-home\") pod \"e2e-same-id-a-4ee1883c-kserve-575874f4bc-jdkf2\" (UID: \"704a150b-a1cf-44c8-a03e-c9d615473943\") " pod="llm/e2e-same-id-a-4ee1883c-kserve-575874f4bc-jdkf2"
Sep 15 12:41:08.294954 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:41:08.294934    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/704a150b-a1cf-44c8-a03e-c9d615473943-dshm\") pod \"e2e-same-id-a-4ee1883c-kserve-575874f4bc-jdkf2\" (UID: \"704a150b-a1cf-44c8-a03e-c9d615473943\") " pod="llm/e2e-same-id-a-4ee1883c-kserve-575874f4bc-jdkf2"
Sep 15 12:41:08.295201 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:41:08.295184    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/704a150b-a1cf-44c8-a03e-c9d615473943-tls-certs\") pod \"e2e-same-id-a-4ee1883c-kserve-575874f4bc-jdkf2\" (UID: \"704a150b-a1cf-44c8-a03e-c9d615473943\") " pod="llm/e2e-same-id-a-4ee1883c-kserve-575874f4bc-jdkf2"
Sep 15 12:41:08.303150 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:41:08.303125    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmfdv\" (UniqueName: \"kubernetes.io/projected/704a150b-a1cf-44c8-a03e-c9d615473943-kube-api-access-lmfdv\") pod \"e2e-same-id-a-4ee1883c-kserve-575874f4bc-jdkf2\" (UID: \"704a150b-a1cf-44c8-a03e-c9d615473943\") " pod="llm/e2e-same-id-a-4ee1883c-kserve-575874f4bc-jdkf2"
Sep 15 12:41:08.423647 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:41:08.423606    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-4ee1883c-kserve-575874f4bc-jdkf2"
Sep 15 12:41:08.697962 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:41:08.697936    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-4ee1883c-kserve-575874f4bc-jdkf2"]
Sep 15 12:41:08.699840 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:41:08.699815    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod704a150b_a1cf_44c8_a03e_c9d615473943.slice/crio-0fec5a1a053f7fad66be614206de4258c24c4cb37190d6a5c194d5b1791611fc WatchSource:0}: Error finding container 0fec5a1a053f7fad66be614206de4258c24c4cb37190d6a5c194d5b1791611fc: Status 404 returned error can't find the container with id 0fec5a1a053f7fad66be614206de4258c24c4cb37190d6a5c194d5b1791611fc
Sep 15 12:41:09.684594 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:41:09.684552    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-4ee1883c-kserve-575874f4bc-jdkf2" event={"ID":"704a150b-a1cf-44c8-a03e-c9d615473943","Type":"ContainerStarted","Data":"81578fb0bf59bc50e3b090afbaba644ce663463f4cb0f1faccc082fc854bac19"}
Sep 15 12:41:09.684594 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:41:09.684600    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-4ee1883c-kserve-575874f4bc-jdkf2" event={"ID":"704a150b-a1cf-44c8-a03e-c9d615473943","Type":"ContainerStarted","Data":"0fec5a1a053f7fad66be614206de4258c24c4cb37190d6a5c194d5b1791611fc"}
Sep 15 12:41:09.703053 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:41:09.703000    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-4ee1883c-kserve-575874f4bc-jdkf2" podStartSLOduration=1.702983771 podStartE2EDuration="1.702983771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:41:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:41:09.701599678 +0000 UTC m=+2326.492522219" watchObservedRunningTime="2026-09-15 12:41:09.702983771 +0000 UTC m=+2326.493906278"
Sep 15 12:41:15.383280 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:41:15.383224    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5gkxs"]
Sep 15 12:41:18.424187 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:41:18.424119    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-4ee1883c-kserve-575874f4bc-jdkf2"
Sep 15 12:41:18.424187 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:41:18.424201    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-4ee1883c-kserve-575874f4bc-jdkf2"
Sep 15 12:41:18.436685 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:41:18.436658    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-4ee1883c-kserve-575874f4bc-jdkf2"
Sep 15 12:41:18.747098 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:41:18.747011    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-4ee1883c-kserve-575874f4bc-jdkf2"
Sep 15 12:41:24.401445 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:41:24.401412    2555 scope.go:117] "RemoveContainer" containerID="ae59287b17762fc27fd4e4e63192c49d3a92e4c50a0e14701e0ffab3a7d49148"
Sep 15 12:41:37.017468 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:41:37.017433    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-4ee1883c-kserve-7d8d6dc547-75qkw"]
Sep 15 12:41:37.020901 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:41:37.020882    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-4ee1883c-kserve-7d8d6dc547-75qkw"
Sep 15 12:41:37.023702 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:41:37.023680    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-4ee1883c-kserve-self-signed-certs\""
Sep 15 12:41:37.033073 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:41:37.033049    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-4ee1883c-kserve-7d8d6dc547-75qkw"]
Sep 15 12:41:37.042653 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:41:37.042626    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37024d56-dad9-4860-b697-100d375a4318-tls-certs\") pod \"e2e-same-id-b-4ee1883c-kserve-7d8d6dc547-75qkw\" (UID: \"37024d56-dad9-4860-b697-100d375a4318\") " pod="llm/e2e-same-id-b-4ee1883c-kserve-7d8d6dc547-75qkw"
Sep 15 12:41:37.042768 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:41:37.042668    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37024d56-dad9-4860-b697-100d375a4318-home\") pod \"e2e-same-id-b-4ee1883c-kserve-7d8d6dc547-75qkw\" (UID: \"37024d56-dad9-4860-b697-100d375a4318\") " pod="llm/e2e-same-id-b-4ee1883c-kserve-7d8d6dc547-75qkw"
Sep 15 12:41:37.042768 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:41:37.042711    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37024d56-dad9-4860-b697-100d375a4318-dshm\") pod \"e2e-same-id-b-4ee1883c-kserve-7d8d6dc547-75qkw\" (UID: \"37024d56-dad9-4860-b697-100d375a4318\") " pod="llm/e2e-same-id-b-4ee1883c-kserve-7d8d6dc547-75qkw"
Sep 15 12:41:37.042768 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:41:37.042745    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44bdj\" (UniqueName: \"kubernetes.io/projected/37024d56-dad9-4860-b697-100d375a4318-kube-api-access-44bdj\") pod \"e2e-same-id-b-4ee1883c-kserve-7d8d6dc547-75qkw\" (UID: \"37024d56-dad9-4860-b697-100d375a4318\") " pod="llm/e2e-same-id-b-4ee1883c-kserve-7d8d6dc547-75qkw"
Sep 15 12:41:37.042875 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:41:37.042850    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37024d56-dad9-4860-b697-100d375a4318-model-cache\") pod \"e2e-same-id-b-4ee1883c-kserve-7d8d6dc547-75qkw\" (UID: \"37024d56-dad9-4860-b697-100d375a4318\") " pod="llm/e2e-same-id-b-4ee1883c-kserve-7d8d6dc547-75qkw"
Sep 15 12:41:37.042915 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:41:37.042897    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37024d56-dad9-4860-b697-100d375a4318-tmp-dir\") pod \"e2e-same-id-b-4ee1883c-kserve-7d8d6dc547-75qkw\" (UID: \"37024d56-dad9-4860-b697-100d375a4318\") " pod="llm/e2e-same-id-b-4ee1883c-kserve-7d8d6dc547-75qkw"
Sep 15 12:41:37.143630 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:41:37.143590    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37024d56-dad9-4860-b697-100d375a4318-model-cache\") pod \"e2e-same-id-b-4ee1883c-kserve-7d8d6dc547-75qkw\" (UID: \"37024d56-dad9-4860-b697-100d375a4318\") " pod="llm/e2e-same-id-b-4ee1883c-kserve-7d8d6dc547-75qkw"
Sep 15 12:41:37.143838 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:41:37.143644    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37024d56-dad9-4860-b697-100d375a4318-tmp-dir\") pod \"e2e-same-id-b-4ee1883c-kserve-7d8d6dc547-75qkw\" (UID: \"37024d56-dad9-4860-b697-100d375a4318\") " pod="llm/e2e-same-id-b-4ee1883c-kserve-7d8d6dc547-75qkw"
Sep 15 12:41:37.143838 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:41:37.143694    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37024d56-dad9-4860-b697-100d375a4318-tls-certs\") pod \"e2e-same-id-b-4ee1883c-kserve-7d8d6dc547-75qkw\" (UID: \"37024d56-dad9-4860-b697-100d375a4318\") " pod="llm/e2e-same-id-b-4ee1883c-kserve-7d8d6dc547-75qkw"
Sep 15 12:41:37.143838 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:41:37.143719    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37024d56-dad9-4860-b697-100d375a4318-home\") pod \"e2e-same-id-b-4ee1883c-kserve-7d8d6dc547-75qkw\" (UID: \"37024d56-dad9-4860-b697-100d375a4318\") " pod="llm/e2e-same-id-b-4ee1883c-kserve-7d8d6dc547-75qkw"
Sep 15 12:41:37.144016 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:41:37.143853    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37024d56-dad9-4860-b697-100d375a4318-dshm\") pod \"e2e-same-id-b-4ee1883c-kserve-7d8d6dc547-75qkw\" (UID: \"37024d56-dad9-4860-b697-100d375a4318\") " pod="llm/e2e-same-id-b-4ee1883c-kserve-7d8d6dc547-75qkw"
Sep 15 12:41:37.144016 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:41:37.143910    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44bdj\" (UniqueName: \"kubernetes.io/projected/37024d56-dad9-4860-b697-100d375a4318-kube-api-access-44bdj\") pod \"e2e-same-id-b-4ee1883c-kserve-7d8d6dc547-75qkw\" (UID: \"37024d56-dad9-4860-b697-100d375a4318\") " pod="llm/e2e-same-id-b-4ee1883c-kserve-7d8d6dc547-75qkw"
Sep 15 12:41:37.144120 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:41:37.144058    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37024d56-dad9-4860-b697-100d375a4318-model-cache\") pod \"e2e-same-id-b-4ee1883c-kserve-7d8d6dc547-75qkw\" (UID: \"37024d56-dad9-4860-b697-100d375a4318\") " pod="llm/e2e-same-id-b-4ee1883c-kserve-7d8d6dc547-75qkw"
Sep 15 12:41:37.144120 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:41:37.144109    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37024d56-dad9-4860-b697-100d375a4318-tmp-dir\") pod \"e2e-same-id-b-4ee1883c-kserve-7d8d6dc547-75qkw\" (UID: \"37024d56-dad9-4860-b697-100d375a4318\") " pod="llm/e2e-same-id-b-4ee1883c-kserve-7d8d6dc547-75qkw"
Sep 15 12:41:37.144221 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:41:37.144143    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37024d56-dad9-4860-b697-100d375a4318-home\") pod \"e2e-same-id-b-4ee1883c-kserve-7d8d6dc547-75qkw\" (UID: \"37024d56-dad9-4860-b697-100d375a4318\") " pod="llm/e2e-same-id-b-4ee1883c-kserve-7d8d6dc547-75qkw"
Sep 15 12:41:37.146026 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:41:37.146004    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37024d56-dad9-4860-b697-100d375a4318-dshm\") pod \"e2e-same-id-b-4ee1883c-kserve-7d8d6dc547-75qkw\" (UID: \"37024d56-dad9-4860-b697-100d375a4318\") " pod="llm/e2e-same-id-b-4ee1883c-kserve-7d8d6dc547-75qkw"
Sep 15 12:41:37.146190 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:41:37.146172    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37024d56-dad9-4860-b697-100d375a4318-tls-certs\") pod \"e2e-same-id-b-4ee1883c-kserve-7d8d6dc547-75qkw\" (UID: \"37024d56-dad9-4860-b697-100d375a4318\") " pod="llm/e2e-same-id-b-4ee1883c-kserve-7d8d6dc547-75qkw"
Sep 15 12:41:37.152973 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:41:37.152932    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-44bdj\" (UniqueName: \"kubernetes.io/projected/37024d56-dad9-4860-b697-100d375a4318-kube-api-access-44bdj\") pod \"e2e-same-id-b-4ee1883c-kserve-7d8d6dc547-75qkw\" (UID: \"37024d56-dad9-4860-b697-100d375a4318\") " pod="llm/e2e-same-id-b-4ee1883c-kserve-7d8d6dc547-75qkw"
Sep 15 12:41:37.329374 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:41:37.329251    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-4ee1883c-kserve-7d8d6dc547-75qkw"
Sep 15 12:41:37.601734 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:41:37.601702    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-4ee1883c-kserve-7d8d6dc547-75qkw"]
Sep 15 12:41:37.603308 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:41:37.603277    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod37024d56_dad9_4860_b697_100d375a4318.slice/crio-cbecc1956d2b29d7914195be607282f2260d64aa1e01327bb1cf94c26fb59888 WatchSource:0}: Error finding container cbecc1956d2b29d7914195be607282f2260d64aa1e01327bb1cf94c26fb59888: Status 404 returned error can't find the container with id cbecc1956d2b29d7914195be607282f2260d64aa1e01327bb1cf94c26fb59888
Sep 15 12:41:37.833532 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:41:37.833495    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-4ee1883c-kserve-7d8d6dc547-75qkw" event={"ID":"37024d56-dad9-4860-b697-100d375a4318","Type":"ContainerStarted","Data":"0ea3f2d97bc59eb369fea4b623fb8331d6d594c09df2b10afe0ace129e503a6a"}
Sep 15 12:41:37.833736 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:41:37.833538    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-4ee1883c-kserve-7d8d6dc547-75qkw" event={"ID":"37024d56-dad9-4860-b697-100d375a4318","Type":"ContainerStarted","Data":"cbecc1956d2b29d7914195be607282f2260d64aa1e01327bb1cf94c26fb59888"}
Sep 15 12:41:37.851805 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:41:37.851697    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-4ee1883c-kserve-7d8d6dc547-75qkw" podStartSLOduration=1.8516805459999999 podStartE2EDuration="1.851680546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:41:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:41:37.849851753 +0000 UTC m=+2354.640774265" watchObservedRunningTime="2026-09-15 12:41:37.851680546 +0000 UTC m=+2354.642603057"
Sep 15 12:41:39.767779 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:41:39.767739    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5gkxs"]
Sep 15 12:41:47.329491 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:41:47.329441    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-4ee1883c-kserve-7d8d6dc547-75qkw"
Sep 15 12:41:47.329908 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:41:47.329504    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-4ee1883c-kserve-7d8d6dc547-75qkw"
Sep 15 12:41:47.342009 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:41:47.341976    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-4ee1883c-kserve-7d8d6dc547-75qkw"
Sep 15 12:41:47.903327 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:41:47.903292    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-4ee1883c-kserve-7d8d6dc547-75qkw"
Sep 15 12:42:04.473149 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:42:04.473113    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-4ee1883c-kserve-575874f4bc-jdkf2"]
Sep 15 12:42:05.312308 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:42:05.312242    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-4ee1883c-kserve-7d8d6dc547-75qkw"]
Sep 15 12:42:07.763329 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:42:07.763289    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5gkxs"]
Sep 15 12:42:17.165731 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:42:17.165694    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5gkxs"]
Sep 15 12:42:19.492746 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:42:19.492702    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-4ee1883c-kserve-575874f4bc-jdkf2" podUID="704a150b-a1cf-44c8-a03e-c9d615473943" containerName="main" containerID="cri-o://81578fb0bf59bc50e3b090afbaba644ce663463f4cb0f1faccc082fc854bac19" gracePeriod=45
Sep 15 12:42:19.749356 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:42:19.749294    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-4ee1883c-kserve-575874f4bc-jdkf2"
Sep 15 12:42:19.828155 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:42:19.828123    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lmfdv\" (UniqueName: \"kubernetes.io/projected/704a150b-a1cf-44c8-a03e-c9d615473943-kube-api-access-lmfdv\") pod \"704a150b-a1cf-44c8-a03e-c9d615473943\" (UID: \"704a150b-a1cf-44c8-a03e-c9d615473943\") "
Sep 15 12:42:19.828340 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:42:19.828172    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/704a150b-a1cf-44c8-a03e-c9d615473943-dshm\") pod \"704a150b-a1cf-44c8-a03e-c9d615473943\" (UID: \"704a150b-a1cf-44c8-a03e-c9d615473943\") "
Sep 15 12:42:19.828340 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:42:19.828229    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/704a150b-a1cf-44c8-a03e-c9d615473943-model-cache\") pod \"704a150b-a1cf-44c8-a03e-c9d615473943\" (UID: \"704a150b-a1cf-44c8-a03e-c9d615473943\") "
Sep 15 12:42:19.828340 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:42:19.828254    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/704a150b-a1cf-44c8-a03e-c9d615473943-tls-certs\") pod \"704a150b-a1cf-44c8-a03e-c9d615473943\" (UID: \"704a150b-a1cf-44c8-a03e-c9d615473943\") "
Sep 15 12:42:19.828340 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:42:19.828308    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/704a150b-a1cf-44c8-a03e-c9d615473943-home\") pod \"704a150b-a1cf-44c8-a03e-c9d615473943\" (UID: \"704a150b-a1cf-44c8-a03e-c9d615473943\") "
Sep 15 12:42:19.828340 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:42:19.828334    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/704a150b-a1cf-44c8-a03e-c9d615473943-tmp-dir\") pod \"704a150b-a1cf-44c8-a03e-c9d615473943\" (UID: \"704a150b-a1cf-44c8-a03e-c9d615473943\") "
Sep 15 12:42:19.828618 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:42:19.828523    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/704a150b-a1cf-44c8-a03e-c9d615473943-model-cache" (OuterVolumeSpecName: "model-cache") pod "704a150b-a1cf-44c8-a03e-c9d615473943" (UID: "704a150b-a1cf-44c8-a03e-c9d615473943"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:42:19.828691 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:42:19.828660    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/704a150b-a1cf-44c8-a03e-c9d615473943-model-cache\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:42:19.828772 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:42:19.828743    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/704a150b-a1cf-44c8-a03e-c9d615473943-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "704a150b-a1cf-44c8-a03e-c9d615473943" (UID: "704a150b-a1cf-44c8-a03e-c9d615473943"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:42:19.828836 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:42:19.828817    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/704a150b-a1cf-44c8-a03e-c9d615473943-home" (OuterVolumeSpecName: "home") pod "704a150b-a1cf-44c8-a03e-c9d615473943" (UID: "704a150b-a1cf-44c8-a03e-c9d615473943"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:42:19.830623 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:42:19.830594    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/704a150b-a1cf-44c8-a03e-c9d615473943-dshm" (OuterVolumeSpecName: "dshm") pod "704a150b-a1cf-44c8-a03e-c9d615473943" (UID: "704a150b-a1cf-44c8-a03e-c9d615473943"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:42:19.830623 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:42:19.830609    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/704a150b-a1cf-44c8-a03e-c9d615473943-kube-api-access-lmfdv" (OuterVolumeSpecName: "kube-api-access-lmfdv") pod "704a150b-a1cf-44c8-a03e-c9d615473943" (UID: "704a150b-a1cf-44c8-a03e-c9d615473943"). InnerVolumeSpecName "kube-api-access-lmfdv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:42:19.830775 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:42:19.830643    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/704a150b-a1cf-44c8-a03e-c9d615473943-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "704a150b-a1cf-44c8-a03e-c9d615473943" (UID: "704a150b-a1cf-44c8-a03e-c9d615473943"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:42:19.929772 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:42:19.929731    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/704a150b-a1cf-44c8-a03e-c9d615473943-dshm\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:42:19.929772 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:42:19.929761    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/704a150b-a1cf-44c8-a03e-c9d615473943-tls-certs\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:42:19.929772 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:42:19.929771    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/704a150b-a1cf-44c8-a03e-c9d615473943-home\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:42:19.929772 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:42:19.929779    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/704a150b-a1cf-44c8-a03e-c9d615473943-tmp-dir\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:42:19.930047 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:42:19.929789    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmfdv\" (UniqueName: \"kubernetes.io/projected/704a150b-a1cf-44c8-a03e-c9d615473943-kube-api-access-lmfdv\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:42:20.064909 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:42:20.064822    2555 generic.go:356] "Generic (PLEG): container finished" podID="704a150b-a1cf-44c8-a03e-c9d615473943" containerID="81578fb0bf59bc50e3b090afbaba644ce663463f4cb0f1faccc082fc854bac19" exitCode=0
Sep 15 12:42:20.064909 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:42:20.064866    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-4ee1883c-kserve-575874f4bc-jdkf2"
Sep 15 12:42:20.064909 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:42:20.064892    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-4ee1883c-kserve-575874f4bc-jdkf2" event={"ID":"704a150b-a1cf-44c8-a03e-c9d615473943","Type":"ContainerDied","Data":"81578fb0bf59bc50e3b090afbaba644ce663463f4cb0f1faccc082fc854bac19"}
Sep 15 12:42:20.065141 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:42:20.064923    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-4ee1883c-kserve-575874f4bc-jdkf2" event={"ID":"704a150b-a1cf-44c8-a03e-c9d615473943","Type":"ContainerDied","Data":"0fec5a1a053f7fad66be614206de4258c24c4cb37190d6a5c194d5b1791611fc"}
Sep 15 12:42:20.065141 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:42:20.064939    2555 scope.go:117] "RemoveContainer" containerID="81578fb0bf59bc50e3b090afbaba644ce663463f4cb0f1faccc082fc854bac19"
Sep 15 12:42:20.074224 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:42:20.074199    2555 scope.go:117] "RemoveContainer" containerID="81578fb0bf59bc50e3b090afbaba644ce663463f4cb0f1faccc082fc854bac19"
Sep 15 12:42:20.074557 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:42:20.074539    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81578fb0bf59bc50e3b090afbaba644ce663463f4cb0f1faccc082fc854bac19\": container with ID starting with 81578fb0bf59bc50e3b090afbaba644ce663463f4cb0f1faccc082fc854bac19 not found: ID does not exist" containerID="81578fb0bf59bc50e3b090afbaba644ce663463f4cb0f1faccc082fc854bac19"
Sep 15 12:42:20.074618 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:42:20.074565    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81578fb0bf59bc50e3b090afbaba644ce663463f4cb0f1faccc082fc854bac19"} err="failed to get container status \"81578fb0bf59bc50e3b090afbaba644ce663463f4cb0f1faccc082fc854bac19\": rpc error: code = NotFound desc = could not find container \"81578fb0bf59bc50e3b090afbaba644ce663463f4cb0f1faccc082fc854bac19\": container with ID starting with 81578fb0bf59bc50e3b090afbaba644ce663463f4cb0f1faccc082fc854bac19 not found: ID does not exist"
Sep 15 12:42:20.086501 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:42:20.086476    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-4ee1883c-kserve-575874f4bc-jdkf2"]
Sep 15 12:42:20.090141 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:42:20.090120    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-4ee1883c-kserve-575874f4bc-jdkf2"]
Sep 15 12:42:20.331765 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:42:20.331671    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-4ee1883c-kserve-7d8d6dc547-75qkw" podUID="37024d56-dad9-4860-b697-100d375a4318" containerName="main" containerID="cri-o://0ea3f2d97bc59eb369fea4b623fb8331d6d594c09df2b10afe0ace129e503a6a" gracePeriod=45
Sep 15 12:42:20.594592 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:42:20.594523    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-4ee1883c-kserve-7d8d6dc547-75qkw"
Sep 15 12:42:20.636461 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:42:20.636427    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37024d56-dad9-4860-b697-100d375a4318-tmp-dir\") pod \"37024d56-dad9-4860-b697-100d375a4318\" (UID: \"37024d56-dad9-4860-b697-100d375a4318\") "
Sep 15 12:42:20.636666 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:42:20.636498    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37024d56-dad9-4860-b697-100d375a4318-tls-certs\") pod \"37024d56-dad9-4860-b697-100d375a4318\" (UID: \"37024d56-dad9-4860-b697-100d375a4318\") "
Sep 15 12:42:20.636666 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:42:20.636549    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37024d56-dad9-4860-b697-100d375a4318-dshm\") pod \"37024d56-dad9-4860-b697-100d375a4318\" (UID: \"37024d56-dad9-4860-b697-100d375a4318\") "
Sep 15 12:42:20.636666 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:42:20.636572    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37024d56-dad9-4860-b697-100d375a4318-home\") pod \"37024d56-dad9-4860-b697-100d375a4318\" (UID: \"37024d56-dad9-4860-b697-100d375a4318\") "
Sep 15 12:42:20.636666 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:42:20.636601    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37024d56-dad9-4860-b697-100d375a4318-model-cache\") pod \"37024d56-dad9-4860-b697-100d375a4318\" (UID: \"37024d56-dad9-4860-b697-100d375a4318\") "
Sep 15 12:42:20.636666 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:42:20.636632    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-44bdj\" (UniqueName: \"kubernetes.io/projected/37024d56-dad9-4860-b697-100d375a4318-kube-api-access-44bdj\") pod \"37024d56-dad9-4860-b697-100d375a4318\" (UID: \"37024d56-dad9-4860-b697-100d375a4318\") "
Sep 15 12:42:20.636967 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:42:20.636726    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37024d56-dad9-4860-b697-100d375a4318-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "37024d56-dad9-4860-b697-100d375a4318" (UID: "37024d56-dad9-4860-b697-100d375a4318"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:42:20.636967 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:42:20.636849    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37024d56-dad9-4860-b697-100d375a4318-home" (OuterVolumeSpecName: "home") pod "37024d56-dad9-4860-b697-100d375a4318" (UID: "37024d56-dad9-4860-b697-100d375a4318"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:42:20.636967 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:42:20.636884    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37024d56-dad9-4860-b697-100d375a4318-model-cache" (OuterVolumeSpecName: "model-cache") pod "37024d56-dad9-4860-b697-100d375a4318" (UID: "37024d56-dad9-4860-b697-100d375a4318"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:42:20.636967 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:42:20.636935    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37024d56-dad9-4860-b697-100d375a4318-home\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:42:20.636967 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:42:20.636948    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37024d56-dad9-4860-b697-100d375a4318-tmp-dir\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:42:20.638686 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:42:20.638656    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37024d56-dad9-4860-b697-100d375a4318-dshm" (OuterVolumeSpecName: "dshm") pod "37024d56-dad9-4860-b697-100d375a4318" (UID: "37024d56-dad9-4860-b697-100d375a4318"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:42:20.638686 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:42:20.638660    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37024d56-dad9-4860-b697-100d375a4318-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "37024d56-dad9-4860-b697-100d375a4318" (UID: "37024d56-dad9-4860-b697-100d375a4318"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:42:20.638833 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:42:20.638723    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37024d56-dad9-4860-b697-100d375a4318-kube-api-access-44bdj" (OuterVolumeSpecName: "kube-api-access-44bdj") pod "37024d56-dad9-4860-b697-100d375a4318" (UID: "37024d56-dad9-4860-b697-100d375a4318"). InnerVolumeSpecName "kube-api-access-44bdj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:42:20.737630 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:42:20.737579    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37024d56-dad9-4860-b697-100d375a4318-tls-certs\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:42:20.737630 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:42:20.737622    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37024d56-dad9-4860-b697-100d375a4318-dshm\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:42:20.737630 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:42:20.737631    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37024d56-dad9-4860-b697-100d375a4318-model-cache\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:42:20.737630 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:42:20.737641    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44bdj\" (UniqueName: \"kubernetes.io/projected/37024d56-dad9-4860-b697-100d375a4318-kube-api-access-44bdj\") on node \"ip-10-0-140-17.ec2.internal\" DevicePath \"\""
Sep 15 12:42:21.072829 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:42:21.072792    2555 generic.go:356] "Generic (PLEG): container finished" podID="37024d56-dad9-4860-b697-100d375a4318" containerID="0ea3f2d97bc59eb369fea4b623fb8331d6d594c09df2b10afe0ace129e503a6a" exitCode=0
Sep 15 12:42:21.073030 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:42:21.072835    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-4ee1883c-kserve-7d8d6dc547-75qkw"
Sep 15 12:42:21.073030 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:42:21.072873    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-4ee1883c-kserve-7d8d6dc547-75qkw" event={"ID":"37024d56-dad9-4860-b697-100d375a4318","Type":"ContainerDied","Data":"0ea3f2d97bc59eb369fea4b623fb8331d6d594c09df2b10afe0ace129e503a6a"}
Sep 15 12:42:21.073030 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:42:21.072913    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-4ee1883c-kserve-7d8d6dc547-75qkw" event={"ID":"37024d56-dad9-4860-b697-100d375a4318","Type":"ContainerDied","Data":"cbecc1956d2b29d7914195be607282f2260d64aa1e01327bb1cf94c26fb59888"}
Sep 15 12:42:21.073030 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:42:21.072929    2555 scope.go:117] "RemoveContainer" containerID="0ea3f2d97bc59eb369fea4b623fb8331d6d594c09df2b10afe0ace129e503a6a"
Sep 15 12:42:21.082222 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:42:21.082204    2555 scope.go:117] "RemoveContainer" containerID="0ea3f2d97bc59eb369fea4b623fb8331d6d594c09df2b10afe0ace129e503a6a"
Sep 15 12:42:21.082543 ip-10-0-140-17 kubenswrapper[2555]: E0915 12:42:21.082525    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ea3f2d97bc59eb369fea4b623fb8331d6d594c09df2b10afe0ace129e503a6a\": container with ID starting with 0ea3f2d97bc59eb369fea4b623fb8331d6d594c09df2b10afe0ace129e503a6a not found: ID does not exist" containerID="0ea3f2d97bc59eb369fea4b623fb8331d6d594c09df2b10afe0ace129e503a6a"
Sep 15 12:42:21.082607 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:42:21.082551    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ea3f2d97bc59eb369fea4b623fb8331d6d594c09df2b10afe0ace129e503a6a"} err="failed to get container status \"0ea3f2d97bc59eb369fea4b623fb8331d6d594c09df2b10afe0ace129e503a6a\": rpc error: code = NotFound desc = could not find container \"0ea3f2d97bc59eb369fea4b623fb8331d6d594c09df2b10afe0ace129e503a6a\": container with ID starting with 0ea3f2d97bc59eb369fea4b623fb8331d6d594c09df2b10afe0ace129e503a6a not found: ID does not exist"
Sep 15 12:42:21.094340 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:42:21.094313    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-4ee1883c-kserve-7d8d6dc547-75qkw"]
Sep 15 12:42:21.097876 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:42:21.097853    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-4ee1883c-kserve-7d8d6dc547-75qkw"]
Sep 15 12:42:21.817959 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:42:21.817917    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37024d56-dad9-4860-b697-100d375a4318" path="/var/lib/kubelet/pods/37024d56-dad9-4860-b697-100d375a4318/volumes"
Sep 15 12:42:21.818536 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:42:21.818515    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="704a150b-a1cf-44c8-a03e-c9d615473943" path="/var/lib/kubelet/pods/704a150b-a1cf-44c8-a03e-c9d615473943/volumes"
Sep 15 12:42:25.159554 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:42:25.159520    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5gkxs"]
Sep 15 12:42:33.262076 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:42:33.262035    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5gkxs"]
Sep 15 12:42:40.459395 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:42:40.459357    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5gkxs"]
Sep 15 12:42:48.571513 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:42:48.571423    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5gkxs"]
Sep 15 12:42:56.070154 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:42:56.070114    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5gkxs"]
Sep 15 12:42:59.642315 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:42:59.642279    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-55d8c6c4b9-6bw79_4a0ae0e8-2ce9-4a44-a6d3-5225f6b920f4/authorino/0.log"
Sep 15 12:43:05.135380 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:43:05.135342    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-5x5dz_fc231159-17e6-4216-93c4-ffb080ec55d8/manager/0.log"
Sep 15 12:43:05.852298 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:43:05.852248    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7b747b9585-zwmdf_b04cd056-f3ff-4332-bc4c-bd80688da3d2/copy-manifests/0.log"
Sep 15 12:43:05.859102 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:43:05.859073    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7b747b9585-zwmdf_b04cd056-f3ff-4332-bc4c-bd80688da3d2/manager/0.log"
Sep 15 12:43:07.383462 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:43:07.383430    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-55d8c6c4b9-6bw79_4a0ae0e8-2ce9-4a44-a6d3-5225f6b920f4/authorino/0.log"
Sep 15 12:43:07.715997 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:43:07.715919    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-r7tk9_88a41244-5e05-46f1-965c-c4bcd1f9ae1b/kuadrant-console-plugin/0.log"
Sep 15 12:43:07.956864 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:43:07.956832    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-5gkxs_c6b055e4-6a78-4dfa-88de-0aba5339f311/limitador/0.log"
Sep 15 12:43:08.652779 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:43:08.652746    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7cf78458b7-j5nq4_48f1beca-892d-4af7-a297-f5c524d5344f/kube-auth-proxy/0.log"
Sep 15 12:43:09.796366 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:43:09.796331    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-p7blm_2409c9c4-0f91-41f5-b598-d9f342a626be/main/0.log"
Sep 15 12:43:10.132131 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:43:10.132044    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-wv5fb_6fb1d655-76fb-4a3c-98b0-3ea71075915d/main/0.log"
Sep 15 12:43:10.244772 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:43:10.244737    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-n7tsf_ab42393b-e5a0-466c-8019-068e769519c0/main/0.log"
Sep 15 12:43:18.668836 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:43:18.668799    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-g25rd_07f9ddc4-14d7-4b04-9797-67507d4063bc/global-pull-secret-syncer/0.log"
Sep 15 12:43:18.758282 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:43:18.758230    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-c47md_5be458c0-7237-4003-b4b9-1218ac635869/konnectivity-agent/0.log"
Sep 15 12:43:18.842795 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:43:18.842764    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-17.ec2.internal_99788892a4247a0a40d9fb25d8f545f5/haproxy/0.log"
Sep 15 12:43:24.003157 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:43:24.003128    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-55d8c6c4b9-6bw79_4a0ae0e8-2ce9-4a44-a6d3-5225f6b920f4/authorino/0.log"
Sep 15 12:43:24.107627 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:43:24.107597    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-r7tk9_88a41244-5e05-46f1-965c-c4bcd1f9ae1b/kuadrant-console-plugin/0.log"
Sep 15 12:43:24.187223 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:43:24.187196    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-5gkxs_c6b055e4-6a78-4dfa-88de-0aba5339f311/limitador/0.log"
Sep 15 12:43:25.826654 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:43:25.826613    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-zm5bs_696d839f-64d5-4025-bf41-74dbacde6019/kube-state-metrics/0.log"
Sep 15 12:43:25.845565 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:43:25.845534    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-zm5bs_696d839f-64d5-4025-bf41-74dbacde6019/kube-rbac-proxy-main/0.log"
Sep 15 12:43:25.864553 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:43:25.864523    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-zm5bs_696d839f-64d5-4025-bf41-74dbacde6019/kube-rbac-proxy-self/0.log"
Sep 15 12:43:25.924419 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:43:25.924391    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-6st6v_b1f7a86b-48a2-4ac0-b57d-2b386a559c89/monitoring-plugin/0.log"
Sep 15 12:43:26.170529 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:43:26.170504    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-m7kmx_dcbfda12-861a-403f-b2c9-4aa9c1351251/node-exporter/0.log"
Sep 15 12:43:26.190707 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:43:26.190683    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-m7kmx_dcbfda12-861a-403f-b2c9-4aa9c1351251/kube-rbac-proxy/0.log"
Sep 15 12:43:26.219038 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:43:26.219012    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-m7kmx_dcbfda12-861a-403f-b2c9-4aa9c1351251/init-textfile/0.log"
Sep 15 12:43:26.615583 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:43:26.615499    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7998877c76-lzgxk_44a9b0fc-aeb3-403e-94ab-9d09e3fa44fc/telemeter-client/0.log"
Sep 15 12:43:26.634952 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:43:26.634926    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7998877c76-lzgxk_44a9b0fc-aeb3-403e-94ab-9d09e3fa44fc/reload/0.log"
Sep 15 12:43:26.657884 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:43:26.657856    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7998877c76-lzgxk_44a9b0fc-aeb3-403e-94ab-9d09e3fa44fc/kube-rbac-proxy/0.log"
Sep 15 12:43:26.686566 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:43:26.686536    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57f7f8b569-5nvq9_fac096e8-e179-4666-b6db-3fbf219ae265/thanos-query/0.log"
Sep 15 12:43:26.706449 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:43:26.706420    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57f7f8b569-5nvq9_fac096e8-e179-4666-b6db-3fbf219ae265/kube-rbac-proxy-web/0.log"
Sep 15 12:43:26.728856 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:43:26.728828    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57f7f8b569-5nvq9_fac096e8-e179-4666-b6db-3fbf219ae265/kube-rbac-proxy/0.log"
Sep 15 12:43:26.752719 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:43:26.752682    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57f7f8b569-5nvq9_fac096e8-e179-4666-b6db-3fbf219ae265/prom-label-proxy/0.log"
Sep 15 12:43:26.773616 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:43:26.773590    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57f7f8b569-5nvq9_fac096e8-e179-4666-b6db-3fbf219ae265/kube-rbac-proxy-rules/0.log"
Sep 15 12:43:26.798162 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:43:26.798132    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57f7f8b569-5nvq9_fac096e8-e179-4666-b6db-3fbf219ae265/kube-rbac-proxy-metrics/0.log"
Sep 15 12:43:29.012180 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:43:29.012146    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-n4f4n/perf-node-gather-daemonset-65775"]
Sep 15 12:43:29.012592 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:43:29.012507    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37024d56-dad9-4860-b697-100d375a4318" containerName="main"
Sep 15 12:43:29.012592 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:43:29.012520    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="37024d56-dad9-4860-b697-100d375a4318" containerName="main"
Sep 15 12:43:29.012592 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:43:29.012543    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="704a150b-a1cf-44c8-a03e-c9d615473943" containerName="main"
Sep 15 12:43:29.012592 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:43:29.012550    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="704a150b-a1cf-44c8-a03e-c9d615473943" containerName="main"
Sep 15 12:43:29.012833 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:43:29.012617    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="37024d56-dad9-4860-b697-100d375a4318" containerName="main"
Sep 15 12:43:29.012833 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:43:29.012627    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="704a150b-a1cf-44c8-a03e-c9d615473943" containerName="main"
Sep 15 12:43:29.015702 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:43:29.015682    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-n4f4n/perf-node-gather-daemonset-65775"
Sep 15 12:43:29.018169 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:43:29.018150    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-n4f4n\"/\"kube-root-ca.crt\""
Sep 15 12:43:29.018299 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:43:29.018150    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-n4f4n\"/\"default-dockercfg-27rxc\""
Sep 15 12:43:29.018299 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:43:29.018150    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-n4f4n\"/\"openshift-service-ca.crt\""
Sep 15 12:43:29.023661 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:43:29.023635    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-n4f4n/perf-node-gather-daemonset-65775"]
Sep 15 12:43:29.149224 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:43:29.149187    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f2778ce2-f113-4a05-ada4-57e1d95f9bdf-sys\") pod \"perf-node-gather-daemonset-65775\" (UID: \"f2778ce2-f113-4a05-ada4-57e1d95f9bdf\") " pod="openshift-must-gather-n4f4n/perf-node-gather-daemonset-65775"
Sep 15 12:43:29.149478 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:43:29.149291    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csgbt\" (UniqueName: \"kubernetes.io/projected/f2778ce2-f113-4a05-ada4-57e1d95f9bdf-kube-api-access-csgbt\") pod \"perf-node-gather-daemonset-65775\" (UID: \"f2778ce2-f113-4a05-ada4-57e1d95f9bdf\") " pod="openshift-must-gather-n4f4n/perf-node-gather-daemonset-65775"
Sep 15 12:43:29.149478 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:43:29.149323    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f2778ce2-f113-4a05-ada4-57e1d95f9bdf-podres\") pod \"perf-node-gather-daemonset-65775\" (UID: \"f2778ce2-f113-4a05-ada4-57e1d95f9bdf\") " pod="openshift-must-gather-n4f4n/perf-node-gather-daemonset-65775"
Sep 15 12:43:29.149478 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:43:29.149359    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f2778ce2-f113-4a05-ada4-57e1d95f9bdf-lib-modules\") pod \"perf-node-gather-daemonset-65775\" (UID: \"f2778ce2-f113-4a05-ada4-57e1d95f9bdf\") " pod="openshift-must-gather-n4f4n/perf-node-gather-daemonset-65775"
Sep 15 12:43:29.149478 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:43:29.149424    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f2778ce2-f113-4a05-ada4-57e1d95f9bdf-proc\") pod \"perf-node-gather-daemonset-65775\" (UID: \"f2778ce2-f113-4a05-ada4-57e1d95f9bdf\") " pod="openshift-must-gather-n4f4n/perf-node-gather-daemonset-65775"
Sep 15 12:43:29.250889 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:43:29.250847    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f2778ce2-f113-4a05-ada4-57e1d95f9bdf-lib-modules\") pod \"perf-node-gather-daemonset-65775\" (UID: \"f2778ce2-f113-4a05-ada4-57e1d95f9bdf\") " pod="openshift-must-gather-n4f4n/perf-node-gather-daemonset-65775"
Sep 15 12:43:29.251095 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:43:29.250912    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f2778ce2-f113-4a05-ada4-57e1d95f9bdf-proc\") pod \"perf-node-gather-daemonset-65775\" (UID: \"f2778ce2-f113-4a05-ada4-57e1d95f9bdf\") " pod="openshift-must-gather-n4f4n/perf-node-gather-daemonset-65775"
Sep 15 12:43:29.251095 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:43:29.250959    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f2778ce2-f113-4a05-ada4-57e1d95f9bdf-sys\") pod \"perf-node-gather-daemonset-65775\" (UID: \"f2778ce2-f113-4a05-ada4-57e1d95f9bdf\") " pod="openshift-must-gather-n4f4n/perf-node-gather-daemonset-65775"
Sep 15 12:43:29.251095 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:43:29.251016    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csgbt\" (UniqueName: \"kubernetes.io/projected/f2778ce2-f113-4a05-ada4-57e1d95f9bdf-kube-api-access-csgbt\") pod \"perf-node-gather-daemonset-65775\" (UID: \"f2778ce2-f113-4a05-ada4-57e1d95f9bdf\") " pod="openshift-must-gather-n4f4n/perf-node-gather-daemonset-65775"
Sep 15 12:43:29.251095 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:43:29.251035    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f2778ce2-f113-4a05-ada4-57e1d95f9bdf-lib-modules\") pod \"perf-node-gather-daemonset-65775\" (UID: \"f2778ce2-f113-4a05-ada4-57e1d95f9bdf\") " pod="openshift-must-gather-n4f4n/perf-node-gather-daemonset-65775"
Sep 15 12:43:29.251095 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:43:29.251034    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f2778ce2-f113-4a05-ada4-57e1d95f9bdf-proc\") pod \"perf-node-gather-daemonset-65775\" (UID: \"f2778ce2-f113-4a05-ada4-57e1d95f9bdf\") " pod="openshift-must-gather-n4f4n/perf-node-gather-daemonset-65775"
Sep 15 12:43:29.251095 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:43:29.251065    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f2778ce2-f113-4a05-ada4-57e1d95f9bdf-sys\") pod \"perf-node-gather-daemonset-65775\" (UID: \"f2778ce2-f113-4a05-ada4-57e1d95f9bdf\") " pod="openshift-must-gather-n4f4n/perf-node-gather-daemonset-65775"
Sep 15 12:43:29.251095 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:43:29.251044    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f2778ce2-f113-4a05-ada4-57e1d95f9bdf-podres\") pod \"perf-node-gather-daemonset-65775\" (UID: \"f2778ce2-f113-4a05-ada4-57e1d95f9bdf\") " pod="openshift-must-gather-n4f4n/perf-node-gather-daemonset-65775"
Sep 15 12:43:29.251428 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:43:29.251142    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f2778ce2-f113-4a05-ada4-57e1d95f9bdf-podres\") pod \"perf-node-gather-daemonset-65775\" (UID: \"f2778ce2-f113-4a05-ada4-57e1d95f9bdf\") " pod="openshift-must-gather-n4f4n/perf-node-gather-daemonset-65775"
Sep 15 12:43:29.260097 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:43:29.260073    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-csgbt\" (UniqueName: \"kubernetes.io/projected/f2778ce2-f113-4a05-ada4-57e1d95f9bdf-kube-api-access-csgbt\") pod \"perf-node-gather-daemonset-65775\" (UID: \"f2778ce2-f113-4a05-ada4-57e1d95f9bdf\") " pod="openshift-must-gather-n4f4n/perf-node-gather-daemonset-65775"
Sep 15 12:43:29.326339 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:43:29.326230    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-n4f4n/perf-node-gather-daemonset-65775"
Sep 15 12:43:29.605069 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:43:29.605042    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-n4f4n/perf-node-gather-daemonset-65775"]
Sep 15 12:43:29.607208 ip-10-0-140-17 kubenswrapper[2555]: W0915 12:43:29.607179    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf2778ce2_f113_4a05_ada4_57e1d95f9bdf.slice/crio-5b299260709b6dc7a247f680261e6a3c972bda4c7e5f6a28267e863d205ea91d WatchSource:0}: Error finding container 5b299260709b6dc7a247f680261e6a3c972bda4c7e5f6a28267e863d205ea91d: Status 404 returned error can't find the container with id 5b299260709b6dc7a247f680261e6a3c972bda4c7e5f6a28267e863d205ea91d
Sep 15 12:43:30.429124 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:43:30.429080    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-n4f4n/perf-node-gather-daemonset-65775" event={"ID":"f2778ce2-f113-4a05-ada4-57e1d95f9bdf","Type":"ContainerStarted","Data":"1a3d3dfca8c087a2f55bad3b4da2c28c1f915d6cc52878343cad09c0906c8840"}
Sep 15 12:43:30.429571 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:43:30.429147    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-n4f4n/perf-node-gather-daemonset-65775" event={"ID":"f2778ce2-f113-4a05-ada4-57e1d95f9bdf","Type":"ContainerStarted","Data":"5b299260709b6dc7a247f680261e6a3c972bda4c7e5f6a28267e863d205ea91d"}
Sep 15 12:43:30.429571 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:43:30.429174    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-n4f4n/perf-node-gather-daemonset-65775"
Sep 15 12:43:30.445474 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:43:30.445424    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-n4f4n/perf-node-gather-daemonset-65775" podStartSLOduration=2.445407296 podStartE2EDuration="2.445407296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 12:43:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:43:30.444516597 +0000 UTC m=+2467.235439132" watchObservedRunningTime="2026-09-15 12:43:30.445407296 +0000 UTC m=+2467.236329825"
Sep 15 12:43:31.131929 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:43:31.131901    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2vzvd_bc3c99ae-7814-40fe-8d0f-d31159e7ac76/dns/0.log"
Sep 15 12:43:31.152101 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:43:31.152077    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2vzvd_bc3c99ae-7814-40fe-8d0f-d31159e7ac76/kube-rbac-proxy/0.log"
Sep 15 12:43:31.285221 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:43:31.285194    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-kgjkj_365587ef-e477-48b4-aa6d-4e284b0b5951/dns-node-resolver/0.log"
Sep 15 12:43:31.834209 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:43:31.834180    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-8678457cc7-t287r_62c63012-fb07-478e-b95e-01a2d6cbd475/registry/0.log"
Sep 15 12:43:31.897202 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:43:31.897177    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mkck2_ee282972-ad6d-4a25-a3f5-218dfe607d6a/node-ca/0.log"
Sep 15 12:43:33.655396 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:43:33.655365    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7cf78458b7-j5nq4_48f1beca-892d-4af7-a297-f5c524d5344f/kube-auth-proxy/0.log"
Sep 15 12:43:34.705672 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:43:34.705639    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-tx76t_e21c5438-c5ca-4375-81e8-db67f537528e/serve-healthcheck-canary/0.log"
Sep 15 12:43:35.230012 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:43:35.229985    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-298j7_b6d14a02-3f21-44df-9db8-0f057f5eb567/kube-rbac-proxy/0.log"
Sep 15 12:43:35.268288 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:43:35.268241    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-298j7_b6d14a02-3f21-44df-9db8-0f057f5eb567/exporter/0.log"
Sep 15 12:43:35.289018 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:43:35.288990    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-298j7_b6d14a02-3f21-44df-9db8-0f057f5eb567/extractor/0.log"
Sep 15 12:43:36.444365 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:43:36.444338    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-n4f4n/perf-node-gather-daemonset-65775"
Sep 15 12:43:37.828803 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:43:37.828774    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-5x5dz_fc231159-17e6-4216-93c4-ffb080ec55d8/manager/0.log"
Sep 15 12:43:38.284004 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:43:38.283976    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7b747b9585-zwmdf_b04cd056-f3ff-4332-bc4c-bd80688da3d2/manager/0.log"
Sep 15 12:43:38.306030 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:43:38.305950    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7b747b9585-zwmdf_b04cd056-f3ff-4332-bc4c-bd80688da3d2/copy-manifests/0.log"
Sep 15 12:43:39.691595 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:43:39.691564    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-86c794cf5f-8k59f_baad4933-bf2c-4c6e-a373-40ae40aa415c/manager/0.log"
Sep 15 12:43:40.268685 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:43:40.268593    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7cf78458b7-j5nq4_48f1beca-892d-4af7-a297-f5c524d5344f/kube-auth-proxy/0.log"
Sep 15 12:43:43.041604 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:43:43.041575    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7cf78458b7-j5nq4_48f1beca-892d-4af7-a297-f5c524d5344f/kube-auth-proxy/0.log"
Sep 15 12:43:45.990325 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:43:45.990284    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-2wkff_e246fd0a-f899-465f-aaeb-661d2a429d06/kube-multus/0.log"
Sep 15 12:43:46.302467 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:43:46.302389    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-p4rvp_b9b05b4f-57d5-454e-b739-2778ddcab5fb/kube-multus-additional-cni-plugins/0.log"
Sep 15 12:43:46.322550 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:43:46.322527    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-p4rvp_b9b05b4f-57d5-454e-b739-2778ddcab5fb/egress-router-binary-copy/0.log"
Sep 15 12:43:46.343101 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:43:46.343070    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-p4rvp_b9b05b4f-57d5-454e-b739-2778ddcab5fb/cni-plugins/0.log"
Sep 15 12:43:46.363040 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:43:46.363011    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-p4rvp_b9b05b4f-57d5-454e-b739-2778ddcab5fb/bond-cni-plugin/0.log"
Sep 15 12:43:46.383119 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:43:46.383089    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-p4rvp_b9b05b4f-57d5-454e-b739-2778ddcab5fb/routeoverride-cni/0.log"
Sep 15 12:43:46.405653 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:43:46.405601    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-p4rvp_b9b05b4f-57d5-454e-b739-2778ddcab5fb/whereabouts-cni-bincopy/0.log"
Sep 15 12:43:46.424561 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:43:46.424526    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-p4rvp_b9b05b4f-57d5-454e-b739-2778ddcab5fb/whereabouts-cni/0.log"
Sep 15 12:43:46.561091 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:43:46.561013    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-bdjwd_f9c68d51-9107-4cd4-a57d-6f76c0762106/network-metrics-daemon/0.log"
Sep 15 12:43:46.577028 ip-10-0-140-17 kubenswrapper[2555]: I0915 12:43:46.576994    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-bdjwd_f9c68d51-9107-4cd4-a57d-6f76c0762106/kube-rbac-proxy/0.log"