Sep 08 01:56:59.740442 ip-10-0-142-78 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 08 01:56:59.740451 ip-10-0-142-78 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 08 01:56:59.740459 ip-10-0-142-78 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 08 01:56:59.740674 ip-10-0-142-78 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 08 01:57:09.832834 ip-10-0-142-78 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 08 01:57:09.832850 ip-10-0-142-78 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 814092d648df4096b914a4ce269d89c0 --
Sep 08 01:58:48.364433 ip-10-0-142-78 systemd[1]: Starting Kubernetes Kubelet...
Sep 08 01:58:48.747899 ip-10-0-142-78 kubenswrapper[2547]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 08 01:58:48.747899 ip-10-0-142-78 kubenswrapper[2547]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 08 01:58:48.747899 ip-10-0-142-78 kubenswrapper[2547]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 08 01:58:48.747899 ip-10-0-142-78 kubenswrapper[2547]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 08 01:58:48.747899 ip-10-0-142-78 kubenswrapper[2547]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 08 01:58:48.749164 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.749046    2547 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 08 01:58:48.755217 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.755195    2547 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 01:58:48.755217 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.755214    2547 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 01:58:48.755217 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.755218    2547 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 01:58:48.755217 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.755221    2547 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 01:58:48.755217 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.755223    2547 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 01:58:48.755217 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.755226    2547 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 01:58:48.755446 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.755229    2547 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 01:58:48.755446 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.755235    2547 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 01:58:48.755446 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.755239    2547 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 01:58:48.755446 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.755242    2547 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 01:58:48.755446 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.755244    2547 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 01:58:48.755446 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.755247    2547 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 01:58:48.755446 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.755250    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 01:58:48.755446 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.755253    2547 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 01:58:48.755446 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.755256    2547 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 01:58:48.755446 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.755258    2547 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 01:58:48.755446 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.755261    2547 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 01:58:48.755446 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.755264    2547 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 01:58:48.755446 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.755266    2547 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 01:58:48.755446 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.755269    2547 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 01:58:48.755446 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.755271    2547 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 01:58:48.755446 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.755274    2547 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 01:58:48.755446 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.755277    2547 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 01:58:48.755446 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.755280    2547 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 01:58:48.755446 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.755287    2547 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 01:58:48.755906 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.755291    2547 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 01:58:48.755906 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.755294    2547 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 01:58:48.755906 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.755297    2547 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 01:58:48.755906 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.755299    2547 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 01:58:48.755906 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.755302    2547 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 01:58:48.755906 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.755304    2547 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 01:58:48.755906 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.755307    2547 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 01:58:48.755906 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.755309    2547 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 01:58:48.755906 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.755312    2547 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 01:58:48.755906 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.755315    2547 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 01:58:48.755906 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.755318    2547 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 01:58:48.755906 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.755321    2547 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 01:58:48.755906 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.755323    2547 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 01:58:48.755906 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.755327    2547 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 01:58:48.755906 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.755331    2547 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 01:58:48.755906 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.755335    2547 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 01:58:48.755906 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.755338    2547 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 01:58:48.755906 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.755341    2547 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 01:58:48.755906 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.755345    2547 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 01:58:48.756429 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.755348    2547 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 01:58:48.756429 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.755351    2547 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 01:58:48.756429 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.755354    2547 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 01:58:48.756429 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.755356    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 01:58:48.756429 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.755359    2547 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 01:58:48.756429 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.755361    2547 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 01:58:48.756429 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.755365    2547 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 01:58:48.756429 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.755367    2547 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 01:58:48.756429 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.755370    2547 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 01:58:48.756429 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.755373    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 01:58:48.756429 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.755375    2547 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 01:58:48.756429 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.755378    2547 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 01:58:48.756429 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.755381    2547 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 01:58:48.756429 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.755383    2547 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 01:58:48.756429 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.755386    2547 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 01:58:48.756429 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.755389    2547 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 01:58:48.756429 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.755391    2547 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 01:58:48.756429 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.755394    2547 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 01:58:48.756429 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.755396    2547 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 01:58:48.756892 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.755399    2547 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 01:58:48.756892 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.755401    2547 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 01:58:48.756892 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.755404    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 01:58:48.756892 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.755406    2547 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 01:58:48.756892 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.755409    2547 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 01:58:48.756892 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.755411    2547 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 01:58:48.756892 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.755415    2547 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 01:58:48.756892 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.755417    2547 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 01:58:48.756892 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.755421    2547 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 01:58:48.756892 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.755423    2547 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 01:58:48.756892 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.755426    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 01:58:48.756892 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.755429    2547 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 01:58:48.756892 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.755431    2547 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 01:58:48.756892 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.755434    2547 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 01:58:48.756892 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.755436    2547 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 01:58:48.756892 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.755439    2547 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 01:58:48.756892 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.755442    2547 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 01:58:48.756892 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.755444    2547 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 01:58:48.756892 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.755447    2547 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 01:58:48.756892 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.755450    2547 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 01:58:48.757397 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.755453    2547 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 01:58:48.757397 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.755455    2547 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 01:58:48.757397 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.755458    2547 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 01:58:48.757397 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.755461    2547 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 01:58:48.757397 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.755465    2547 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 01:58:48.757397 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.755467    2547 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 01:58:48.757397 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.755470    2547 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 01:58:48.757397 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.755472    2547 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 01:58:48.757397 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.755475    2547 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 01:58:48.757397 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.755477    2547 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 01:58:48.757397 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.755480    2547 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 01:58:48.757397 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.755482    2547 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 01:58:48.757397 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.755485    2547 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 01:58:48.757397 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.755487    2547 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 01:58:48.757397 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.755490    2547 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 01:58:48.757397 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.755493    2547 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 01:58:48.757397 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756201    2547 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 01:58:48.757397 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756208    2547 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 01:58:48.757397 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756211    2547 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 01:58:48.757397 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756214    2547 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 01:58:48.757878 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756217    2547 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 01:58:48.757878 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756220    2547 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 01:58:48.757878 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756223    2547 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 01:58:48.757878 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756225    2547 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 01:58:48.757878 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756228    2547 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 01:58:48.757878 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756231    2547 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 01:58:48.757878 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756233    2547 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 01:58:48.757878 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756236    2547 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 01:58:48.757878 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756238    2547 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 01:58:48.757878 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756241    2547 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 01:58:48.757878 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756244    2547 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 01:58:48.757878 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756246    2547 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 01:58:48.757878 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756249    2547 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 01:58:48.757878 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756254    2547 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 01:58:48.757878 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756258    2547 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 01:58:48.757878 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756261    2547 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 01:58:48.757878 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756264    2547 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 01:58:48.757878 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756267    2547 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 01:58:48.757878 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756270    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 01:58:48.757878 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756273    2547 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 01:58:48.758398 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756276    2547 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 01:58:48.758398 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756278    2547 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 01:58:48.758398 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756281    2547 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 01:58:48.758398 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756283    2547 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 01:58:48.758398 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756286    2547 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 01:58:48.758398 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756288    2547 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 01:58:48.758398 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756291    2547 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 01:58:48.758398 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756293    2547 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 01:58:48.758398 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756296    2547 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 01:58:48.758398 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756298    2547 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 01:58:48.758398 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756301    2547 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 01:58:48.758398 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756304    2547 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 01:58:48.758398 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756307    2547 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 01:58:48.758398 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756310    2547 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 01:58:48.758398 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756313    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 01:58:48.758398 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756316    2547 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 01:58:48.758398 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756318    2547 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 01:58:48.758398 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756321    2547 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 01:58:48.758398 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756324    2547 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 01:58:48.758398 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756326    2547 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 01:58:48.758880 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756328    2547 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 01:58:48.758880 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756331    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 01:58:48.758880 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756334    2547 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 01:58:48.758880 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756336    2547 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 01:58:48.758880 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756339    2547 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 01:58:48.758880 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756341    2547 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 01:58:48.758880 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756345    2547 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 01:58:48.758880 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756347    2547 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 01:58:48.758880 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756350    2547 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 01:58:48.758880 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756352    2547 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 01:58:48.758880 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756355    2547 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 01:58:48.758880 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756357    2547 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 01:58:48.758880 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756360    2547 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 01:58:48.758880 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756362    2547 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 01:58:48.758880 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756365    2547 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 01:58:48.758880 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756367    2547 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 01:58:48.758880 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756371    2547 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 01:58:48.758880 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756375    2547 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 01:58:48.758880 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756377    2547 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 01:58:48.758880 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756380    2547 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 01:58:48.759384 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756383    2547 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 01:58:48.759384 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756386    2547 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 01:58:48.759384 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756389    2547 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 01:58:48.759384 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756392    2547 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 01:58:48.759384 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756395    2547 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 01:58:48.759384 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756397    2547 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 01:58:48.759384 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756400    2547 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 01:58:48.759384 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756403    2547 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 01:58:48.759384 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756406    2547 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 01:58:48.759384 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756408    2547 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 01:58:48.759384 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756410    2547 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 01:58:48.759384 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756413    2547 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 01:58:48.759384 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756417    2547 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 01:58:48.759384 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756419    2547 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 01:58:48.759384 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756422    2547 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 01:58:48.759384 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756424    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 01:58:48.759384 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756427    2547 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 01:58:48.759384 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756429    2547 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 01:58:48.759384 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756432    2547 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 01:58:48.759850 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756435    2547 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 01:58:48.759850 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756438    2547 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 01:58:48.759850 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756440    2547 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 01:58:48.759850 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756443    2547 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 01:58:48.759850 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756446    2547 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 01:58:48.759850 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756449    2547 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 01:58:48.759850 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756453    2547 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 01:58:48.759850 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756456    2547 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 01:58:48.759850 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756459    2547 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 01:58:48.759850 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756463    2547 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 01:58:48.759850 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756466    2547 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 01:58:48.759850 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756468    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 01:58:48.759850 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756471    2547 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 01:58:48.759850 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756473    2547 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 01:58:48.759850 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756476    2547 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 01:58:48.759850 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756478    2547 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 01:58:48.759850 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756922    2547 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 01:58:48.759850 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756928    2547 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 01:58:48.759850 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756931    2547 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 01:58:48.760340 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756934    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 01:58:48.760340 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756937    2547 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 01:58:48.760340 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756940    2547 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 01:58:48.760340 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756943    2547 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 01:58:48.760340 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756945    2547 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 01:58:48.760340 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756948    2547 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 01:58:48.760340 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756950    2547 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 01:58:48.760340 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756953    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 01:58:48.760340 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756955    2547 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 01:58:48.760340 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756959    2547 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 01:58:48.760340 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756961    2547 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 01:58:48.760340 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756964    2547 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 01:58:48.760340 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756966    2547 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 01:58:48.760340 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756969    2547 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 01:58:48.760340 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756972    2547 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 01:58:48.760340 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756974    2547 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 01:58:48.760340 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756977    2547 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 01:58:48.760340 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756982    2547 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 01:58:48.760340 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756985    2547 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 01:58:48.760340 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756988    2547 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 01:58:48.760825 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756992    2547 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 01:58:48.760825 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756995    2547 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 01:58:48.760825 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.756999    2547 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 01:58:48.760825 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.757002    2547 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 01:58:48.760825 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.757005    2547 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 01:58:48.760825 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.757009    2547 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 01:58:48.760825 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.757012    2547 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 01:58:48.760825 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.757015    2547 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 01:58:48.760825 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.757017    2547 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 01:58:48.760825 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.757020    2547 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 01:58:48.760825 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.757023    2547 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 01:58:48.760825 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.757026    2547 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 01:58:48.760825 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.757028    2547 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 01:58:48.760825 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.757031    2547 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 01:58:48.760825 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.757034    2547 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 01:58:48.760825 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.757036    2547 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 01:58:48.760825 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.757039    2547 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 01:58:48.760825 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.757041    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 01:58:48.760825 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.757044    2547 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 01:58:48.760825 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.757046    2547 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 01:58:48.761317 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.757049    2547 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 01:58:48.761317 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.757051    2547 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 01:58:48.761317 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.757054    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 01:58:48.761317 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.757057    2547 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 01:58:48.761317 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.757059    2547 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 01:58:48.761317 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.757062    2547 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 01:58:48.761317 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.757065    2547 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 01:58:48.761317 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.757067    2547 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 01:58:48.761317 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.757070    2547 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 01:58:48.761317 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.757074    2547 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 01:58:48.761317 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.757077    2547 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 01:58:48.761317 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.757080    2547 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 01:58:48.761317 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.757082    2547 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 01:58:48.761317 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.757085    2547 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 01:58:48.761317 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.757088    2547 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 01:58:48.761317 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.757096    2547 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 01:58:48.761317 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.757100    2547 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 01:58:48.761317 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.757102    2547 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 01:58:48.761317 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.757105    2547 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 01:58:48.761827 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.757108    2547 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 01:58:48.761827 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.757110    2547 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 01:58:48.761827 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.757113    2547 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 01:58:48.761827 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.757115    2547 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 01:58:48.761827 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.757118    2547 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 01:58:48.761827 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.757120    2547 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 01:58:48.761827 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.757123    2547 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 01:58:48.761827 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.757125    2547 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 01:58:48.761827 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.757128    2547 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 01:58:48.761827 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.757130    2547 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 01:58:48.761827 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.757150    2547 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 01:58:48.761827 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.757155    2547 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 01:58:48.761827 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.757158    2547 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 01:58:48.761827 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.757162    2547 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 01:58:48.761827 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.757165    2547 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 01:58:48.761827 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.757170    2547 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 01:58:48.761827 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.757173    2547 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 01:58:48.761827 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.757175    2547 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 01:58:48.761827 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.757179    2547 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 01:58:48.762312 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.757183    2547 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 01:58:48.762312 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.757189    2547 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 01:58:48.762312 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.757193    2547 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 01:58:48.762312 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.757197    2547 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 01:58:48.762312 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.757200    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 01:58:48.762312 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.757203    2547 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 01:58:48.762312 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.757206    2547 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 01:58:48.762312 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.757208    2547 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 01:58:48.762312 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.757211    2547 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 01:58:48.762312 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.757214    2547 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 01:58:48.762312 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.757217    2547 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 01:58:48.762312 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.757219    2547 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 01:58:48.762312 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.757222    2547 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 01:58:48.762312 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.757224    2547 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 01:58:48.762312 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.757227    2547 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 01:58:48.762312 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.757229    2547 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 01:58:48.762312 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.757232    2547 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 01:58:48.762312 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.757234    2547 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 01:58:48.762312 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757299    2547 flags.go:64] FLAG: --address="0.0.0.0"
Sep 08 01:58:48.762312 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757333    2547 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 08 01:58:48.762312 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757339    2547 flags.go:64] FLAG: --anonymous-auth="true"
Sep 08 01:58:48.762816 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757343    2547 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 08 01:58:48.762816 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757347    2547 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 08 01:58:48.762816 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757351    2547 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 08 01:58:48.762816 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757355    2547 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 08 01:58:48.762816 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757359    2547 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 08 01:58:48.762816 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757362    2547 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 08 01:58:48.762816 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757365    2547 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 08 01:58:48.762816 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757377    2547 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 08 01:58:48.762816 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757381    2547 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 08 01:58:48.762816 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757384    2547 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 08 01:58:48.762816 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757387    2547 flags.go:64] FLAG: --cgroup-root=""
Sep 08 01:58:48.762816 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757390    2547 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 08 01:58:48.762816 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757393    2547 flags.go:64] FLAG: --client-ca-file=""
Sep 08 01:58:48.762816 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757397    2547 flags.go:64] FLAG: --cloud-provider="external"
Sep 08 01:58:48.762816 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757402    2547 flags.go:64] FLAG: --cluster-dns="[]"
Sep 08 01:58:48.762816 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757407    2547 flags.go:64] FLAG: --cluster-domain=""
Sep 08 01:58:48.762816 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757410    2547 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 08 01:58:48.762816 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757413    2547 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 08 01:58:48.762816 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757416    2547 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 08 01:58:48.762816 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757420    2547 flags.go:64] FLAG: --container-log-max-files="5"
Sep 08 01:58:48.762816 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757424    2547 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 08 01:58:48.762816 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757427    2547 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 08 01:58:48.762816 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757430    2547 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 08 01:58:48.763430 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757434    2547 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 08 01:58:48.763430 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757436    2547 flags.go:64] FLAG: --contention-profiling="false"
Sep 08 01:58:48.763430 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757440    2547 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 08 01:58:48.763430 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757443    2547 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 08 01:58:48.763430 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757446    2547 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 08 01:58:48.763430 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757449    2547 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 08 01:58:48.763430 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757453    2547 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 08 01:58:48.763430 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757456    2547 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 08 01:58:48.763430 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757459    2547 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 08 01:58:48.763430 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757461    2547 flags.go:64] FLAG: --enable-load-reader="false"
Sep 08 01:58:48.763430 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757464    2547 flags.go:64] FLAG: --enable-server="true"
Sep 08 01:58:48.763430 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757467    2547 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 08 01:58:48.763430 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757471    2547 flags.go:64] FLAG: --event-burst="100"
Sep 08 01:58:48.763430 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757474    2547 flags.go:64] FLAG: --event-qps="50"
Sep 08 01:58:48.763430 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757477    2547 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 08 01:58:48.763430 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757480    2547 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 08 01:58:48.763430 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757484    2547 flags.go:64] FLAG: --eviction-hard=""
Sep 08 01:58:48.763430 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757492    2547 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 08 01:58:48.763430 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757496    2547 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 08 01:58:48.763430 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757498    2547 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 08 01:58:48.763430 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757501    2547 flags.go:64] FLAG: --eviction-soft=""
Sep 08 01:58:48.763430 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757504    2547 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 08 01:58:48.763430 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757507    2547 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 08 01:58:48.763430 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757511    2547 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 08 01:58:48.763430 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757514    2547 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 08 01:58:48.764065 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757517    2547 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 08 01:58:48.764065 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757520    2547 flags.go:64] FLAG: --fail-swap-on="true"
Sep 08 01:58:48.764065 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757522    2547 flags.go:64] FLAG: --feature-gates=""
Sep 08 01:58:48.764065 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757526    2547 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 08 01:58:48.764065 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757529    2547 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 08 01:58:48.764065 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757532    2547 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 08 01:58:48.764065 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757535    2547 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 08 01:58:48.764065 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757538    2547 flags.go:64] FLAG: --healthz-port="10248"
Sep 08 01:58:48.764065 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757541    2547 flags.go:64] FLAG: --help="false"
Sep 08 01:58:48.764065 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757544    2547 flags.go:64] FLAG: --hostname-override="ip-10-0-142-78.ec2.internal"
Sep 08 01:58:48.764065 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757547    2547 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 08 01:58:48.764065 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757549    2547 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 08 01:58:48.764065 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757552    2547 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 08 01:58:48.764065 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757555    2547 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 08 01:58:48.764065 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757559    2547 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 08 01:58:48.764065 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757562    2547 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 08 01:58:48.764065 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757564    2547 flags.go:64] FLAG: --image-service-endpoint=""
Sep 08 01:58:48.764065 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757567    2547 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 08 01:58:48.764065 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757570    2547 flags.go:64] FLAG: --kube-api-burst="100"
Sep 08 01:58:48.764065 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757573    2547 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 08 01:58:48.764065 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757576    2547 flags.go:64] FLAG: --kube-api-qps="50"
Sep 08 01:58:48.764065 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757578    2547 flags.go:64] FLAG: --kube-reserved=""
Sep 08 01:58:48.764065 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757581    2547 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 08 01:58:48.764065 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757585    2547 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 08 01:58:48.764678 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757588    2547 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 08 01:58:48.764678 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757591    2547 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 08 01:58:48.764678 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757594    2547 flags.go:64] FLAG: --lock-file=""
Sep 08 01:58:48.764678 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757596    2547 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 08 01:58:48.764678 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757599    2547 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 08 01:58:48.764678 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757602    2547 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 08 01:58:48.764678 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757615    2547 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 08 01:58:48.764678 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757625    2547 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 08 01:58:48.764678 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757628    2547 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 08 01:58:48.764678 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757631    2547 flags.go:64] FLAG: --logging-format="text"
Sep 08 01:58:48.764678 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757634    2547 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 08 01:58:48.764678 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757637    2547 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 08 01:58:48.764678 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757640    2547 flags.go:64] FLAG: --manifest-url=""
Sep 08 01:58:48.764678 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757643    2547 flags.go:64] FLAG: --manifest-url-header=""
Sep 08 01:58:48.764678 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757647    2547 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 08 01:58:48.764678 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757650    2547 flags.go:64] FLAG: --max-open-files="1000000"
Sep 08 01:58:48.764678 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757653    2547 flags.go:64] FLAG: --max-pods="110"
Sep 08 01:58:48.764678 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757656    2547 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 08 01:58:48.764678 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757659    2547 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 08 01:58:48.764678 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757662    2547 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 08 01:58:48.764678 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757665    2547 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 08 01:58:48.764678 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757668    2547 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 08 01:58:48.764678 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757671    2547 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 08 01:58:48.764678 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757674    2547 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 08 01:58:48.764678 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757681    2547 flags.go:64] FLAG: --node-status-max-images="50"
Sep 08 01:58:48.765390 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757684    2547 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 08 01:58:48.765390 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757687    2547 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 08 01:58:48.765390 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757690    2547 flags.go:64] FLAG: --pod-cidr=""
Sep 08 01:58:48.765390 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757693    2547 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 08 01:58:48.765390 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757699    2547 flags.go:64] FLAG: --pod-manifest-path=""
Sep 08 01:58:48.765390 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757702    2547 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 08 01:58:48.765390 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757707    2547 flags.go:64] FLAG: --pods-per-core="0"
Sep 08 01:58:48.765390 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757709    2547 flags.go:64] FLAG: --port="10250"
Sep 08 01:58:48.765390 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757712    2547 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 08 01:58:48.765390 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757715    2547 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-02db6a8d76e13d11c"
Sep 08 01:58:48.765390 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757718    2547 flags.go:64] FLAG: --qos-reserved=""
Sep 08 01:58:48.765390 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757721    2547 flags.go:64] FLAG: --read-only-port="10255"
Sep 08 01:58:48.765390 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757724    2547 flags.go:64] FLAG: --register-node="true"
Sep 08 01:58:48.765390 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757727    2547 flags.go:64] FLAG: --register-with-taints=""
Sep 08 01:58:48.765390 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757731    2547 flags.go:64] FLAG: --registry-burst="10"
Sep 08 01:58:48.765390 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757739    2547 flags.go:64] FLAG: --registry-qps="5"
Sep 08 01:58:48.765390 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757742    2547 flags.go:64] FLAG: --reserved-cpus=""
Sep 08 01:58:48.765390 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757745    2547 flags.go:64] FLAG: --reserved-memory=""
Sep 08 01:58:48.765390 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757748    2547 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 08 01:58:48.765390 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757751    2547 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 08 01:58:48.765390 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757754    2547 flags.go:64] FLAG: --rotate-certificates="false"
Sep 08 01:58:48.765390 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757757    2547 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 08 01:58:48.765390 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757759    2547 flags.go:64] FLAG: --runonce="false"
Sep 08 01:58:48.765390 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757762    2547 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 08 01:58:48.765390 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757765    2547 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 08 01:58:48.766086 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757768    2547 flags.go:64] FLAG: --seccomp-default="false"
Sep 08 01:58:48.766086 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757771    2547 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 08 01:58:48.766086 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757774    2547 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 08 01:58:48.766086 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757776    2547 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 08 01:58:48.766086 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757779    2547 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 08 01:58:48.766086 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757782    2547 flags.go:64] FLAG: --storage-driver-password="root"
Sep 08 01:58:48.766086 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757785    2547 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 08 01:58:48.766086 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757788    2547 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 08 01:58:48.766086 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757791    2547 flags.go:64] FLAG: --storage-driver-user="root"
Sep 08 01:58:48.766086 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757793    2547 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 08 01:58:48.766086 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757796    2547 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 08 01:58:48.766086 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757800    2547 flags.go:64] FLAG: --system-cgroups=""
Sep 08 01:58:48.766086 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757802    2547 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 08 01:58:48.766086 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757809    2547 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 08 01:58:48.766086 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757812    2547 flags.go:64] FLAG: --tls-cert-file=""
Sep 08 01:58:48.766086 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757815    2547 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 08 01:58:48.766086 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757822    2547 flags.go:64] FLAG: --tls-min-version=""
Sep 08 01:58:48.766086 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757824    2547 flags.go:64] FLAG: --tls-private-key-file=""
Sep 08 01:58:48.766086 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757827    2547 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 08 01:58:48.766086 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757830    2547 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 08 01:58:48.766086 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757833    2547 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 08 01:58:48.766086 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757835    2547 flags.go:64] FLAG: --v="2"
Sep 08 01:58:48.766086 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757839    2547 flags.go:64] FLAG: --version="false"
Sep 08 01:58:48.766086 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757849    2547 flags.go:64] FLAG: --vmodule=""
Sep 08 01:58:48.766086 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757853    2547 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 08 01:58:48.766813 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.757856    2547 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 08 01:58:48.766813 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.757971    2547 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 01:58:48.766813 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.757975    2547 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 01:58:48.766813 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.757978    2547 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 01:58:48.766813 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.757981    2547 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 01:58:48.766813 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.757983    2547 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 01:58:48.766813 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.757986    2547 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 01:58:48.766813 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.757989    2547 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 01:58:48.766813 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.757991    2547 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 01:58:48.766813 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.757994    2547 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 01:58:48.766813 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.757996    2547 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 01:58:48.766813 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.757999    2547 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 01:58:48.766813 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.758001    2547 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 01:58:48.766813 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.758004    2547 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 01:58:48.766813 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.758006    2547 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 01:58:48.766813 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.758009    2547 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 01:58:48.766813 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.758011    2547 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 01:58:48.766813 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.758014    2547 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 01:58:48.766813 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.758016    2547 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 01:58:48.766813 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.758019    2547 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 01:58:48.767487 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.758023    2547 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 01:58:48.767487 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.758025    2547 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 01:58:48.767487 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.758028    2547 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 01:58:48.767487 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.758030    2547 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 01:58:48.767487 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.758033    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 01:58:48.767487 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.758036    2547 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 01:58:48.767487 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.758040    2547 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 01:58:48.767487 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.758043    2547 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 01:58:48.767487 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.758045    2547 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 01:58:48.767487 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.758048    2547 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 01:58:48.767487 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.758050    2547 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 01:58:48.767487 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.758053    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 01:58:48.767487 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.758056    2547 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 01:58:48.767487 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.758058    2547 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 01:58:48.767487 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.758061    2547 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 01:58:48.767487 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.758063    2547 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 01:58:48.767487 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.758067    2547 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 01:58:48.767487 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.758071    2547 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 01:58:48.767487 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.758073    2547 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 01:58:48.767487 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.758076    2547 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 01:58:48.768036 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.758079    2547 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 01:58:48.768036 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.758083    2547 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 01:58:48.768036 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.758086    2547 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 01:58:48.768036 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.758089    2547 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 01:58:48.768036 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.758092    2547 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 01:58:48.768036 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.758094    2547 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 01:58:48.768036 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.758097    2547 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 01:58:48.768036 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.758099    2547 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 01:58:48.768036 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.758102    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 01:58:48.768036 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.758104    2547 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 01:58:48.768036 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.758107    2547 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 01:58:48.768036 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.758109    2547 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 01:58:48.768036 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.758114    2547 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 01:58:48.768036 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.758118    2547 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 01:58:48.768036 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.758120    2547 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 01:58:48.768036 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.758123    2547 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 01:58:48.768036 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.758125    2547 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 01:58:48.768036 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.758128    2547 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 01:58:48.768036 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.758146    2547 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 01:58:48.768572 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.758149    2547 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 01:58:48.768572 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.758151    2547 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 01:58:48.768572 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.758154    2547 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 01:58:48.768572 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.758157    2547 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 01:58:48.768572 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.758159    2547 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 01:58:48.768572 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.758163    2547 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 01:58:48.768572 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.758165    2547 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 01:58:48.768572 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.758168    2547 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 01:58:48.768572 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.758170    2547 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 01:58:48.768572 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.758173    2547 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 01:58:48.768572 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.758176    2547 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 01:58:48.768572 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.758178    2547 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 01:58:48.768572 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.758181    2547 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 01:58:48.768572 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.758184    2547 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 01:58:48.768572 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.758186    2547 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 01:58:48.768572 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.758189    2547 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 01:58:48.768572 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.758191    2547 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 01:58:48.768572 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.758194    2547 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 01:58:48.768572 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.758197    2547 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 01:58:48.769093 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.758199    2547 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 01:58:48.769093 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.758202    2547 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 01:58:48.769093 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.758205    2547 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 01:58:48.769093 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.758208    2547 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 01:58:48.769093 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.758210    2547 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 01:58:48.769093 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.758213    2547 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 01:58:48.769093 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.758217    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 01:58:48.769093 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.758220    2547 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 01:58:48.769093 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.758223    2547 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 01:58:48.769093 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.758225    2547 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 01:58:48.769093 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.758228    2547 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 01:58:48.769093 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.758230    2547 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 01:58:48.769093 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.758234    2547 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 01:58:48.769093 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.758237    2547 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 01:58:48.769093 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.758240    2547 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 01:58:48.769093 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.758242    2547 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 01:58:48.769093 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.758251    2547 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 01:58:48.769093 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.758254    2547 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 01:58:48.769093 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.758256    2547 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 01:58:48.769093 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.758259    2547 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 01:58:48.769648 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.758262    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 01:58:48.769648 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.758265    2547 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 01:58:48.769648 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.759189    2547 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 08 01:58:48.769648 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.766811    2547 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 08 01:58:48.769648 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.766828    2547 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 08 01:58:48.769648 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.766848    2547 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 08 01:58:48.769648 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.766858    2547 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 08 01:58:48.769648 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.766934    2547 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 01:58:48.769648 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.766939    2547 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 01:58:48.769648 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.766943    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 01:58:48.769648 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.766946    2547 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 01:58:48.769648 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.766950    2547 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 01:58:48.769648 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.766953    2547 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 01:58:48.769648 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.766956    2547 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 01:58:48.770020 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.766959    2547 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 01:58:48.770020 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.766963    2547 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 01:58:48.770020 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.766966    2547 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 01:58:48.770020 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.766969    2547 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 01:58:48.770020 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.766972    2547 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 01:58:48.770020 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.766975    2547 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 01:58:48.770020 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.766977    2547 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 01:58:48.770020 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.766980    2547 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 01:58:48.770020 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.766983    2547 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 01:58:48.770020 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.766986    2547 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 01:58:48.770020 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.766988    2547 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 01:58:48.770020 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.766992    2547 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 01:58:48.770020 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.766997    2547 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 01:58:48.770020 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767000    2547 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 01:58:48.770020 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767003    2547 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 01:58:48.770020 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767006    2547 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 01:58:48.770020 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767009    2547 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 01:58:48.770020 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767013    2547 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 01:58:48.770020 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767016    2547 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 01:58:48.770499 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767019    2547 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 01:58:48.770499 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767021    2547 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 01:58:48.770499 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767025    2547 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 01:58:48.770499 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767028    2547 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 01:58:48.770499 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767031    2547 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 01:58:48.770499 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767034    2547 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 01:58:48.770499 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767037    2547 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 01:58:48.770499 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767040    2547 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 01:58:48.770499 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767043    2547 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 01:58:48.770499 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767046    2547 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 01:58:48.770499 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767048    2547 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 01:58:48.770499 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767051    2547 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 01:58:48.770499 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767053    2547 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 01:58:48.770499 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767057    2547 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 01:58:48.770499 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767060    2547 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 01:58:48.770499 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767063    2547 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 01:58:48.770499 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767066    2547 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 01:58:48.770499 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767069    2547 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 01:58:48.770499 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767071    2547 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 01:58:48.771005 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767074    2547 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 01:58:48.771005 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767076    2547 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 01:58:48.771005 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767079    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 01:58:48.771005 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767082    2547 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 01:58:48.771005 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767085    2547 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 01:58:48.771005 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767087    2547 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 01:58:48.771005 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767090    2547 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 01:58:48.771005 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767093    2547 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 01:58:48.771005 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767095    2547 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 01:58:48.771005 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767098    2547 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 01:58:48.771005 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767100    2547 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 01:58:48.771005 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767103    2547 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 01:58:48.771005 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767112    2547 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 01:58:48.771005 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767114    2547 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 01:58:48.771005 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767117    2547 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 01:58:48.771005 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767120    2547 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 01:58:48.771005 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767123    2547 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 01:58:48.771005 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767126    2547 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 01:58:48.771005 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767128    2547 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 01:58:48.771005 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767131    2547 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 01:58:48.771555 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767147    2547 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 01:58:48.771555 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767150    2547 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 01:58:48.771555 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767152    2547 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 01:58:48.771555 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767155    2547 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 01:58:48.771555 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767157    2547 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 01:58:48.771555 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767161    2547 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 01:58:48.771555 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767164    2547 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 01:58:48.771555 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767167    2547 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 01:58:48.771555 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767170    2547 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 01:58:48.771555 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767173    2547 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 01:58:48.771555 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767176    2547 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 01:58:48.771555 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767178    2547 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 01:58:48.771555 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767181    2547 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 01:58:48.771555 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767184    2547 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 01:58:48.771555 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767187    2547 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 01:58:48.771555 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767189    2547 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 01:58:48.771555 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767192    2547 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 01:58:48.771555 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767195    2547 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 01:58:48.771555 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767198    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 01:58:48.772027 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767201    2547 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 01:58:48.772027 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767203    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 01:58:48.772027 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767206    2547 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 01:58:48.772027 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767209    2547 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 01:58:48.772027 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767211    2547 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 01:58:48.772027 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767213    2547 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 01:58:48.772027 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767222    2547 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 01:58:48.772027 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767224    2547 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 01:58:48.772027 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767231    2547 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 01:58:48.772027 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767234    2547 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 01:58:48.772027 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767237    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 01:58:48.772027 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767240    2547 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 01:58:48.772027 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767242    2547 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 01:58:48.772027 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767245    2547 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 01:58:48.772027 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767247    2547 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 01:58:48.772400 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.767252    2547 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 08 01:58:48.772400 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767402    2547 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 01:58:48.772400 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767406    2547 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 01:58:48.772400 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767410    2547 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 01:58:48.772400 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767413    2547 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 01:58:48.772400 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767416    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 01:58:48.772400 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767418    2547 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 01:58:48.772400 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767421    2547 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 01:58:48.772400 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767424    2547 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 01:58:48.772400 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767426    2547 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 01:58:48.772400 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767429    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 01:58:48.772400 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767431    2547 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 01:58:48.772400 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767434    2547 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 01:58:48.772400 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767437    2547 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 01:58:48.772400 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767440    2547 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 01:58:48.772763 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767442    2547 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 01:58:48.772763 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767445    2547 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 01:58:48.772763 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767448    2547 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 01:58:48.772763 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767451    2547 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 01:58:48.772763 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767453    2547 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 01:58:48.772763 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767456    2547 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 01:58:48.772763 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767458    2547 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 01:58:48.772763 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767461    2547 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 01:58:48.772763 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767463    2547 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 01:58:48.772763 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767471    2547 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 01:58:48.772763 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767474    2547 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 01:58:48.772763 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767476    2547 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 01:58:48.772763 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767479    2547 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 01:58:48.772763 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767481    2547 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 01:58:48.772763 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767484    2547 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 01:58:48.772763 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767486    2547 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 01:58:48.772763 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767489    2547 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 01:58:48.772763 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767492    2547 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 01:58:48.772763 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767494    2547 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 01:58:48.772763 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767497    2547 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 01:58:48.773269 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767500    2547 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 01:58:48.773269 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767502    2547 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 01:58:48.773269 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767506    2547 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 01:58:48.773269 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767510    2547 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 01:58:48.773269 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767513    2547 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 01:58:48.773269 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767516    2547 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 01:58:48.773269 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767518    2547 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 01:58:48.773269 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767521    2547 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 01:58:48.773269 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767524    2547 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 01:58:48.773269 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767527    2547 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 01:58:48.773269 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767530    2547 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 01:58:48.773269 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767533    2547 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 01:58:48.773269 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767536    2547 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 01:58:48.773269 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767539    2547 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 01:58:48.773269 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767542    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 01:58:48.773269 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767545    2547 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 01:58:48.773269 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767548    2547 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 01:58:48.773269 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767551    2547 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 01:58:48.773269 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767553    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 01:58:48.773747 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767556    2547 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 01:58:48.773747 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767558    2547 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 01:58:48.773747 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767561    2547 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 01:58:48.773747 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767569    2547 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 01:58:48.773747 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767572    2547 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 01:58:48.773747 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767575    2547 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 01:58:48.773747 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767577    2547 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 01:58:48.773747 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767580    2547 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 01:58:48.773747 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767582    2547 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 01:58:48.773747 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767585    2547 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 01:58:48.773747 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767588    2547 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 01:58:48.773747 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767590    2547 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 01:58:48.773747 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767593    2547 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 01:58:48.773747 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767595    2547 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 01:58:48.773747 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767598    2547 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 01:58:48.773747 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767600    2547 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 01:58:48.773747 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767602    2547 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 01:58:48.773747 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767605    2547 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 01:58:48.773747 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767607    2547 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 01:58:48.774220 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767610    2547 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 01:58:48.774220 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767612    2547 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 01:58:48.774220 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767615    2547 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 01:58:48.774220 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767617    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 01:58:48.774220 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767627    2547 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 01:58:48.774220 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767629    2547 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 01:58:48.774220 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767632    2547 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 01:58:48.774220 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767635    2547 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 01:58:48.774220 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767638    2547 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 01:58:48.774220 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767641    2547 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 01:58:48.774220 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767643    2547 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 01:58:48.774220 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767646    2547 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 01:58:48.774220 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767648    2547 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 01:58:48.774220 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767651    2547 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 01:58:48.774220 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767654    2547 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 01:58:48.774220 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767656    2547 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 01:58:48.774220 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767659    2547 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 01:58:48.774220 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767666    2547 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 01:58:48.774220 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767669    2547 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 01:58:48.774741 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767673    2547 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 01:58:48.774741 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767676    2547 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 01:58:48.774741 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767678    2547 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 01:58:48.774741 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767681    2547 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 01:58:48.774741 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767683    2547 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 01:58:48.774741 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767686    2547 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 01:58:48.774741 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767688    2547 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 01:58:48.774741 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:48.767691    2547 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 01:58:48.774741 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.767696    2547 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 08 01:58:48.774741 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.767811    2547 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 08 01:58:48.774741 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.771336    2547 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 08 01:58:48.774741 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.772411    2547 server.go:1019] "Starting client certificate rotation"
Sep 08 01:58:48.774741 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.772511    2547 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 08 01:58:48.774741 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.772539    2547 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 08 01:58:48.775655 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.775633    2547 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 08 01:58:48.781312 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.781290    2547 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 08 01:58:48.795428 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.795405    2547 log.go:25] "Validated CRI v1 runtime API"
Sep 08 01:58:48.799532 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.799515    2547 log.go:25] "Validated CRI v1 image API"
Sep 08 01:58:48.800443 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.800425    2547 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 08 01:58:48.802114 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.802098    2547 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 08 01:58:48.804084 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.804065    2547 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 986a1d98-21d1-4152-948e-d408fa6f374e:/dev/nvme0n1p3 cca744ba-901a-46bf-9e47-b15ff0a87553:/dev/nvme0n1p4]
Sep 08 01:58:48.804154 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.804084    2547 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 08 01:58:48.809816 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.809699    2547 manager.go:216] Machine: {Timestamp:2026-09-08 01:58:48.808347075 +0000 UTC m=+0.332122147 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098699 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ccb2242c503f5851fcf4fa0d2d4dd SystemUUID:ec2ccb22-42c5-03f5-851f-cf4fa0d2d4dd BootID:814092d6-48df-4096-b914-a4ce269d89c0 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:4c:d6:68:9f:01 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:4c:d6:68:9f:01 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:e2:f0:72:64:fb:aa Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 08 01:58:48.809816 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.809808    2547 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 08 01:58:48.809928 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.809890    2547 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 08 01:58:48.813254 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.813226    2547 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 08 01:58:48.813403 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.813257    2547 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-78.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 08 01:58:48.814071 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.814062    2547 topology_manager.go:138] "Creating topology manager with none policy"
Sep 08 01:58:48.814106 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.814074    2547 container_manager_linux.go:309] "Creating device plugin manager"
Sep 08 01:58:48.814106 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.814082    2547 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 08 01:58:48.814859 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.814848    2547 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 08 01:58:48.816412 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.816402    2547 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 08 01:58:48.818036 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.818019    2547 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6td6d"
Sep 08 01:58:48.820643 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.820631    2547 state_mem.go:36] "Initialized new in-memory state store"
Sep 08 01:58:48.820760 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.820750    2547 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 08 01:58:48.820793 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.820788    2547 kubelet.go:486] "Attempting to sync node with API server"
Sep 08 01:58:48.820828 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.820804    2547 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 08 01:58:48.820828 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.820816    2547 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 08 01:58:48.820828 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.820825    2547 kubelet.go:398] "Adding apiserver pod source"
Sep 08 01:58:48.820908 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.820834    2547 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 08 01:58:48.821235 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.821223    2547 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 08 01:58:48.821273 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.821245    2547 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 08 01:58:48.824060 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.824043    2547 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 08 01:58:48.825021 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.824999    2547 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6td6d"
Sep 08 01:58:48.827231 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.827214    2547 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 08 01:58:48.827290 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.827263    2547 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 08 01:58:48.827472 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.827460    2547 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 08 01:58:48.827504 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.827480    2547 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 08 01:58:48.827504 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.827487    2547 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 08 01:58:48.827504 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.827492    2547 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 08 01:58:48.827504 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.827500    2547 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 08 01:58:48.827504 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.827505    2547 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 08 01:58:48.828101 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.828092    2547 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 08 01:58:48.828101 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.828101    2547 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 08 01:58:48.828183 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.828107    2547 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 08 01:58:48.828183 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.828112    2547 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 08 01:58:48.828183 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.828121    2547 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 08 01:58:48.828183 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.828159    2547 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 08 01:58:48.829090 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.829076    2547 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 08 01:58:48.829090 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.829087    2547 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 08 01:58:48.829474 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:58:48.829452    2547 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-78.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 08 01:58:48.829912 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:58:48.829888    2547 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 08 01:58:48.833883 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.833862    2547 server.go:1268] "Started kubelet"
Sep 08 01:58:48.833993 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.833931    2547 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 08 01:58:48.834051 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.833955    2547 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 08 01:58:48.834103 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.834050    2547 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 08 01:58:48.834807 ip-10-0-142-78 systemd[1]: Started Kubernetes Kubelet.
Sep 08 01:58:48.835915 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.835899    2547 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 08 01:58:48.836630 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.836612    2547 server.go:310] "Adding debug handlers to kubelet server"
Sep 08 01:58:48.837311 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.837293    2547 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-142-78.ec2.internal" not found
Sep 08 01:58:48.838268 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:58:48.838242    2547 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 08 01:58:48.842115 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.842097    2547 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 08 01:58:48.842819 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.842806    2547 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 08 01:58:48.842904 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:58:48.842883    2547 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-78.ec2.internal\" not found"
Sep 08 01:58:48.843651 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.843625    2547 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 08 01:58:48.843651 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.843626    2547 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 08 01:58:48.843768 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.843660    2547 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 08 01:58:48.843768 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.843730    2547 factory.go:153] Registering CRI-O factory
Sep 08 01:58:48.843768 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.843743    2547 factory.go:223] Registration of the crio container factory successfully
Sep 08 01:58:48.843768 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.843752    2547 reconstruct.go:97] "Volume reconstruction finished"
Sep 08 01:58:48.843768 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.843763    2547 reconciler.go:29] "Reconciler: start to sync state"
Sep 08 01:58:48.843960 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.843794    2547 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 08 01:58:48.843960 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.843801    2547 factory.go:55] Registering systemd factory
Sep 08 01:58:48.843960 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.843807    2547 factory.go:223] Registration of the systemd container factory successfully
Sep 08 01:58:48.843960 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.843818    2547 factory.go:103] Registering Raw factory
Sep 08 01:58:48.843960 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.843831    2547 manager.go:1196] Started watching for new ooms in manager
Sep 08 01:58:48.844280 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.844268    2547 manager.go:349] Starting recovery of all containers
Sep 08 01:58:48.845743 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.845726    2547 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 01:58:48.847947 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:58:48.847922    2547 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-142-78.ec2.internal\" not found" node="ip-10-0-142-78.ec2.internal"
Sep 08 01:58:48.851203 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.851166    2547 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 08 01:58:48.851305 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.851281    2547 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-142-78.ec2.internal" not found
Sep 08 01:58:48.854821 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.854648    2547 manager.go:354] Recovery completed
Sep 08 01:58:48.861094 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.860969    2547 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 01:58:48.862661 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.862644    2547 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-78.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 01:58:48.862708 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.862670    2547 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-78.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 01:58:48.862708 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.862680    2547 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-78.ec2.internal" event="NodeHasSufficientPID"
Sep 08 01:58:48.863490 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.863471    2547 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 08 01:58:48.863490 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.863481    2547 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 01:58:48.863490 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.863488    2547 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 08 01:58:48.863688 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.863509    2547 state_mem.go:36] "Initialized new in-memory state store"
Sep 08 01:58:48.864275 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.864261    2547 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-78.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 01:58:48.864341 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.864288    2547 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-78.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 01:58:48.864341 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.864301    2547 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-78.ec2.internal" event="NodeHasSufficientPID"
Sep 08 01:58:48.908413 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.908392    2547 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-142-78.ec2.internal" not found
Sep 08 01:58:49.009283 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:58:48.943536    2547 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-78.ec2.internal\" not found"
Sep 08 01:58:49.009283 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.987526    2547 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 08 01:58:49.009283 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.987585    2547 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 08 01:58:49.009283 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.987611    2547 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 08 01:58:49.009283 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:58:48.987709    2547 kubelet.go:2472] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Sep 08 01:58:49.009283 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:48.989646    2547 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 01:58:49.009283 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:49.007963    2547 policy_none.go:49] "None policy: Start"
Sep 08 01:58:49.009283 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:49.007998    2547 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 08 01:58:49.009283 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:49.008016    2547 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 08 01:58:49.010131 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:49.010115    2547 policy_none.go:47] "Start"
Sep 08 01:58:49.042744 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:49.042726    2547 manager.go:337] "Starting Device Plugin manager"
Sep 08 01:58:49.042837 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:58:49.042774    2547 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 08 01:58:49.042837 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:49.042790    2547 server.go:87] "Starting device plugin registration server"
Sep 08 01:58:49.043225 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:49.043211    2547 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 08 01:58:49.043296 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:49.043225    2547 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 08 01:58:49.043360 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:49.043340    2547 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 08 01:58:49.043455 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:49.043437    2547 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 08 01:58:49.043455 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:49.043452    2547 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 08 01:58:49.045562 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:58:49.045537    2547 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 08 01:58:49.045669 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:58:49.045595    2547 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-78.ec2.internal\" not found"
Sep 08 01:58:49.088407 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:49.088350    2547 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-78.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-78.ec2.internal"]
Sep 08 01:58:49.088579 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:49.088454    2547 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 01:58:49.090068 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:49.090037    2547 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-78.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 01:58:49.090194 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:49.090076    2547 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-78.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 01:58:49.090194 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:49.090086    2547 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-78.ec2.internal" event="NodeHasSufficientPID"
Sep 08 01:58:49.092359 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:49.092346    2547 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 01:58:49.092476 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:49.092462    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-78.ec2.internal"
Sep 08 01:58:49.092511 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:49.092492    2547 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 01:58:49.093198 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:49.093181    2547 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-78.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 01:58:49.093270 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:49.093211    2547 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-78.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 01:58:49.093270 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:49.093221    2547 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-78.ec2.internal" event="NodeHasSufficientPID"
Sep 08 01:58:49.093270 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:49.093183    2547 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-78.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 01:58:49.093357 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:49.093281    2547 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-78.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 01:58:49.093357 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:49.093291    2547 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-78.ec2.internal" event="NodeHasSufficientPID"
Sep 08 01:58:49.095506 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:49.095491    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-78.ec2.internal"
Sep 08 01:58:49.095600 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:49.095522    2547 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 01:58:49.096319 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:49.096303    2547 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-78.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 01:58:49.096386 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:49.096336    2547 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-78.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 01:58:49.096386 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:49.096352    2547 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-78.ec2.internal" event="NodeHasSufficientPID"
Sep 08 01:58:49.117908 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:58:49.117885    2547 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-78.ec2.internal\" not found" node="ip-10-0-142-78.ec2.internal"
Sep 08 01:58:49.122577 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:58:49.122561    2547 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-78.ec2.internal\" not found" node="ip-10-0-142-78.ec2.internal"
Sep 08 01:58:49.143962 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:49.143937    2547 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 01:58:49.145193 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:49.145175    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3d670938dd837b2cf06ee8ff34788dde-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-78.ec2.internal\" (UID: \"3d670938dd837b2cf06ee8ff34788dde\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-78.ec2.internal"
Sep 08 01:58:49.145285 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:49.145200    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5cfe1f564aebe22c530741e9b588e31d-config\") pod \"kube-apiserver-proxy-ip-10-0-142-78.ec2.internal\" (UID: \"5cfe1f564aebe22c530741e9b588e31d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-78.ec2.internal"
Sep 08 01:58:49.145285 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:49.145216    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3d670938dd837b2cf06ee8ff34788dde-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-78.ec2.internal\" (UID: \"3d670938dd837b2cf06ee8ff34788dde\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-78.ec2.internal"
Sep 08 01:58:49.145769 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:49.145753    2547 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-78.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 01:58:49.145861 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:49.145782    2547 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-78.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 01:58:49.145861 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:49.145798    2547 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-78.ec2.internal" event="NodeHasSufficientPID"
Sep 08 01:58:49.145861 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:49.145832    2547 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-78.ec2.internal"
Sep 08 01:58:49.153726 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:49.153709    2547 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-78.ec2.internal"
Sep 08 01:58:49.153812 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:58:49.153731    2547 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-78.ec2.internal\": node \"ip-10-0-142-78.ec2.internal\" not found"
Sep 08 01:58:49.168449 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:58:49.168423    2547 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-78.ec2.internal\" not found"
Sep 08 01:58:49.245452 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:49.245416    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3d670938dd837b2cf06ee8ff34788dde-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-78.ec2.internal\" (UID: \"3d670938dd837b2cf06ee8ff34788dde\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-78.ec2.internal"
Sep 08 01:58:49.245452 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:49.245454    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3d670938dd837b2cf06ee8ff34788dde-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-78.ec2.internal\" (UID: \"3d670938dd837b2cf06ee8ff34788dde\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-78.ec2.internal"
Sep 08 01:58:49.245662 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:49.245474    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5cfe1f564aebe22c530741e9b588e31d-config\") pod \"kube-apiserver-proxy-ip-10-0-142-78.ec2.internal\" (UID: \"5cfe1f564aebe22c530741e9b588e31d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-78.ec2.internal"
Sep 08 01:58:49.245662 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:49.245522    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5cfe1f564aebe22c530741e9b588e31d-config\") pod \"kube-apiserver-proxy-ip-10-0-142-78.ec2.internal\" (UID: \"5cfe1f564aebe22c530741e9b588e31d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-78.ec2.internal"
Sep 08 01:58:49.245662 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:49.245525    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3d670938dd837b2cf06ee8ff34788dde-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-78.ec2.internal\" (UID: \"3d670938dd837b2cf06ee8ff34788dde\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-78.ec2.internal"
Sep 08 01:58:49.245662 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:49.245559    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3d670938dd837b2cf06ee8ff34788dde-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-78.ec2.internal\" (UID: \"3d670938dd837b2cf06ee8ff34788dde\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-78.ec2.internal"
Sep 08 01:58:49.269595 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:58:49.269515    2547 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-78.ec2.internal\" not found"
Sep 08 01:58:49.370289 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:58:49.370241    2547 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-78.ec2.internal\" not found"
Sep 08 01:58:49.420436 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:49.420409    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-78.ec2.internal"
Sep 08 01:58:49.425074 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:49.425015    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-78.ec2.internal"
Sep 08 01:58:49.470668 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:58:49.470634    2547 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-78.ec2.internal\" not found"
Sep 08 01:58:49.571898 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:58:49.571827    2547 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-78.ec2.internal\" not found"
Sep 08 01:58:49.672487 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:58:49.672461    2547 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-78.ec2.internal\" not found"
Sep 08 01:58:49.772235 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:49.772191    2547 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 08 01:58:49.772774 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:49.772418    2547 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 08 01:58:49.772774 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:49.772418    2547 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 08 01:58:49.773293 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:58:49.773259    2547 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-78.ec2.internal\" not found"
Sep 08 01:58:49.831774 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:49.831675    2547 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-07 01:53:49 +0000 UTC" deadline="2028-03-11 03:29:23.700296386 +0000 UTC"
Sep 08 01:58:49.831774 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:49.831714    2547 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13201h30m33.868586801s"
Sep 08 01:58:49.875407 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:58:49.875381    2547 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-78.ec2.internal\" not found"
Sep 08 01:58:49.976273 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:58:49.976234    2547 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-78.ec2.internal\" not found"
Sep 08 01:58:50.077352 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:58:50.077316    2547 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-78.ec2.internal\" not found"
Sep 08 01:58:50.126026 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.125956    2547 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 01:58:50.158724 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:50.158693    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d670938dd837b2cf06ee8ff34788dde.slice/crio-33ca4536c129feefe30e11a16a11e24a01f0abf7b5fa10390c0dd9956bfe3bfe WatchSource:0}: Error finding container 33ca4536c129feefe30e11a16a11e24a01f0abf7b5fa10390c0dd9956bfe3bfe: Status 404 returned error can't find the container with id 33ca4536c129feefe30e11a16a11e24a01f0abf7b5fa10390c0dd9956bfe3bfe
Sep 08 01:58:50.159043 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:50.159021    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cfe1f564aebe22c530741e9b588e31d.slice/crio-95808e3829d4141c2b64cf6ce26f49da043b5b388dbefca07488502693f258a6 WatchSource:0}: Error finding container 95808e3829d4141c2b64cf6ce26f49da043b5b388dbefca07488502693f258a6: Status 404 returned error can't find the container with id 95808e3829d4141c2b64cf6ce26f49da043b5b388dbefca07488502693f258a6
Sep 08 01:58:50.163942 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.163922    2547 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 01:58:50.175794 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.175773    2547 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 01:58:50.248108 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.248071    2547 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-78.ec2.internal"
Sep 08 01:58:50.259238 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.259211    2547 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 08 01:58:50.260235 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.260220    2547 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-78.ec2.internal"
Sep 08 01:58:50.268360 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.268343    2547 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 08 01:58:50.822997 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.822938    2547 apiserver.go:52] "Watching apiserver"
Sep 08 01:58:50.834281 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.834247    2547 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 08 01:58:50.834809 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.834781    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-2v24n","kube-system/konnectivity-agent-xr9c8","openshift-cluster-node-tuning-operator/tuned-x72v8","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-78.ec2.internal","openshift-multus/network-metrics-daemon-jr45c","openshift-network-operator/iptables-alerter-b7dzq","openshift-ovn-kubernetes/ovnkube-node-xskxl","kube-system/global-pull-secret-syncer-l7hw8","kube-system/kube-apiserver-proxy-ip-10-0-142-78.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lbtkf","openshift-dns/node-resolver-4s9cm","openshift-image-registry/node-ca-pl5qh","openshift-multus/multus-7wjjb","openshift-multus/multus-additional-cni-plugins-b4ms7"]
Sep 08 01:58:50.837778 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.837753    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lbtkf"
Sep 08 01:58:50.840276 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.840254    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 08 01:58:50.840276 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.840266    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 08 01:58:50.841562 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.840568    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-bx8f9\""
Sep 08 01:58:50.841562 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.840748    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 08 01:58:50.841562 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.840919    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7wjjb"
Sep 08 01:58:50.845156 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.844670    2547 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 08 01:58:50.845156 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.844723    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-x72v8"
Sep 08 01:58:50.845671 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.845651    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 08 01:58:50.845869 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.845852    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 08 01:58:50.846043 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.846023    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 08 01:58:50.846124 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.846095    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 08 01:58:50.846311 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.846292    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-djzfq\""
Sep 08 01:58:50.846395 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.846339    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-b4ms7"
Sep 08 01:58:50.847085 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.847066    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 08 01:58:50.847494 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.847361    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 08 01:58:50.847861 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.847704    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-qwsdw\""
Sep 08 01:58:50.848352 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.848331    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jr45c"
Sep 08 01:58:50.848454 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:58:50.848406    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jr45c" podUID="b82c5af2-4c1c-41a9-8fa4-af8a85bcc8c6"
Sep 08 01:58:50.848675 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.848652    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-zs9wd\""
Sep 08 01:58:50.848756 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.848743    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 08 01:58:50.848832 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.848659    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 08 01:58:50.851066 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.850740    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-b7dzq"
Sep 08 01:58:50.853779 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.852755    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 08 01:58:50.853779 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.852790    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 08 01:58:50.853779 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.853061    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-nlrpv\""
Sep 08 01:58:50.853779 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.853263    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 08 01:58:50.853779 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.853521    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l7hw8"
Sep 08 01:58:50.853779 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:58:50.853582    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l7hw8" podUID="917f6c17-7902-48d3-a0f4-535e5b602d26"
Sep 08 01:58:50.854129 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.853943    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2d7adf2a-2cfe-40ff-90d3-4886a747463a-system-cni-dir\") pod \"multus-additional-cni-plugins-b4ms7\" (UID: \"2d7adf2a-2cfe-40ff-90d3-4886a747463a\") " pod="openshift-multus/multus-additional-cni-plugins-b4ms7"
Sep 08 01:58:50.854129 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.853981    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2d7adf2a-2cfe-40ff-90d3-4886a747463a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-b4ms7\" (UID: \"2d7adf2a-2cfe-40ff-90d3-4886a747463a\") " pod="openshift-multus/multus-additional-cni-plugins-b4ms7"
Sep 08 01:58:50.854129 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.854015    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvrnj\" (UniqueName: \"kubernetes.io/projected/2d7adf2a-2cfe-40ff-90d3-4886a747463a-kube-api-access-pvrnj\") pod \"multus-additional-cni-plugins-b4ms7\" (UID: \"2d7adf2a-2cfe-40ff-90d3-4886a747463a\") " pod="openshift-multus/multus-additional-cni-plugins-b4ms7"
Sep 08 01:58:50.854129 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.854038    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0ce55233-87a7-4f6d-88a0-79d5d8eb2e3c-cni-binary-copy\") pod \"multus-7wjjb\" (UID: \"0ce55233-87a7-4f6d-88a0-79d5d8eb2e3c\") " pod="openshift-multus/multus-7wjjb"
Sep 08 01:58:50.854129 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.854092    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/44506a48-1a0a-4143-8ced-0dea3455ad2d-etc-selinux\") pod \"aws-ebs-csi-driver-node-lbtkf\" (UID: \"44506a48-1a0a-4143-8ced-0dea3455ad2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lbtkf"
Sep 08 01:58:50.854129 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.854127    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0ce55233-87a7-4f6d-88a0-79d5d8eb2e3c-host-run-netns\") pod \"multus-7wjjb\" (UID: \"0ce55233-87a7-4f6d-88a0-79d5d8eb2e3c\") " pod="openshift-multus/multus-7wjjb"
Sep 08 01:58:50.854486 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.854166    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0ce55233-87a7-4f6d-88a0-79d5d8eb2e3c-host-var-lib-cni-multus\") pod \"multus-7wjjb\" (UID: \"0ce55233-87a7-4f6d-88a0-79d5d8eb2e3c\") " pod="openshift-multus/multus-7wjjb"
Sep 08 01:58:50.854486 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.854238    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0ce55233-87a7-4f6d-88a0-79d5d8eb2e3c-etc-kubernetes\") pod \"multus-7wjjb\" (UID: \"0ce55233-87a7-4f6d-88a0-79d5d8eb2e3c\") " pod="openshift-multus/multus-7wjjb"
Sep 08 01:58:50.854486 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.854265    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hq9x\" (UniqueName: \"kubernetes.io/projected/af873b12-c8cd-4c48-a016-0be5e88e9894-kube-api-access-2hq9x\") pod \"tuned-x72v8\" (UID: \"af873b12-c8cd-4c48-a016-0be5e88e9894\") " pod="openshift-cluster-node-tuning-operator/tuned-x72v8"
Sep 08 01:58:50.854486 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.854309    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2d7adf2a-2cfe-40ff-90d3-4886a747463a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-b4ms7\" (UID: \"2d7adf2a-2cfe-40ff-90d3-4886a747463a\") " pod="openshift-multus/multus-additional-cni-plugins-b4ms7"
Sep 08 01:58:50.854486 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.854342    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0ce55233-87a7-4f6d-88a0-79d5d8eb2e3c-multus-conf-dir\") pod \"multus-7wjjb\" (UID: \"0ce55233-87a7-4f6d-88a0-79d5d8eb2e3c\") " pod="openshift-multus/multus-7wjjb"
Sep 08 01:58:50.854486 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.854400    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0ce55233-87a7-4f6d-88a0-79d5d8eb2e3c-multus-daemon-config\") pod \"multus-7wjjb\" (UID: \"0ce55233-87a7-4f6d-88a0-79d5d8eb2e3c\") " pod="openshift-multus/multus-7wjjb"
Sep 08 01:58:50.854486 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.854432    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/af873b12-c8cd-4c48-a016-0be5e88e9894-etc-sysconfig\") pod \"tuned-x72v8\" (UID: \"af873b12-c8cd-4c48-a016-0be5e88e9894\") " pod="openshift-cluster-node-tuning-operator/tuned-x72v8"
Sep 08 01:58:50.854486 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.854465    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/af873b12-c8cd-4c48-a016-0be5e88e9894-var-lib-kubelet\") pod \"tuned-x72v8\" (UID: \"af873b12-c8cd-4c48-a016-0be5e88e9894\") " pod="openshift-cluster-node-tuning-operator/tuned-x72v8"
Sep 08 01:58:50.854877 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.854491    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/af873b12-c8cd-4c48-a016-0be5e88e9894-host\") pod \"tuned-x72v8\" (UID: \"af873b12-c8cd-4c48-a016-0be5e88e9894\") " pod="openshift-cluster-node-tuning-operator/tuned-x72v8"
Sep 08 01:58:50.854877 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.854519    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/44506a48-1a0a-4143-8ced-0dea3455ad2d-device-dir\") pod \"aws-ebs-csi-driver-node-lbtkf\" (UID: \"44506a48-1a0a-4143-8ced-0dea3455ad2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lbtkf"
Sep 08 01:58:50.854877 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.854572    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0ce55233-87a7-4f6d-88a0-79d5d8eb2e3c-multus-socket-dir-parent\") pod \"multus-7wjjb\" (UID: \"0ce55233-87a7-4f6d-88a0-79d5d8eb2e3c\") " pod="openshift-multus/multus-7wjjb"
Sep 08 01:58:50.854877 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.854609    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0ce55233-87a7-4f6d-88a0-79d5d8eb2e3c-host-var-lib-cni-bin\") pod \"multus-7wjjb\" (UID: \"0ce55233-87a7-4f6d-88a0-79d5d8eb2e3c\") " pod="openshift-multus/multus-7wjjb"
Sep 08 01:58:50.854877 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.854635    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0ce55233-87a7-4f6d-88a0-79d5d8eb2e3c-hostroot\") pod \"multus-7wjjb\" (UID: \"0ce55233-87a7-4f6d-88a0-79d5d8eb2e3c\") " pod="openshift-multus/multus-7wjjb"
Sep 08 01:58:50.854877 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.854659    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/af873b12-c8cd-4c48-a016-0be5e88e9894-etc-kubernetes\") pod \"tuned-x72v8\" (UID: \"af873b12-c8cd-4c48-a016-0be5e88e9894\") " pod="openshift-cluster-node-tuning-operator/tuned-x72v8"
Sep 08 01:58:50.854877 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.854683    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/af873b12-c8cd-4c48-a016-0be5e88e9894-etc-sysctl-conf\") pod \"tuned-x72v8\" (UID: \"af873b12-c8cd-4c48-a016-0be5e88e9894\") " pod="openshift-cluster-node-tuning-operator/tuned-x72v8"
Sep 08 01:58:50.854877 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.854707    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/af873b12-c8cd-4c48-a016-0be5e88e9894-run\") pod \"tuned-x72v8\" (UID: \"af873b12-c8cd-4c48-a016-0be5e88e9894\") " pod="openshift-cluster-node-tuning-operator/tuned-x72v8"
Sep 08 01:58:50.854877 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.854733    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/44506a48-1a0a-4143-8ced-0dea3455ad2d-socket-dir\") pod \"aws-ebs-csi-driver-node-lbtkf\" (UID: \"44506a48-1a0a-4143-8ced-0dea3455ad2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lbtkf"
Sep 08 01:58:50.854877 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.854758    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvwpz\" (UniqueName: \"kubernetes.io/projected/0ce55233-87a7-4f6d-88a0-79d5d8eb2e3c-kube-api-access-tvwpz\") pod \"multus-7wjjb\" (UID: \"0ce55233-87a7-4f6d-88a0-79d5d8eb2e3c\") " pod="openshift-multus/multus-7wjjb"
Sep 08 01:58:50.854877 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.854795    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2d7adf2a-2cfe-40ff-90d3-4886a747463a-cni-binary-copy\") pod \"multus-additional-cni-plugins-b4ms7\" (UID: \"2d7adf2a-2cfe-40ff-90d3-4886a747463a\") " pod="openshift-multus/multus-additional-cni-plugins-b4ms7"
Sep 08 01:58:50.854877 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.854825    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0ce55233-87a7-4f6d-88a0-79d5d8eb2e3c-host-var-lib-kubelet\") pod \"multus-7wjjb\" (UID: \"0ce55233-87a7-4f6d-88a0-79d5d8eb2e3c\") " pod="openshift-multus/multus-7wjjb"
Sep 08 01:58:50.854877 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.854857    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/af873b12-c8cd-4c48-a016-0be5e88e9894-etc-sysctl-d\") pod \"tuned-x72v8\" (UID: \"af873b12-c8cd-4c48-a016-0be5e88e9894\") " pod="openshift-cluster-node-tuning-operator/tuned-x72v8"
Sep 08 01:58:50.854877 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.854881    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/af873b12-c8cd-4c48-a016-0be5e88e9894-etc-systemd\") pod \"tuned-x72v8\" (UID: \"af873b12-c8cd-4c48-a016-0be5e88e9894\") " pod="openshift-cluster-node-tuning-operator/tuned-x72v8"
Sep 08 01:58:50.855599 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.854903    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/af873b12-c8cd-4c48-a016-0be5e88e9894-sys\") pod \"tuned-x72v8\" (UID: \"af873b12-c8cd-4c48-a016-0be5e88e9894\") " pod="openshift-cluster-node-tuning-operator/tuned-x72v8"
Sep 08 01:58:50.855599 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.854928    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/af873b12-c8cd-4c48-a016-0be5e88e9894-tmp\") pod \"tuned-x72v8\" (UID: \"af873b12-c8cd-4c48-a016-0be5e88e9894\") " pod="openshift-cluster-node-tuning-operator/tuned-x72v8"
Sep 08 01:58:50.855599 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.854951    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2d7adf2a-2cfe-40ff-90d3-4886a747463a-cnibin\") pod \"multus-additional-cni-plugins-b4ms7\" (UID: \"2d7adf2a-2cfe-40ff-90d3-4886a747463a\") " pod="openshift-multus/multus-additional-cni-plugins-b4ms7"
Sep 08 01:58:50.855599 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.854977    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2d7adf2a-2cfe-40ff-90d3-4886a747463a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b4ms7\" (UID: \"2d7adf2a-2cfe-40ff-90d3-4886a747463a\") " pod="openshift-multus/multus-additional-cni-plugins-b4ms7"
Sep 08 01:58:50.855599 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.855001    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0ce55233-87a7-4f6d-88a0-79d5d8eb2e3c-multus-cni-dir\") pod \"multus-7wjjb\" (UID: \"0ce55233-87a7-4f6d-88a0-79d5d8eb2e3c\") " pod="openshift-multus/multus-7wjjb"
Sep 08 01:58:50.855599 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.855038    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0ce55233-87a7-4f6d-88a0-79d5d8eb2e3c-cnibin\") pod \"multus-7wjjb\" (UID: \"0ce55233-87a7-4f6d-88a0-79d5d8eb2e3c\") " pod="openshift-multus/multus-7wjjb"
Sep 08 01:58:50.855599 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.855071    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/44506a48-1a0a-4143-8ced-0dea3455ad2d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lbtkf\" (UID: \"44506a48-1a0a-4143-8ced-0dea3455ad2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lbtkf"
Sep 08 01:58:50.855599 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.855118    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/44506a48-1a0a-4143-8ced-0dea3455ad2d-sys-fs\") pod \"aws-ebs-csi-driver-node-lbtkf\" (UID: \"44506a48-1a0a-4143-8ced-0dea3455ad2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lbtkf"
Sep 08 01:58:50.855599 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.855228    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbdmc\" (UniqueName: \"kubernetes.io/projected/44506a48-1a0a-4143-8ced-0dea3455ad2d-kube-api-access-hbdmc\") pod \"aws-ebs-csi-driver-node-lbtkf\" (UID: \"44506a48-1a0a-4143-8ced-0dea3455ad2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lbtkf"
Sep 08 01:58:50.855599 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.855258    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0ce55233-87a7-4f6d-88a0-79d5d8eb2e3c-system-cni-dir\") pod \"multus-7wjjb\" (UID: \"0ce55233-87a7-4f6d-88a0-79d5d8eb2e3c\") " pod="openshift-multus/multus-7wjjb"
Sep 08 01:58:50.855599 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.855283    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0ce55233-87a7-4f6d-88a0-79d5d8eb2e3c-host-run-k8s-cni-cncf-io\") pod \"multus-7wjjb\" (UID: \"0ce55233-87a7-4f6d-88a0-79d5d8eb2e3c\") " pod="openshift-multus/multus-7wjjb"
Sep 08 01:58:50.855599 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.855367    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0ce55233-87a7-4f6d-88a0-79d5d8eb2e3c-host-run-multus-certs\") pod \"multus-7wjjb\" (UID: \"0ce55233-87a7-4f6d-88a0-79d5d8eb2e3c\") " pod="openshift-multus/multus-7wjjb"
Sep 08 01:58:50.855599 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.855395    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/44506a48-1a0a-4143-8ced-0dea3455ad2d-registration-dir\") pod \"aws-ebs-csi-driver-node-lbtkf\" (UID: \"44506a48-1a0a-4143-8ced-0dea3455ad2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lbtkf"
Sep 08 01:58:50.855599 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.855419    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/af873b12-c8cd-4c48-a016-0be5e88e9894-etc-modprobe-d\") pod \"tuned-x72v8\" (UID: \"af873b12-c8cd-4c48-a016-0be5e88e9894\") " pod="openshift-cluster-node-tuning-operator/tuned-x72v8"
Sep 08 01:58:50.855599 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.855444    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/af873b12-c8cd-4c48-a016-0be5e88e9894-lib-modules\") pod \"tuned-x72v8\" (UID: \"af873b12-c8cd-4c48-a016-0be5e88e9894\") " pod="openshift-cluster-node-tuning-operator/tuned-x72v8"
Sep 08 01:58:50.855599 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.855489    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/af873b12-c8cd-4c48-a016-0be5e88e9894-etc-tuned\") pod \"tuned-x72v8\" (UID: \"af873b12-c8cd-4c48-a016-0be5e88e9894\") " pod="openshift-cluster-node-tuning-operator/tuned-x72v8"
Sep 08 01:58:50.858085 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.857679    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xskxl"
Sep 08 01:58:50.861246 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.861208    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xr9c8"
Sep 08 01:58:50.861538 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.861481    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 08 01:58:50.861755 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.861649    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 08 01:58:50.862096 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.861868    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 08 01:58:50.862096 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.861881    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 08 01:58:50.862096 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.861984    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 08 01:58:50.862096 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.862043    2547 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 08 01:58:50.862355 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.862143    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 08 01:58:50.862355 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.861883    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-wjlgx\""
Sep 08 01:58:50.863958 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.863933    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 08 01:58:50.865350 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.864400    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 08 01:58:50.865350 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.864558    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2v24n"
Sep 08 01:58:50.865350 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:58:50.864618    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2v24n" podUID="430aad96-9d26-4055-8133-b7f50b0f2978"
Sep 08 01:58:50.865350 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.864747    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-b2vtb\""
Sep 08 01:58:50.866892 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.866872    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4s9cm"
Sep 08 01:58:50.871195 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.871164    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 08 01:58:50.872123 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.872103    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-5xfgw\""
Sep 08 01:58:50.873059 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.872386    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 08 01:58:50.874843 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.874824    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-pl5qh"
Sep 08 01:58:50.877384 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.877364    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 08 01:58:50.877609 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.877592    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 08 01:58:50.877691 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.877672    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 08 01:58:50.877916 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.877899    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-kz4hl\""
Sep 08 01:58:50.882670 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.882646    2547 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-bmrsx"
Sep 08 01:58:50.890695 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.890671    2547 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-bmrsx"
Sep 08 01:58:50.944533 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.944509    2547 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 01:58:50.945506 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.945484    2547 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 08 01:58:50.956181 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.956137    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4b89e1b1-bcc0-4027-b778-19a7aba264a9-iptables-alerter-script\") pod \"iptables-alerter-b7dzq\" (UID: \"4b89e1b1-bcc0-4027-b778-19a7aba264a9\") " pod="openshift-network-operator/iptables-alerter-b7dzq"
Sep 08 01:58:50.956322 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.956196    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dt4k\" (UniqueName: \"kubernetes.io/projected/430aad96-9d26-4055-8133-b7f50b0f2978-kube-api-access-5dt4k\") pod \"network-check-target-2v24n\" (UID: \"430aad96-9d26-4055-8133-b7f50b0f2978\") " pod="openshift-network-diagnostics/network-check-target-2v24n"
Sep 08 01:58:50.956322 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.956227    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvrnj\" (UniqueName: \"kubernetes.io/projected/2d7adf2a-2cfe-40ff-90d3-4886a747463a-kube-api-access-pvrnj\") pod \"multus-additional-cni-plugins-b4ms7\" (UID: \"2d7adf2a-2cfe-40ff-90d3-4886a747463a\") " pod="openshift-multus/multus-additional-cni-plugins-b4ms7"
Sep 08 01:58:50.956322 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.956253    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/275d1493-2484-4412-b30d-b2aa36723bef-log-socket\") pod \"ovnkube-node-xskxl\" (UID: \"275d1493-2484-4412-b30d-b2aa36723bef\") " pod="openshift-ovn-kubernetes/ovnkube-node-xskxl"
Sep 08 01:58:50.956322 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.956280    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/275d1493-2484-4412-b30d-b2aa36723bef-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xskxl\" (UID: \"275d1493-2484-4412-b30d-b2aa36723bef\") " pod="openshift-ovn-kubernetes/ovnkube-node-xskxl"
Sep 08 01:58:50.956541 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.956319    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4b89e1b1-bcc0-4027-b778-19a7aba264a9-host-slash\") pod \"iptables-alerter-b7dzq\" (UID: \"4b89e1b1-bcc0-4027-b778-19a7aba264a9\") " pod="openshift-network-operator/iptables-alerter-b7dzq"
Sep 08 01:58:50.956541 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.956370    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0ce55233-87a7-4f6d-88a0-79d5d8eb2e3c-cni-binary-copy\") pod \"multus-7wjjb\" (UID: \"0ce55233-87a7-4f6d-88a0-79d5d8eb2e3c\") " pod="openshift-multus/multus-7wjjb"
Sep 08 01:58:50.956541 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.956397    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0ce55233-87a7-4f6d-88a0-79d5d8eb2e3c-host-run-netns\") pod \"multus-7wjjb\" (UID: \"0ce55233-87a7-4f6d-88a0-79d5d8eb2e3c\") " pod="openshift-multus/multus-7wjjb"
Sep 08 01:58:50.956541 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.956432    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/275d1493-2484-4412-b30d-b2aa36723bef-run-openvswitch\") pod \"ovnkube-node-xskxl\" (UID: \"275d1493-2484-4412-b30d-b2aa36723bef\") " pod="openshift-ovn-kubernetes/ovnkube-node-xskxl"
Sep 08 01:58:50.956541 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.956484    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0ce55233-87a7-4f6d-88a0-79d5d8eb2e3c-host-run-netns\") pod \"multus-7wjjb\" (UID: \"0ce55233-87a7-4f6d-88a0-79d5d8eb2e3c\") " pod="openshift-multus/multus-7wjjb"
Sep 08 01:58:50.956541 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.956527    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0ce55233-87a7-4f6d-88a0-79d5d8eb2e3c-multus-conf-dir\") pod \"multus-7wjjb\" (UID: \"0ce55233-87a7-4f6d-88a0-79d5d8eb2e3c\") " pod="openshift-multus/multus-7wjjb"
Sep 08 01:58:50.956856 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.956562    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0ce55233-87a7-4f6d-88a0-79d5d8eb2e3c-multus-daemon-config\") pod \"multus-7wjjb\" (UID: \"0ce55233-87a7-4f6d-88a0-79d5d8eb2e3c\") " pod="openshift-multus/multus-7wjjb"
Sep 08 01:58:50.956856 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.956590    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/af873b12-c8cd-4c48-a016-0be5e88e9894-var-lib-kubelet\") pod \"tuned-x72v8\" (UID: \"af873b12-c8cd-4c48-a016-0be5e88e9894\") " pod="openshift-cluster-node-tuning-operator/tuned-x72v8"
Sep 08 01:58:50.956856 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.956636    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0ce55233-87a7-4f6d-88a0-79d5d8eb2e3c-multus-conf-dir\") pod \"multus-7wjjb\" (UID: \"0ce55233-87a7-4f6d-88a0-79d5d8eb2e3c\") " pod="openshift-multus/multus-7wjjb"
Sep 08 01:58:50.956856 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.956687    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/af873b12-c8cd-4c48-a016-0be5e88e9894-var-lib-kubelet\") pod \"tuned-x72v8\" (UID: \"af873b12-c8cd-4c48-a016-0be5e88e9894\") " pod="openshift-cluster-node-tuning-operator/tuned-x72v8"
Sep 08 01:58:50.956856 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.956719    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/af873b12-c8cd-4c48-a016-0be5e88e9894-host\") pod \"tuned-x72v8\" (UID: \"af873b12-c8cd-4c48-a016-0be5e88e9894\") " pod="openshift-cluster-node-tuning-operator/tuned-x72v8"
Sep 08 01:58:50.956856 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.956754    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/275d1493-2484-4412-b30d-b2aa36723bef-env-overrides\") pod \"ovnkube-node-xskxl\" (UID: \"275d1493-2484-4412-b30d-b2aa36723bef\") " pod="openshift-ovn-kubernetes/ovnkube-node-xskxl"
Sep 08 01:58:50.956856 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.956798    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0ce55233-87a7-4f6d-88a0-79d5d8eb2e3c-multus-socket-dir-parent\") pod \"multus-7wjjb\" (UID: \"0ce55233-87a7-4f6d-88a0-79d5d8eb2e3c\") " pod="openshift-multus/multus-7wjjb"
Sep 08 01:58:50.956856 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.956825    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0ce55233-87a7-4f6d-88a0-79d5d8eb2e3c-host-var-lib-cni-bin\") pod \"multus-7wjjb\" (UID: \"0ce55233-87a7-4f6d-88a0-79d5d8eb2e3c\") " pod="openshift-multus/multus-7wjjb"
Sep 08 01:58:50.957346 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.956859    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0ce55233-87a7-4f6d-88a0-79d5d8eb2e3c-hostroot\") pod \"multus-7wjjb\" (UID: \"0ce55233-87a7-4f6d-88a0-79d5d8eb2e3c\") " pod="openshift-multus/multus-7wjjb"
Sep 08 01:58:50.957346 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.956883    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/af873b12-c8cd-4c48-a016-0be5e88e9894-etc-kubernetes\") pod \"tuned-x72v8\" (UID: \"af873b12-c8cd-4c48-a016-0be5e88e9894\") " pod="openshift-cluster-node-tuning-operator/tuned-x72v8"
Sep 08 01:58:50.957346 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.956889    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0ce55233-87a7-4f6d-88a0-79d5d8eb2e3c-multus-socket-dir-parent\") pod \"multus-7wjjb\" (UID: \"0ce55233-87a7-4f6d-88a0-79d5d8eb2e3c\") " pod="openshift-multus/multus-7wjjb"
Sep 08 01:58:50.957346 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.956833    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/af873b12-c8cd-4c48-a016-0be5e88e9894-host\") pod \"tuned-x72v8\" (UID: \"af873b12-c8cd-4c48-a016-0be5e88e9894\") " pod="openshift-cluster-node-tuning-operator/tuned-x72v8"
Sep 08 01:58:50.957346 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.956913    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/917f6c17-7902-48d3-a0f4-535e5b602d26-kubelet-config\") pod \"global-pull-secret-syncer-l7hw8\" (UID: \"917f6c17-7902-48d3-a0f4-535e5b602d26\") " pod="kube-system/global-pull-secret-syncer-l7hw8"
Sep 08 01:58:50.957346 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.956948    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7bs2\" (UniqueName: \"kubernetes.io/projected/b82c5af2-4c1c-41a9-8fa4-af8a85bcc8c6-kube-api-access-n7bs2\") pod \"network-metrics-daemon-jr45c\" (UID: \"b82c5af2-4c1c-41a9-8fa4-af8a85bcc8c6\") " pod="openshift-multus/network-metrics-daemon-jr45c"
Sep 08 01:58:50.957346 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.956970    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0ce55233-87a7-4f6d-88a0-79d5d8eb2e3c-hostroot\") pod \"multus-7wjjb\" (UID: \"0ce55233-87a7-4f6d-88a0-79d5d8eb2e3c\") " pod="openshift-multus/multus-7wjjb"
Sep 08 01:58:50.957346 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.956974    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/275d1493-2484-4412-b30d-b2aa36723bef-host-cni-bin\") pod \"ovnkube-node-xskxl\" (UID: \"275d1493-2484-4412-b30d-b2aa36723bef\") " pod="openshift-ovn-kubernetes/ovnkube-node-xskxl"
Sep 08 01:58:50.957346 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.957002    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwg6c\" (UniqueName: \"kubernetes.io/projected/4b89e1b1-bcc0-4027-b778-19a7aba264a9-kube-api-access-hwg6c\") pod \"iptables-alerter-b7dzq\" (UID: \"4b89e1b1-bcc0-4027-b778-19a7aba264a9\") " pod="openshift-network-operator/iptables-alerter-b7dzq"
Sep 08 01:58:50.957346 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.957030    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/af873b12-c8cd-4c48-a016-0be5e88e9894-etc-sysctl-d\") pod \"tuned-x72v8\" (UID: \"af873b12-c8cd-4c48-a016-0be5e88e9894\") " pod="openshift-cluster-node-tuning-operator/tuned-x72v8"
Sep 08 01:58:50.957346 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.957055    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/af873b12-c8cd-4c48-a016-0be5e88e9894-etc-systemd\") pod \"tuned-x72v8\" (UID: \"af873b12-c8cd-4c48-a016-0be5e88e9894\") " pod="openshift-cluster-node-tuning-operator/tuned-x72v8"
Sep 08 01:58:50.957346 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.957080    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/af873b12-c8cd-4c48-a016-0be5e88e9894-sys\") pod \"tuned-x72v8\" (UID: \"af873b12-c8cd-4c48-a016-0be5e88e9894\") " pod="openshift-cluster-node-tuning-operator/tuned-x72v8"
Sep 08 01:58:50.957346 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.957103    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/af873b12-c8cd-4c48-a016-0be5e88e9894-tmp\") pod \"tuned-x72v8\" (UID: \"af873b12-c8cd-4c48-a016-0be5e88e9894\") " pod="openshift-cluster-node-tuning-operator/tuned-x72v8"
Sep 08 01:58:50.957346 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.957128    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2d7adf2a-2cfe-40ff-90d3-4886a747463a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b4ms7\" (UID: \"2d7adf2a-2cfe-40ff-90d3-4886a747463a\") " pod="openshift-multus/multus-additional-cni-plugins-b4ms7"
Sep 08 01:58:50.957346 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.957155    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/275d1493-2484-4412-b30d-b2aa36723bef-ovnkube-config\") pod \"ovnkube-node-xskxl\" (UID: \"275d1493-2484-4412-b30d-b2aa36723bef\") " pod="openshift-ovn-kubernetes/ovnkube-node-xskxl"
Sep 08 01:58:50.957346 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.957197    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0ce55233-87a7-4f6d-88a0-79d5d8eb2e3c-host-var-lib-cni-bin\") pod \"multus-7wjjb\" (UID: \"0ce55233-87a7-4f6d-88a0-79d5d8eb2e3c\") " pod="openshift-multus/multus-7wjjb"
Sep 08 01:58:50.957346 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.957224    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/275d1493-2484-4412-b30d-b2aa36723bef-ovn-node-metrics-cert\") pod \"ovnkube-node-xskxl\" (UID: \"275d1493-2484-4412-b30d-b2aa36723bef\") " pod="openshift-ovn-kubernetes/ovnkube-node-xskxl"
Sep 08 01:58:50.958168 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.957233    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/af873b12-c8cd-4c48-a016-0be5e88e9894-sys\") pod \"tuned-x72v8\" (UID: \"af873b12-c8cd-4c48-a016-0be5e88e9894\") " pod="openshift-cluster-node-tuning-operator/tuned-x72v8"
Sep 08 01:58:50.958168 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.957253    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ae4a55d0-ad38-48a9-9a36-279949e2e4c5-serviceca\") pod \"node-ca-pl5qh\" (UID: \"ae4a55d0-ad38-48a9-9a36-279949e2e4c5\") " pod="openshift-image-registry/node-ca-pl5qh"
Sep 08 01:58:50.958168 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.957278    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0536c61b-c71c-41d9-8b2c-18228a7ddff4-hosts-file\") pod \"node-resolver-4s9cm\" (UID: \"0536c61b-c71c-41d9-8b2c-18228a7ddff4\") " pod="openshift-dns/node-resolver-4s9cm"
Sep 08 01:58:50.958168 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.957296    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/af873b12-c8cd-4c48-a016-0be5e88e9894-etc-systemd\") pod \"tuned-x72v8\" (UID: \"af873b12-c8cd-4c48-a016-0be5e88e9894\") " pod="openshift-cluster-node-tuning-operator/tuned-x72v8"
Sep 08 01:58:50.958168 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.957294    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/af873b12-c8cd-4c48-a016-0be5e88e9894-etc-kubernetes\") pod \"tuned-x72v8\" (UID: \"af873b12-c8cd-4c48-a016-0be5e88e9894\") " pod="openshift-cluster-node-tuning-operator/tuned-x72v8"
Sep 08 01:58:50.958168 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.957300    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c793f939-4584-46af-a495-36b57bbdff41-agent-certs\") pod \"konnectivity-agent-xr9c8\" (UID: \"c793f939-4584-46af-a495-36b57bbdff41\") " pod="kube-system/konnectivity-agent-xr9c8"
Sep 08 01:58:50.958168 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.957357    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0ce55233-87a7-4f6d-88a0-79d5d8eb2e3c-cnibin\") pod \"multus-7wjjb\" (UID: \"0ce55233-87a7-4f6d-88a0-79d5d8eb2e3c\") " pod="openshift-multus/multus-7wjjb"
Sep 08 01:58:50.958168 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.957357    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/af873b12-c8cd-4c48-a016-0be5e88e9894-etc-sysctl-d\") pod \"tuned-x72v8\" (UID: \"af873b12-c8cd-4c48-a016-0be5e88e9894\") " pod="openshift-cluster-node-tuning-operator/tuned-x72v8"
Sep 08 01:58:50.958168 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.957389    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/44506a48-1a0a-4143-8ced-0dea3455ad2d-sys-fs\") pod \"aws-ebs-csi-driver-node-lbtkf\" (UID: \"44506a48-1a0a-4143-8ced-0dea3455ad2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lbtkf"
Sep 08 01:58:50.958168 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.957429    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0ce55233-87a7-4f6d-88a0-79d5d8eb2e3c-cnibin\") pod \"multus-7wjjb\" (UID: \"0ce55233-87a7-4f6d-88a0-79d5d8eb2e3c\") " pod="openshift-multus/multus-7wjjb"
Sep 08 01:58:50.958168 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.957429    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbdmc\" (UniqueName: \"kubernetes.io/projected/44506a48-1a0a-4143-8ced-0dea3455ad2d-kube-api-access-hbdmc\") pod \"aws-ebs-csi-driver-node-lbtkf\" (UID: \"44506a48-1a0a-4143-8ced-0dea3455ad2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lbtkf"
Sep 08 01:58:50.958168 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.957538    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2d7adf2a-2cfe-40ff-90d3-4886a747463a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b4ms7\" (UID: \"2d7adf2a-2cfe-40ff-90d3-4886a747463a\") " pod="openshift-multus/multus-additional-cni-plugins-b4ms7"
Sep 08 01:58:50.958168 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.957552    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0ce55233-87a7-4f6d-88a0-79d5d8eb2e3c-system-cni-dir\") pod \"multus-7wjjb\" (UID: \"0ce55233-87a7-4f6d-88a0-79d5d8eb2e3c\") " pod="openshift-multus/multus-7wjjb"
Sep 08 01:58:50.958168 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.957589    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/44506a48-1a0a-4143-8ced-0dea3455ad2d-sys-fs\") pod \"aws-ebs-csi-driver-node-lbtkf\" (UID: \"44506a48-1a0a-4143-8ced-0dea3455ad2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lbtkf"
Sep 08 01:58:50.958168 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.957626    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0ce55233-87a7-4f6d-88a0-79d5d8eb2e3c-host-run-multus-certs\") pod \"multus-7wjjb\" (UID: \"0ce55233-87a7-4f6d-88a0-79d5d8eb2e3c\") " pod="openshift-multus/multus-7wjjb"
Sep 08 01:58:50.958168 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.957640    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0ce55233-87a7-4f6d-88a0-79d5d8eb2e3c-cni-binary-copy\") pod \"multus-7wjjb\" (UID: \"0ce55233-87a7-4f6d-88a0-79d5d8eb2e3c\") " pod="openshift-multus/multus-7wjjb"
Sep 08 01:58:50.958168 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.957656    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2d7adf2a-2cfe-40ff-90d3-4886a747463a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-b4ms7\" (UID: \"2d7adf2a-2cfe-40ff-90d3-4886a747463a\") " pod="openshift-multus/multus-additional-cni-plugins-b4ms7"
Sep 08 01:58:50.958979 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.957684    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/275d1493-2484-4412-b30d-b2aa36723bef-etc-openvswitch\") pod \"ovnkube-node-xskxl\" (UID: \"275d1493-2484-4412-b30d-b2aa36723bef\") " pod="openshift-ovn-kubernetes/ovnkube-node-xskxl"
Sep 08 01:58:50.958979 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.957705    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ae4a55d0-ad38-48a9-9a36-279949e2e4c5-host\") pod \"node-ca-pl5qh\" (UID: \"ae4a55d0-ad38-48a9-9a36-279949e2e4c5\") " pod="openshift-image-registry/node-ca-pl5qh"
Sep 08 01:58:50.958979 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.957731    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/44506a48-1a0a-4143-8ced-0dea3455ad2d-registration-dir\") pod \"aws-ebs-csi-driver-node-lbtkf\" (UID: \"44506a48-1a0a-4143-8ced-0dea3455ad2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lbtkf"
Sep 08 01:58:50.958979 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.957752    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/af873b12-c8cd-4c48-a016-0be5e88e9894-etc-modprobe-d\") pod \"tuned-x72v8\" (UID: \"af873b12-c8cd-4c48-a016-0be5e88e9894\") " pod="openshift-cluster-node-tuning-operator/tuned-x72v8"
Sep 08 01:58:50.958979 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.957754    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0ce55233-87a7-4f6d-88a0-79d5d8eb2e3c-system-cni-dir\") pod \"multus-7wjjb\" (UID: \"0ce55233-87a7-4f6d-88a0-79d5d8eb2e3c\") " pod="openshift-multus/multus-7wjjb"
Sep 08 01:58:50.958979 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.957799    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0ce55233-87a7-4f6d-88a0-79d5d8eb2e3c-host-run-multus-certs\") pod \"multus-7wjjb\" (UID: \"0ce55233-87a7-4f6d-88a0-79d5d8eb2e3c\") " pod="openshift-multus/multus-7wjjb"
Sep 08 01:58:50.958979 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.957829    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/af873b12-c8cd-4c48-a016-0be5e88e9894-etc-modprobe-d\") pod \"tuned-x72v8\" (UID: \"af873b12-c8cd-4c48-a016-0be5e88e9894\") " pod="openshift-cluster-node-tuning-operator/tuned-x72v8"
Sep 08 01:58:50.958979 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.957829    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/af873b12-c8cd-4c48-a016-0be5e88e9894-lib-modules\") pod \"tuned-x72v8\" (UID: \"af873b12-c8cd-4c48-a016-0be5e88e9894\") " pod="openshift-cluster-node-tuning-operator/tuned-x72v8"
Sep 08 01:58:50.958979 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.957942    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/af873b12-c8cd-4c48-a016-0be5e88e9894-etc-tuned\") pod \"tuned-x72v8\" (UID: \"af873b12-c8cd-4c48-a016-0be5e88e9894\") " pod="openshift-cluster-node-tuning-operator/tuned-x72v8"
Sep 08 01:58:50.958979 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.957969    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2d7adf2a-2cfe-40ff-90d3-4886a747463a-system-cni-dir\") pod \"multus-additional-cni-plugins-b4ms7\" (UID: \"2d7adf2a-2cfe-40ff-90d3-4886a747463a\") " pod="openshift-multus/multus-additional-cni-plugins-b4ms7"
Sep 08 01:58:50.958979 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.957976    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/af873b12-c8cd-4c48-a016-0be5e88e9894-lib-modules\") pod \"tuned-x72v8\" (UID: \"af873b12-c8cd-4c48-a016-0be5e88e9894\") " pod="openshift-cluster-node-tuning-operator/tuned-x72v8"
Sep 08 01:58:50.958979 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.958012    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/917f6c17-7902-48d3-a0f4-535e5b602d26-original-pull-secret\") pod \"global-pull-secret-syncer-l7hw8\" (UID: \"917f6c17-7902-48d3-a0f4-535e5b602d26\") " pod="kube-system/global-pull-secret-syncer-l7hw8"
Sep 08 01:58:50.958979 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.958057    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2d7adf2a-2cfe-40ff-90d3-4886a747463a-system-cni-dir\") pod \"multus-additional-cni-plugins-b4ms7\" (UID: \"2d7adf2a-2cfe-40ff-90d3-4886a747463a\") " pod="openshift-multus/multus-additional-cni-plugins-b4ms7"
Sep 08 01:58:50.958979 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.958048    2547 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 08 01:58:50.958979 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.958116    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/275d1493-2484-4412-b30d-b2aa36723bef-var-lib-openvswitch\") pod \"ovnkube-node-xskxl\" (UID: \"275d1493-2484-4412-b30d-b2aa36723bef\") " pod="openshift-ovn-kubernetes/ovnkube-node-xskxl"
Sep 08 01:58:50.958979 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.958153    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/44506a48-1a0a-4143-8ced-0dea3455ad2d-registration-dir\") pod \"aws-ebs-csi-driver-node-lbtkf\" (UID: \"44506a48-1a0a-4143-8ced-0dea3455ad2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lbtkf"
Sep 08 01:58:50.958979 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.958189    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/275d1493-2484-4412-b30d-b2aa36723bef-host-cni-netd\") pod \"ovnkube-node-xskxl\" (UID: \"275d1493-2484-4412-b30d-b2aa36723bef\") " pod="openshift-ovn-kubernetes/ovnkube-node-xskxl"
Sep 08 01:58:50.959890 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.958233    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/44506a48-1a0a-4143-8ced-0dea3455ad2d-etc-selinux\") pod \"aws-ebs-csi-driver-node-lbtkf\" (UID: \"44506a48-1a0a-4143-8ced-0dea3455ad2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lbtkf"
Sep 08 01:58:50.959890 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.958269    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2d7adf2a-2cfe-40ff-90d3-4886a747463a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-b4ms7\" (UID: \"2d7adf2a-2cfe-40ff-90d3-4886a747463a\") " pod="openshift-multus/multus-additional-cni-plugins-b4ms7"
Sep 08 01:58:50.959890 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.958279    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/44506a48-1a0a-4143-8ced-0dea3455ad2d-etc-selinux\") pod \"aws-ebs-csi-driver-node-lbtkf\" (UID: \"44506a48-1a0a-4143-8ced-0dea3455ad2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lbtkf"
Sep 08 01:58:50.959890 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.958280    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0ce55233-87a7-4f6d-88a0-79d5d8eb2e3c-host-var-lib-cni-multus\") pod \"multus-7wjjb\" (UID: \"0ce55233-87a7-4f6d-88a0-79d5d8eb2e3c\") " pod="openshift-multus/multus-7wjjb"
Sep 08 01:58:50.959890 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.958322    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0ce55233-87a7-4f6d-88a0-79d5d8eb2e3c-host-var-lib-cni-multus\") pod \"multus-7wjjb\" (UID: \"0ce55233-87a7-4f6d-88a0-79d5d8eb2e3c\") " pod="openshift-multus/multus-7wjjb"
Sep 08 01:58:50.959890 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.958340    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0ce55233-87a7-4f6d-88a0-79d5d8eb2e3c-etc-kubernetes\") pod \"multus-7wjjb\" (UID: \"0ce55233-87a7-4f6d-88a0-79d5d8eb2e3c\") " pod="openshift-multus/multus-7wjjb"
Sep 08 01:58:50.959890 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.958380    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hq9x\" (UniqueName: \"kubernetes.io/projected/af873b12-c8cd-4c48-a016-0be5e88e9894-kube-api-access-2hq9x\") pod \"tuned-x72v8\" (UID: \"af873b12-c8cd-4c48-a016-0be5e88e9894\") " pod="openshift-cluster-node-tuning-operator/tuned-x72v8"
Sep 08 01:58:50.959890 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.958385    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0ce55233-87a7-4f6d-88a0-79d5d8eb2e3c-etc-kubernetes\") pod \"multus-7wjjb\" (UID: \"0ce55233-87a7-4f6d-88a0-79d5d8eb2e3c\") " pod="openshift-multus/multus-7wjjb"
Sep 08 01:58:50.959890 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.958411    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2d7adf2a-2cfe-40ff-90d3-4886a747463a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-b4ms7\" (UID: \"2d7adf2a-2cfe-40ff-90d3-4886a747463a\") " pod="openshift-multus/multus-additional-cni-plugins-b4ms7"
Sep 08 01:58:50.959890 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.958443    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/275d1493-2484-4412-b30d-b2aa36723bef-run-systemd\") pod \"ovnkube-node-xskxl\" (UID: \"275d1493-2484-4412-b30d-b2aa36723bef\") " pod="openshift-ovn-kubernetes/ovnkube-node-xskxl"
Sep 08 01:58:50.959890 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.958469    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/275d1493-2484-4412-b30d-b2aa36723bef-ovnkube-script-lib\") pod \"ovnkube-node-xskxl\" (UID: \"275d1493-2484-4412-b30d-b2aa36723bef\") " pod="openshift-ovn-kubernetes/ovnkube-node-xskxl"
Sep 08 01:58:50.959890 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.958518    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/af873b12-c8cd-4c48-a016-0be5e88e9894-etc-sysconfig\") pod \"tuned-x72v8\" (UID: \"af873b12-c8cd-4c48-a016-0be5e88e9894\") " pod="openshift-cluster-node-tuning-operator/tuned-x72v8"
Sep 08 01:58:50.959890 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.958550    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/275d1493-2484-4412-b30d-b2aa36723bef-host-run-ovn-kubernetes\") pod \"ovnkube-node-xskxl\" (UID: \"275d1493-2484-4412-b30d-b2aa36723bef\") " pod="openshift-ovn-kubernetes/ovnkube-node-xskxl"
Sep 08 01:58:50.959890 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.958578    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qbcg\" (UniqueName: \"kubernetes.io/projected/275d1493-2484-4412-b30d-b2aa36723bef-kube-api-access-9qbcg\") pod \"ovnkube-node-xskxl\" (UID: \"275d1493-2484-4412-b30d-b2aa36723bef\") " pod="openshift-ovn-kubernetes/ovnkube-node-xskxl"
Sep 08 01:58:50.959890 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.958603    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/af873b12-c8cd-4c48-a016-0be5e88e9894-etc-sysconfig\") pod \"tuned-x72v8\" (UID: \"af873b12-c8cd-4c48-a016-0be5e88e9894\") " pod="openshift-cluster-node-tuning-operator/tuned-x72v8"
Sep 08 01:58:50.959890 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.958604    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0536c61b-c71c-41d9-8b2c-18228a7ddff4-tmp-dir\") pod \"node-resolver-4s9cm\" (UID: \"0536c61b-c71c-41d9-8b2c-18228a7ddff4\") " pod="openshift-dns/node-resolver-4s9cm"
Sep 08 01:58:50.959890 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.958653    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bbpj\" (UniqueName: \"kubernetes.io/projected/ae4a55d0-ad38-48a9-9a36-279949e2e4c5-kube-api-access-7bbpj\") pod \"node-ca-pl5qh\" (UID: \"ae4a55d0-ad38-48a9-9a36-279949e2e4c5\") " pod="openshift-image-registry/node-ca-pl5qh"
Sep 08 01:58:50.960756 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.958685    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c793f939-4584-46af-a495-36b57bbdff41-konnectivity-ca\") pod \"konnectivity-agent-xr9c8\" (UID: \"c793f939-4584-46af-a495-36b57bbdff41\") " pod="kube-system/konnectivity-agent-xr9c8"
Sep 08 01:58:50.960756 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.958714    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/44506a48-1a0a-4143-8ced-0dea3455ad2d-device-dir\") pod \"aws-ebs-csi-driver-node-lbtkf\" (UID: \"44506a48-1a0a-4143-8ced-0dea3455ad2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lbtkf"
Sep 08 01:58:50.960756 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.958739    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/af873b12-c8cd-4c48-a016-0be5e88e9894-etc-sysctl-conf\") pod \"tuned-x72v8\" (UID: \"af873b12-c8cd-4c48-a016-0be5e88e9894\") " pod="openshift-cluster-node-tuning-operator/tuned-x72v8"
Sep 08 01:58:50.960756 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.958794    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/44506a48-1a0a-4143-8ced-0dea3455ad2d-device-dir\") pod \"aws-ebs-csi-driver-node-lbtkf\" (UID: \"44506a48-1a0a-4143-8ced-0dea3455ad2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lbtkf"
Sep 08 01:58:50.960756 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.958827    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/af873b12-c8cd-4c48-a016-0be5e88e9894-run\") pod \"tuned-x72v8\" (UID: \"af873b12-c8cd-4c48-a016-0be5e88e9894\") " pod="openshift-cluster-node-tuning-operator/tuned-x72v8"
Sep 08 01:58:50.960756 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.958886    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2d7adf2a-2cfe-40ff-90d3-4886a747463a-cni-binary-copy\") pod \"multus-additional-cni-plugins-b4ms7\" (UID: \"2d7adf2a-2cfe-40ff-90d3-4886a747463a\") " pod="openshift-multus/multus-additional-cni-plugins-b4ms7"
Sep 08 01:58:50.960756 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.958907    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/af873b12-c8cd-4c48-a016-0be5e88e9894-etc-sysctl-conf\") pod \"tuned-x72v8\" (UID: \"af873b12-c8cd-4c48-a016-0be5e88e9894\") " pod="openshift-cluster-node-tuning-operator/tuned-x72v8"
Sep 08 01:58:50.960756 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.958917    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/275d1493-2484-4412-b30d-b2aa36723bef-systemd-units\") pod \"ovnkube-node-xskxl\" (UID: \"275d1493-2484-4412-b30d-b2aa36723bef\") " pod="openshift-ovn-kubernetes/ovnkube-node-xskxl"
Sep 08 01:58:50.960756 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.958934    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2d7adf2a-2cfe-40ff-90d3-4886a747463a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-b4ms7\" (UID: \"2d7adf2a-2cfe-40ff-90d3-4886a747463a\") " pod="openshift-multus/multus-additional-cni-plugins-b4ms7"
Sep 08 01:58:50.960756 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.958957    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/275d1493-2484-4412-b30d-b2aa36723bef-host-run-netns\") pod \"ovnkube-node-xskxl\" (UID: \"275d1493-2484-4412-b30d-b2aa36723bef\") " pod="openshift-ovn-kubernetes/ovnkube-node-xskxl"
Sep 08 01:58:50.960756 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.959013    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/44506a48-1a0a-4143-8ced-0dea3455ad2d-socket-dir\") pod \"aws-ebs-csi-driver-node-lbtkf\" (UID: \"44506a48-1a0a-4143-8ced-0dea3455ad2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lbtkf"
Sep 08 01:58:50.960756 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.959061    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0ce55233-87a7-4f6d-88a0-79d5d8eb2e3c-multus-daemon-config\") pod \"multus-7wjjb\" (UID: \"0ce55233-87a7-4f6d-88a0-79d5d8eb2e3c\") " pod="openshift-multus/multus-7wjjb"
Sep 08 01:58:50.960756 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.959070    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvwpz\" (UniqueName: \"kubernetes.io/projected/0ce55233-87a7-4f6d-88a0-79d5d8eb2e3c-kube-api-access-tvwpz\") pod \"multus-7wjjb\" (UID: \"0ce55233-87a7-4f6d-88a0-79d5d8eb2e3c\") " pod="openshift-multus/multus-7wjjb"
Sep 08 01:58:50.960756 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.959079    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/af873b12-c8cd-4c48-a016-0be5e88e9894-run\") pod \"tuned-x72v8\" (UID: \"af873b12-c8cd-4c48-a016-0be5e88e9894\") " pod="openshift-cluster-node-tuning-operator/tuned-x72v8"
Sep 08 01:58:50.960756 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.959103    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlnz5\" (UniqueName: \"kubernetes.io/projected/0536c61b-c71c-41d9-8b2c-18228a7ddff4-kube-api-access-hlnz5\") pod \"node-resolver-4s9cm\" (UID: \"0536c61b-c71c-41d9-8b2c-18228a7ddff4\") " pod="openshift-dns/node-resolver-4s9cm"
Sep 08 01:58:50.960756 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.959131    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/917f6c17-7902-48d3-a0f4-535e5b602d26-dbus\") pod \"global-pull-secret-syncer-l7hw8\" (UID: \"917f6c17-7902-48d3-a0f4-535e5b602d26\") " pod="kube-system/global-pull-secret-syncer-l7hw8"
Sep 08 01:58:50.960756 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.959162    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/44506a48-1a0a-4143-8ced-0dea3455ad2d-socket-dir\") pod \"aws-ebs-csi-driver-node-lbtkf\" (UID: \"44506a48-1a0a-4143-8ced-0dea3455ad2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lbtkf"
Sep 08 01:58:50.961619 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.959210    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/275d1493-2484-4412-b30d-b2aa36723bef-host-kubelet\") pod \"ovnkube-node-xskxl\" (UID: \"275d1493-2484-4412-b30d-b2aa36723bef\") " pod="openshift-ovn-kubernetes/ovnkube-node-xskxl"
Sep 08 01:58:50.961619 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.959242    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0ce55233-87a7-4f6d-88a0-79d5d8eb2e3c-host-var-lib-kubelet\") pod \"multus-7wjjb\" (UID: \"0ce55233-87a7-4f6d-88a0-79d5d8eb2e3c\") " pod="openshift-multus/multus-7wjjb"
Sep 08 01:58:50.961619 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.959269    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2d7adf2a-2cfe-40ff-90d3-4886a747463a-cnibin\") pod \"multus-additional-cni-plugins-b4ms7\" (UID: \"2d7adf2a-2cfe-40ff-90d3-4886a747463a\") " pod="openshift-multus/multus-additional-cni-plugins-b4ms7"
Sep 08 01:58:50.961619 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.959282    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0ce55233-87a7-4f6d-88a0-79d5d8eb2e3c-host-var-lib-kubelet\") pod \"multus-7wjjb\" (UID: \"0ce55233-87a7-4f6d-88a0-79d5d8eb2e3c\") " pod="openshift-multus/multus-7wjjb"
Sep 08 01:58:50.961619 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.959296    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b82c5af2-4c1c-41a9-8fa4-af8a85bcc8c6-metrics-certs\") pod \"network-metrics-daemon-jr45c\" (UID: \"b82c5af2-4c1c-41a9-8fa4-af8a85bcc8c6\") " pod="openshift-multus/network-metrics-daemon-jr45c"
Sep 08 01:58:50.961619 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.959346    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/275d1493-2484-4412-b30d-b2aa36723bef-node-log\") pod \"ovnkube-node-xskxl\" (UID: \"275d1493-2484-4412-b30d-b2aa36723bef\") " pod="openshift-ovn-kubernetes/ovnkube-node-xskxl"
Sep 08 01:58:50.961619 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.959355    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2d7adf2a-2cfe-40ff-90d3-4886a747463a-cnibin\") pod \"multus-additional-cni-plugins-b4ms7\" (UID: \"2d7adf2a-2cfe-40ff-90d3-4886a747463a\") " pod="openshift-multus/multus-additional-cni-plugins-b4ms7"
Sep 08 01:58:50.961619 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.959375    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/275d1493-2484-4412-b30d-b2aa36723bef-run-ovn\") pod \"ovnkube-node-xskxl\" (UID: \"275d1493-2484-4412-b30d-b2aa36723bef\") " pod="openshift-ovn-kubernetes/ovnkube-node-xskxl"
Sep 08 01:58:50.961619 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.959402    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2d7adf2a-2cfe-40ff-90d3-4886a747463a-cni-binary-copy\") pod \"multus-additional-cni-plugins-b4ms7\" (UID: \"2d7adf2a-2cfe-40ff-90d3-4886a747463a\") " pod="openshift-multus/multus-additional-cni-plugins-b4ms7"
Sep 08 01:58:50.961619 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.959412    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0ce55233-87a7-4f6d-88a0-79d5d8eb2e3c-multus-cni-dir\") pod \"multus-7wjjb\" (UID: \"0ce55233-87a7-4f6d-88a0-79d5d8eb2e3c\") " pod="openshift-multus/multus-7wjjb"
Sep 08 01:58:50.961619 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.959441    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/44506a48-1a0a-4143-8ced-0dea3455ad2d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lbtkf\" (UID: \"44506a48-1a0a-4143-8ced-0dea3455ad2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lbtkf"
Sep 08 01:58:50.961619 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.959468    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0ce55233-87a7-4f6d-88a0-79d5d8eb2e3c-host-run-k8s-cni-cncf-io\") pod \"multus-7wjjb\" (UID: \"0ce55233-87a7-4f6d-88a0-79d5d8eb2e3c\") " pod="openshift-multus/multus-7wjjb"
Sep 08 01:58:50.961619 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.959491    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/275d1493-2484-4412-b30d-b2aa36723bef-host-slash\") pod \"ovnkube-node-xskxl\" (UID: \"275d1493-2484-4412-b30d-b2aa36723bef\") " pod="openshift-ovn-kubernetes/ovnkube-node-xskxl"
Sep 08 01:58:50.961619 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.959529    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0ce55233-87a7-4f6d-88a0-79d5d8eb2e3c-host-run-k8s-cni-cncf-io\") pod \"multus-7wjjb\" (UID: \"0ce55233-87a7-4f6d-88a0-79d5d8eb2e3c\") " pod="openshift-multus/multus-7wjjb"
Sep 08 01:58:50.961619 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.959490    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/44506a48-1a0a-4143-8ced-0dea3455ad2d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lbtkf\" (UID: \"44506a48-1a0a-4143-8ced-0dea3455ad2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lbtkf"
Sep 08 01:58:50.961619 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.959491    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0ce55233-87a7-4f6d-88a0-79d5d8eb2e3c-multus-cni-dir\") pod \"multus-7wjjb\" (UID: \"0ce55233-87a7-4f6d-88a0-79d5d8eb2e3c\") " pod="openshift-multus/multus-7wjjb"
Sep 08 01:58:50.963521 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.963498    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/af873b12-c8cd-4c48-a016-0be5e88e9894-tmp\") pod \"tuned-x72v8\" (UID: \"af873b12-c8cd-4c48-a016-0be5e88e9894\") " pod="openshift-cluster-node-tuning-operator/tuned-x72v8"
Sep 08 01:58:50.965239 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.965022    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/af873b12-c8cd-4c48-a016-0be5e88e9894-etc-tuned\") pod \"tuned-x72v8\" (UID: \"af873b12-c8cd-4c48-a016-0be5e88e9894\") " pod="openshift-cluster-node-tuning-operator/tuned-x72v8"
Sep 08 01:58:50.965946 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.965921    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbdmc\" (UniqueName: \"kubernetes.io/projected/44506a48-1a0a-4143-8ced-0dea3455ad2d-kube-api-access-hbdmc\") pod \"aws-ebs-csi-driver-node-lbtkf\" (UID: \"44506a48-1a0a-4143-8ced-0dea3455ad2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lbtkf"
Sep 08 01:58:50.967336 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.967308    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvrnj\" (UniqueName: \"kubernetes.io/projected/2d7adf2a-2cfe-40ff-90d3-4886a747463a-kube-api-access-pvrnj\") pod \"multus-additional-cni-plugins-b4ms7\" (UID: \"2d7adf2a-2cfe-40ff-90d3-4886a747463a\") " pod="openshift-multus/multus-additional-cni-plugins-b4ms7"
Sep 08 01:58:50.969765 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.969741    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvwpz\" (UniqueName: \"kubernetes.io/projected/0ce55233-87a7-4f6d-88a0-79d5d8eb2e3c-kube-api-access-tvwpz\") pod \"multus-7wjjb\" (UID: \"0ce55233-87a7-4f6d-88a0-79d5d8eb2e3c\") " pod="openshift-multus/multus-7wjjb"
Sep 08 01:58:50.970978 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.970950    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hq9x\" (UniqueName: \"kubernetes.io/projected/af873b12-c8cd-4c48-a016-0be5e88e9894-kube-api-access-2hq9x\") pod \"tuned-x72v8\" (UID: \"af873b12-c8cd-4c48-a016-0be5e88e9894\") " pod="openshift-cluster-node-tuning-operator/tuned-x72v8"
Sep 08 01:58:50.984381 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.984353    2547 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 01:58:50.997436 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.997378    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-78.ec2.internal" event={"ID":"5cfe1f564aebe22c530741e9b588e31d","Type":"ContainerStarted","Data":"95808e3829d4141c2b64cf6ce26f49da043b5b388dbefca07488502693f258a6"}
Sep 08 01:58:50.998836 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:50.998806    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-78.ec2.internal" event={"ID":"3d670938dd837b2cf06ee8ff34788dde","Type":"ContainerStarted","Data":"33ca4536c129feefe30e11a16a11e24a01f0abf7b5fa10390c0dd9956bfe3bfe"}
Sep 08 01:58:51.060247 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:51.060199    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/275d1493-2484-4412-b30d-b2aa36723bef-ovnkube-config\") pod \"ovnkube-node-xskxl\" (UID: \"275d1493-2484-4412-b30d-b2aa36723bef\") " pod="openshift-ovn-kubernetes/ovnkube-node-xskxl"
Sep 08 01:58:51.060437 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:51.060256    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/275d1493-2484-4412-b30d-b2aa36723bef-ovn-node-metrics-cert\") pod \"ovnkube-node-xskxl\" (UID: \"275d1493-2484-4412-b30d-b2aa36723bef\") " pod="openshift-ovn-kubernetes/ovnkube-node-xskxl"
Sep 08 01:58:51.060437 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:51.060287    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ae4a55d0-ad38-48a9-9a36-279949e2e4c5-serviceca\") pod \"node-ca-pl5qh\" (UID: \"ae4a55d0-ad38-48a9-9a36-279949e2e4c5\") " pod="openshift-image-registry/node-ca-pl5qh"
Sep 08 01:58:51.060437 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:51.060312    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0536c61b-c71c-41d9-8b2c-18228a7ddff4-hosts-file\") pod \"node-resolver-4s9cm\" (UID: \"0536c61b-c71c-41d9-8b2c-18228a7ddff4\") " pod="openshift-dns/node-resolver-4s9cm"
Sep 08 01:58:51.060437 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:51.060338    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c793f939-4584-46af-a495-36b57bbdff41-agent-certs\") pod \"konnectivity-agent-xr9c8\" (UID: \"c793f939-4584-46af-a495-36b57bbdff41\") " pod="kube-system/konnectivity-agent-xr9c8"
Sep 08 01:58:51.060437 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:51.060367    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/275d1493-2484-4412-b30d-b2aa36723bef-etc-openvswitch\") pod \"ovnkube-node-xskxl\" (UID: \"275d1493-2484-4412-b30d-b2aa36723bef\") " pod="openshift-ovn-kubernetes/ovnkube-node-xskxl"
Sep 08 01:58:51.060437 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:51.060391    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ae4a55d0-ad38-48a9-9a36-279949e2e4c5-host\") pod \"node-ca-pl5qh\" (UID: \"ae4a55d0-ad38-48a9-9a36-279949e2e4c5\") " pod="openshift-image-registry/node-ca-pl5qh"
Sep 08 01:58:51.060437 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:51.060424    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/917f6c17-7902-48d3-a0f4-535e5b602d26-original-pull-secret\") pod \"global-pull-secret-syncer-l7hw8\" (UID: \"917f6c17-7902-48d3-a0f4-535e5b602d26\") " pod="kube-system/global-pull-secret-syncer-l7hw8"
Sep 08 01:58:51.060792 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:51.060451    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/275d1493-2484-4412-b30d-b2aa36723bef-var-lib-openvswitch\") pod \"ovnkube-node-xskxl\" (UID: \"275d1493-2484-4412-b30d-b2aa36723bef\") " pod="openshift-ovn-kubernetes/ovnkube-node-xskxl"
Sep 08 01:58:51.060792 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:51.060478    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/275d1493-2484-4412-b30d-b2aa36723bef-host-cni-netd\") pod \"ovnkube-node-xskxl\" (UID: \"275d1493-2484-4412-b30d-b2aa36723bef\") " pod="openshift-ovn-kubernetes/ovnkube-node-xskxl"
Sep 08 01:58:51.060792 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:51.060512    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/275d1493-2484-4412-b30d-b2aa36723bef-run-systemd\") pod \"ovnkube-node-xskxl\" (UID: \"275d1493-2484-4412-b30d-b2aa36723bef\") " pod="openshift-ovn-kubernetes/ovnkube-node-xskxl"
Sep 08 01:58:51.060792 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:51.060537    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/275d1493-2484-4412-b30d-b2aa36723bef-ovnkube-script-lib\") pod \"ovnkube-node-xskxl\" (UID: \"275d1493-2484-4412-b30d-b2aa36723bef\") " pod="openshift-ovn-kubernetes/ovnkube-node-xskxl"
Sep 08 01:58:51.060792 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:51.060566    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/275d1493-2484-4412-b30d-b2aa36723bef-host-run-ovn-kubernetes\") pod \"ovnkube-node-xskxl\" (UID: \"275d1493-2484-4412-b30d-b2aa36723bef\") " pod="openshift-ovn-kubernetes/ovnkube-node-xskxl"
Sep 08 01:58:51.060792 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:51.060592    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qbcg\" (UniqueName: \"kubernetes.io/projected/275d1493-2484-4412-b30d-b2aa36723bef-kube-api-access-9qbcg\") pod \"ovnkube-node-xskxl\" (UID: \"275d1493-2484-4412-b30d-b2aa36723bef\") " pod="openshift-ovn-kubernetes/ovnkube-node-xskxl"
Sep 08 01:58:51.060792 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:51.060620    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0536c61b-c71c-41d9-8b2c-18228a7ddff4-tmp-dir\") pod \"node-resolver-4s9cm\" (UID: \"0536c61b-c71c-41d9-8b2c-18228a7ddff4\") " pod="openshift-dns/node-resolver-4s9cm"
Sep 08 01:58:51.060792 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:51.060646    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bbpj\" (UniqueName: \"kubernetes.io/projected/ae4a55d0-ad38-48a9-9a36-279949e2e4c5-kube-api-access-7bbpj\") pod \"node-ca-pl5qh\" (UID: \"ae4a55d0-ad38-48a9-9a36-279949e2e4c5\") " pod="openshift-image-registry/node-ca-pl5qh"
Sep 08 01:58:51.060792 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:51.060671    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c793f939-4584-46af-a495-36b57bbdff41-konnectivity-ca\") pod \"konnectivity-agent-xr9c8\" (UID: \"c793f939-4584-46af-a495-36b57bbdff41\") " pod="kube-system/konnectivity-agent-xr9c8"
Sep 08 01:58:51.060792 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:51.060700    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/275d1493-2484-4412-b30d-b2aa36723bef-systemd-units\") pod \"ovnkube-node-xskxl\" (UID: \"275d1493-2484-4412-b30d-b2aa36723bef\") " pod="openshift-ovn-kubernetes/ovnkube-node-xskxl"
Sep 08 01:58:51.060792 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:51.060724    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/275d1493-2484-4412-b30d-b2aa36723bef-host-run-netns\") pod \"ovnkube-node-xskxl\" (UID: \"275d1493-2484-4412-b30d-b2aa36723bef\") " pod="openshift-ovn-kubernetes/ovnkube-node-xskxl"
Sep 08 01:58:51.060792 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:51.060752    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlnz5\" (UniqueName: \"kubernetes.io/projected/0536c61b-c71c-41d9-8b2c-18228a7ddff4-kube-api-access-hlnz5\") pod \"node-resolver-4s9cm\" (UID: \"0536c61b-c71c-41d9-8b2c-18228a7ddff4\") " pod="openshift-dns/node-resolver-4s9cm"
Sep 08 01:58:51.060792 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:51.060783    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/917f6c17-7902-48d3-a0f4-535e5b602d26-dbus\") pod \"global-pull-secret-syncer-l7hw8\" (UID: \"917f6c17-7902-48d3-a0f4-535e5b602d26\") " pod="kube-system/global-pull-secret-syncer-l7hw8"
Sep 08 01:58:51.061807 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:51.060795    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/275d1493-2484-4412-b30d-b2aa36723bef-host-cni-netd\") pod \"ovnkube-node-xskxl\" (UID: \"275d1493-2484-4412-b30d-b2aa36723bef\") " pod="openshift-ovn-kubernetes/ovnkube-node-xskxl"
Sep 08 01:58:51.061807 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:51.060869    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/275d1493-2484-4412-b30d-b2aa36723bef-run-systemd\") pod \"ovnkube-node-xskxl\" (UID: \"275d1493-2484-4412-b30d-b2aa36723bef\") " pod="openshift-ovn-kubernetes/ovnkube-node-xskxl"
Sep 08 01:58:51.061807 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:51.060869    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/275d1493-2484-4412-b30d-b2aa36723bef-ovnkube-config\") pod \"ovnkube-node-xskxl\" (UID: \"275d1493-2484-4412-b30d-b2aa36723bef\") " pod="openshift-ovn-kubernetes/ovnkube-node-xskxl"
Sep 08 01:58:51.061807 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:51.060917    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ae4a55d0-ad38-48a9-9a36-279949e2e4c5-host\") pod \"node-ca-pl5qh\" (UID: \"ae4a55d0-ad38-48a9-9a36-279949e2e4c5\") " pod="openshift-image-registry/node-ca-pl5qh"
Sep 08 01:58:51.061807 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:51.060936    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/275d1493-2484-4412-b30d-b2aa36723bef-var-lib-openvswitch\") pod \"ovnkube-node-xskxl\" (UID: \"275d1493-2484-4412-b30d-b2aa36723bef\") " pod="openshift-ovn-kubernetes/ovnkube-node-xskxl"
Sep 08 01:58:51.061807 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:51.060811    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/275d1493-2484-4412-b30d-b2aa36723bef-host-kubelet\") pod \"ovnkube-node-xskxl\" (UID: \"275d1493-2484-4412-b30d-b2aa36723bef\") " pod="openshift-ovn-kubernetes/ovnkube-node-xskxl"
Sep 08 01:58:51.061807 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:58:51.060976    2547 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 01:58:51.061807 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:51.060985    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b82c5af2-4c1c-41a9-8fa4-af8a85bcc8c6-metrics-certs\") pod \"network-metrics-daemon-jr45c\" (UID: \"b82c5af2-4c1c-41a9-8fa4-af8a85bcc8c6\") " pod="openshift-multus/network-metrics-daemon-jr45c"
Sep 08 01:58:51.061807 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:51.061011    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/275d1493-2484-4412-b30d-b2aa36723bef-node-log\") pod \"ovnkube-node-xskxl\" (UID: \"275d1493-2484-4412-b30d-b2aa36723bef\") " pod="openshift-ovn-kubernetes/ovnkube-node-xskxl"
Sep 08 01:58:51.061807 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:58:51.061046    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/917f6c17-7902-48d3-a0f4-535e5b602d26-original-pull-secret podName:917f6c17-7902-48d3-a0f4-535e5b602d26 nodeName:}" failed. No retries permitted until 2026-09-08 01:58:51.561024133 +0000 UTC m=+3.084799217 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/917f6c17-7902-48d3-a0f4-535e5b602d26-original-pull-secret") pod "global-pull-secret-syncer-l7hw8" (UID: "917f6c17-7902-48d3-a0f4-535e5b602d26") : object "kube-system"/"original-pull-secret" not registered
Sep 08 01:58:51.061807 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:51.061052    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/275d1493-2484-4412-b30d-b2aa36723bef-node-log\") pod \"ovnkube-node-xskxl\" (UID: \"275d1493-2484-4412-b30d-b2aa36723bef\") " pod="openshift-ovn-kubernetes/ovnkube-node-xskxl"
Sep 08 01:58:51.061807 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:51.061069    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/275d1493-2484-4412-b30d-b2aa36723bef-run-ovn\") pod \"ovnkube-node-xskxl\" (UID: \"275d1493-2484-4412-b30d-b2aa36723bef\") " pod="openshift-ovn-kubernetes/ovnkube-node-xskxl"
Sep 08 01:58:51.061807 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:51.061099    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/275d1493-2484-4412-b30d-b2aa36723bef-host-slash\") pod \"ovnkube-node-xskxl\" (UID: \"275d1493-2484-4412-b30d-b2aa36723bef\") " pod="openshift-ovn-kubernetes/ovnkube-node-xskxl"
Sep 08 01:58:51.061807 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:51.061127    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4b89e1b1-bcc0-4027-b778-19a7aba264a9-iptables-alerter-script\") pod \"iptables-alerter-b7dzq\" (UID: \"4b89e1b1-bcc0-4027-b778-19a7aba264a9\") " pod="openshift-network-operator/iptables-alerter-b7dzq"
Sep 08 01:58:51.061807 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:58:51.061135    2547 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 01:58:51.061807 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:51.061156    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dt4k\" (UniqueName: \"kubernetes.io/projected/430aad96-9d26-4055-8133-b7f50b0f2978-kube-api-access-5dt4k\") pod \"network-check-target-2v24n\" (UID: \"430aad96-9d26-4055-8133-b7f50b0f2978\") " pod="openshift-network-diagnostics/network-check-target-2v24n"
Sep 08 01:58:51.061807 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:51.061221    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/275d1493-2484-4412-b30d-b2aa36723bef-log-socket\") pod \"ovnkube-node-xskxl\" (UID: \"275d1493-2484-4412-b30d-b2aa36723bef\") " pod="openshift-ovn-kubernetes/ovnkube-node-xskxl"
Sep 08 01:58:51.061807 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:51.061265    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/275d1493-2484-4412-b30d-b2aa36723bef-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xskxl\" (UID: \"275d1493-2484-4412-b30d-b2aa36723bef\") " pod="openshift-ovn-kubernetes/ovnkube-node-xskxl"
Sep 08 01:58:51.062828 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:51.061292    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4b89e1b1-bcc0-4027-b778-19a7aba264a9-host-slash\") pod \"iptables-alerter-b7dzq\" (UID: \"4b89e1b1-bcc0-4027-b778-19a7aba264a9\") " pod="openshift-network-operator/iptables-alerter-b7dzq"
Sep 08 01:58:51.062828 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:51.061317    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/275d1493-2484-4412-b30d-b2aa36723bef-run-openvswitch\") pod \"ovnkube-node-xskxl\" (UID: \"275d1493-2484-4412-b30d-b2aa36723bef\") " pod="openshift-ovn-kubernetes/ovnkube-node-xskxl"
Sep 08 01:58:51.062828 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:51.061344    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/275d1493-2484-4412-b30d-b2aa36723bef-env-overrides\") pod \"ovnkube-node-xskxl\" (UID: \"275d1493-2484-4412-b30d-b2aa36723bef\") " pod="openshift-ovn-kubernetes/ovnkube-node-xskxl"
Sep 08 01:58:51.062828 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:51.061351    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/275d1493-2484-4412-b30d-b2aa36723bef-host-slash\") pod \"ovnkube-node-xskxl\" (UID: \"275d1493-2484-4412-b30d-b2aa36723bef\") " pod="openshift-ovn-kubernetes/ovnkube-node-xskxl"
Sep 08 01:58:51.062828 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:51.061373    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/917f6c17-7902-48d3-a0f4-535e5b602d26-kubelet-config\") pod \"global-pull-secret-syncer-l7hw8\" (UID: \"917f6c17-7902-48d3-a0f4-535e5b602d26\") " pod="kube-system/global-pull-secret-syncer-l7hw8"
Sep 08 01:58:51.062828 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:51.061400    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7bs2\" (UniqueName: \"kubernetes.io/projected/b82c5af2-4c1c-41a9-8fa4-af8a85bcc8c6-kube-api-access-n7bs2\") pod \"network-metrics-daemon-jr45c\" (UID: \"b82c5af2-4c1c-41a9-8fa4-af8a85bcc8c6\") " pod="openshift-multus/network-metrics-daemon-jr45c"
Sep 08 01:58:51.062828 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:51.061450    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/275d1493-2484-4412-b30d-b2aa36723bef-host-cni-bin\") pod \"ovnkube-node-xskxl\" (UID: \"275d1493-2484-4412-b30d-b2aa36723bef\") " pod="openshift-ovn-kubernetes/ovnkube-node-xskxl"
Sep 08 01:58:51.062828 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:51.061476    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/275d1493-2484-4412-b30d-b2aa36723bef-run-ovn\") pod \"ovnkube-node-xskxl\" (UID: \"275d1493-2484-4412-b30d-b2aa36723bef\") " pod="openshift-ovn-kubernetes/ovnkube-node-xskxl"
Sep 08 01:58:51.062828 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:51.061477    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwg6c\" (UniqueName: \"kubernetes.io/projected/4b89e1b1-bcc0-4027-b778-19a7aba264a9-kube-api-access-hwg6c\") pod \"iptables-alerter-b7dzq\" (UID: \"4b89e1b1-bcc0-4027-b778-19a7aba264a9\") " pod="openshift-network-operator/iptables-alerter-b7dzq"
Sep 08 01:58:51.062828 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:51.061516    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/275d1493-2484-4412-b30d-b2aa36723bef-ovnkube-script-lib\") pod \"ovnkube-node-xskxl\" (UID: \"275d1493-2484-4412-b30d-b2aa36723bef\") " pod="openshift-ovn-kubernetes/ovnkube-node-xskxl"
Sep 08 01:58:51.062828 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:51.061582    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/275d1493-2484-4412-b30d-b2aa36723bef-systemd-units\") pod \"ovnkube-node-xskxl\" (UID: \"275d1493-2484-4412-b30d-b2aa36723bef\") " pod="openshift-ovn-kubernetes/ovnkube-node-xskxl"
Sep 08 01:58:51.062828 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:51.061629    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/275d1493-2484-4412-b30d-b2aa36723bef-host-run-netns\") pod \"ovnkube-node-xskxl\" (UID: \"275d1493-2484-4412-b30d-b2aa36723bef\") " pod="openshift-ovn-kubernetes/ovnkube-node-xskxl"
Sep 08 01:58:51.062828 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:51.061739    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/275d1493-2484-4412-b30d-b2aa36723bef-log-socket\") pod \"ovnkube-node-xskxl\" (UID: \"275d1493-2484-4412-b30d-b2aa36723bef\") " pod="openshift-ovn-kubernetes/ovnkube-node-xskxl"
Sep 08 01:58:51.062828 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:58:51.061782    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b82c5af2-4c1c-41a9-8fa4-af8a85bcc8c6-metrics-certs podName:b82c5af2-4c1c-41a9-8fa4-af8a85bcc8c6 nodeName:}" failed. No retries permitted until 2026-09-08 01:58:51.561763563 +0000 UTC m=+3.085538870 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b82c5af2-4c1c-41a9-8fa4-af8a85bcc8c6-metrics-certs") pod "network-metrics-daemon-jr45c" (UID: "b82c5af2-4c1c-41a9-8fa4-af8a85bcc8c6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 01:58:51.062828 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:51.061817    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/275d1493-2484-4412-b30d-b2aa36723bef-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xskxl\" (UID: \"275d1493-2484-4412-b30d-b2aa36723bef\") " pod="openshift-ovn-kubernetes/ovnkube-node-xskxl"
Sep 08 01:58:51.062828 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:51.061858    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4b89e1b1-bcc0-4027-b778-19a7aba264a9-host-slash\") pod \"iptables-alerter-b7dzq\" (UID: \"4b89e1b1-bcc0-4027-b778-19a7aba264a9\") " pod="openshift-network-operator/iptables-alerter-b7dzq"
Sep 08 01:58:51.063691 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:51.061865    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4b89e1b1-bcc0-4027-b778-19a7aba264a9-iptables-alerter-script\") pod \"iptables-alerter-b7dzq\" (UID: \"4b89e1b1-bcc0-4027-b778-19a7aba264a9\") " pod="openshift-network-operator/iptables-alerter-b7dzq"
Sep 08 01:58:51.063691 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:51.061899    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/275d1493-2484-4412-b30d-b2aa36723bef-run-openvswitch\") pod \"ovnkube-node-xskxl\" (UID: \"275d1493-2484-4412-b30d-b2aa36723bef\") " pod="openshift-ovn-kubernetes/ovnkube-node-xskxl"
Sep 08 01:58:51.063691 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:51.061902    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/917f6c17-7902-48d3-a0f4-535e5b602d26-dbus\") pod \"global-pull-secret-syncer-l7hw8\" (UID: \"917f6c17-7902-48d3-a0f4-535e5b602d26\") " pod="kube-system/global-pull-secret-syncer-l7hw8"
Sep 08 01:58:51.063691 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:51.060872    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/275d1493-2484-4412-b30d-b2aa36723bef-host-kubelet\") pod \"ovnkube-node-xskxl\" (UID: \"275d1493-2484-4412-b30d-b2aa36723bef\") " pod="openshift-ovn-kubernetes/ovnkube-node-xskxl"
Sep 08 01:58:51.063691 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:51.061946    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0536c61b-c71c-41d9-8b2c-18228a7ddff4-hosts-file\") pod \"node-resolver-4s9cm\" (UID: \"0536c61b-c71c-41d9-8b2c-18228a7ddff4\") " pod="openshift-dns/node-resolver-4s9cm"
Sep 08 01:58:51.063691 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:51.061990    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/275d1493-2484-4412-b30d-b2aa36723bef-etc-openvswitch\") pod \"ovnkube-node-xskxl\" (UID: \"275d1493-2484-4412-b30d-b2aa36723bef\") " pod="openshift-ovn-kubernetes/ovnkube-node-xskxl"
Sep 08 01:58:51.063691 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:51.062035    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/917f6c17-7902-48d3-a0f4-535e5b602d26-kubelet-config\") pod \"global-pull-secret-syncer-l7hw8\" (UID: \"917f6c17-7902-48d3-a0f4-535e5b602d26\") " pod="kube-system/global-pull-secret-syncer-l7hw8"
Sep 08 01:58:51.063691 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:51.062122    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c793f939-4584-46af-a495-36b57bbdff41-konnectivity-ca\") pod \"konnectivity-agent-xr9c8\" (UID: \"c793f939-4584-46af-a495-36b57bbdff41\") " pod="kube-system/konnectivity-agent-xr9c8"
Sep 08 01:58:51.063691 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:51.062281    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/275d1493-2484-4412-b30d-b2aa36723bef-host-run-ovn-kubernetes\") pod \"ovnkube-node-xskxl\" (UID: \"275d1493-2484-4412-b30d-b2aa36723bef\") " pod="openshift-ovn-kubernetes/ovnkube-node-xskxl"
Sep 08 01:58:51.063691 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:51.062357    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/275d1493-2484-4412-b30d-b2aa36723bef-env-overrides\") pod \"ovnkube-node-xskxl\" (UID: \"275d1493-2484-4412-b30d-b2aa36723bef\") " pod="openshift-ovn-kubernetes/ovnkube-node-xskxl"
Sep 08 01:58:51.063691 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:51.062440    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0536c61b-c71c-41d9-8b2c-18228a7ddff4-tmp-dir\") pod \"node-resolver-4s9cm\" (UID: \"0536c61b-c71c-41d9-8b2c-18228a7ddff4\") " pod="openshift-dns/node-resolver-4s9cm"
Sep 08 01:58:51.063691 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:51.062494    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/275d1493-2484-4412-b30d-b2aa36723bef-host-cni-bin\") pod \"ovnkube-node-xskxl\" (UID: \"275d1493-2484-4412-b30d-b2aa36723bef\") " pod="openshift-ovn-kubernetes/ovnkube-node-xskxl"
Sep 08 01:58:51.063691 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:51.062667    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ae4a55d0-ad38-48a9-9a36-279949e2e4c5-serviceca\") pod \"node-ca-pl5qh\" (UID: \"ae4a55d0-ad38-48a9-9a36-279949e2e4c5\") " pod="openshift-image-registry/node-ca-pl5qh"
Sep 08 01:58:51.065323 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:51.065008    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c793f939-4584-46af-a495-36b57bbdff41-agent-certs\") pod \"konnectivity-agent-xr9c8\" (UID: \"c793f939-4584-46af-a495-36b57bbdff41\") " pod="kube-system/konnectivity-agent-xr9c8"
Sep 08 01:58:51.065323 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:51.065309    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/275d1493-2484-4412-b30d-b2aa36723bef-ovn-node-metrics-cert\") pod \"ovnkube-node-xskxl\" (UID: \"275d1493-2484-4412-b30d-b2aa36723bef\") " pod="openshift-ovn-kubernetes/ovnkube-node-xskxl"
Sep 08 01:58:51.067962 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:58:51.067941    2547 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 01:58:51.068049 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:58:51.067966    2547 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 01:58:51.068049 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:58:51.067980    2547 projected.go:196] Error preparing data for projected volume kube-api-access-5dt4k for pod openshift-network-diagnostics/network-check-target-2v24n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 01:58:51.068179 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:58:51.068057    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/430aad96-9d26-4055-8133-b7f50b0f2978-kube-api-access-5dt4k podName:430aad96-9d26-4055-8133-b7f50b0f2978 nodeName:}" failed. No retries permitted until 2026-09-08 01:58:51.568039672 +0000 UTC m=+3.091814738 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-5dt4k" (UniqueName: "kubernetes.io/projected/430aad96-9d26-4055-8133-b7f50b0f2978-kube-api-access-5dt4k") pod "network-check-target-2v24n" (UID: "430aad96-9d26-4055-8133-b7f50b0f2978") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 01:58:51.069932 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:51.069908    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlnz5\" (UniqueName: \"kubernetes.io/projected/0536c61b-c71c-41d9-8b2c-18228a7ddff4-kube-api-access-hlnz5\") pod \"node-resolver-4s9cm\" (UID: \"0536c61b-c71c-41d9-8b2c-18228a7ddff4\") " pod="openshift-dns/node-resolver-4s9cm"
Sep 08 01:58:51.070034 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:51.069917    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwg6c\" (UniqueName: \"kubernetes.io/projected/4b89e1b1-bcc0-4027-b778-19a7aba264a9-kube-api-access-hwg6c\") pod \"iptables-alerter-b7dzq\" (UID: \"4b89e1b1-bcc0-4027-b778-19a7aba264a9\") " pod="openshift-network-operator/iptables-alerter-b7dzq"
Sep 08 01:58:51.071043 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:51.071017    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qbcg\" (UniqueName: \"kubernetes.io/projected/275d1493-2484-4412-b30d-b2aa36723bef-kube-api-access-9qbcg\") pod \"ovnkube-node-xskxl\" (UID: \"275d1493-2484-4412-b30d-b2aa36723bef\") " pod="openshift-ovn-kubernetes/ovnkube-node-xskxl"
Sep 08 01:58:51.071489 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:51.071468    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7bs2\" (UniqueName: \"kubernetes.io/projected/b82c5af2-4c1c-41a9-8fa4-af8a85bcc8c6-kube-api-access-n7bs2\") pod \"network-metrics-daemon-jr45c\" (UID: \"b82c5af2-4c1c-41a9-8fa4-af8a85bcc8c6\") " pod="openshift-multus/network-metrics-daemon-jr45c"
Sep 08 01:58:51.072084 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:51.072060    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bbpj\" (UniqueName: \"kubernetes.io/projected/ae4a55d0-ad38-48a9-9a36-279949e2e4c5-kube-api-access-7bbpj\") pod \"node-ca-pl5qh\" (UID: \"ae4a55d0-ad38-48a9-9a36-279949e2e4c5\") " pod="openshift-image-registry/node-ca-pl5qh"
Sep 08 01:58:51.156344 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:51.156306    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lbtkf"
Sep 08 01:58:51.166885 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:51.166855    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7wjjb"
Sep 08 01:58:51.167490 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:51.167466    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44506a48_1a0a_4143_8ced_0dea3455ad2d.slice/crio-001e0f2b5f02bce7a96a8eded1c219c2632c8cf342187e561404f1042117cbdb WatchSource:0}: Error finding container 001e0f2b5f02bce7a96a8eded1c219c2632c8cf342187e561404f1042117cbdb: Status 404 returned error can't find the container with id 001e0f2b5f02bce7a96a8eded1c219c2632c8cf342187e561404f1042117cbdb
Sep 08 01:58:51.183187 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:51.183162    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-x72v8"
Sep 08 01:58:51.189660 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:51.189639    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-b4ms7"
Sep 08 01:58:51.200237 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:51.200212    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-b7dzq"
Sep 08 01:58:51.207285 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:51.207262    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xskxl"
Sep 08 01:58:51.217221 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:51.217191    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xr9c8"
Sep 08 01:58:51.228081 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:51.228048    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4s9cm"
Sep 08 01:58:51.233100 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:51.233075    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-pl5qh"
Sep 08 01:58:51.566484 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:51.566389    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/917f6c17-7902-48d3-a0f4-535e5b602d26-original-pull-secret\") pod \"global-pull-secret-syncer-l7hw8\" (UID: \"917f6c17-7902-48d3-a0f4-535e5b602d26\") " pod="kube-system/global-pull-secret-syncer-l7hw8"
Sep 08 01:58:51.566484 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:51.566456    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b82c5af2-4c1c-41a9-8fa4-af8a85bcc8c6-metrics-certs\") pod \"network-metrics-daemon-jr45c\" (UID: \"b82c5af2-4c1c-41a9-8fa4-af8a85bcc8c6\") " pod="openshift-multus/network-metrics-daemon-jr45c"
Sep 08 01:58:51.566694 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:58:51.566577    2547 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 01:58:51.566694 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:58:51.566655    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b82c5af2-4c1c-41a9-8fa4-af8a85bcc8c6-metrics-certs podName:b82c5af2-4c1c-41a9-8fa4-af8a85bcc8c6 nodeName:}" failed. No retries permitted until 2026-09-08 01:58:52.566637779 +0000 UTC m=+4.090412844 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b82c5af2-4c1c-41a9-8fa4-af8a85bcc8c6-metrics-certs") pod "network-metrics-daemon-jr45c" (UID: "b82c5af2-4c1c-41a9-8fa4-af8a85bcc8c6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 01:58:51.566694 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:58:51.566577    2547 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 01:58:51.566850 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:58:51.566737    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/917f6c17-7902-48d3-a0f4-535e5b602d26-original-pull-secret podName:917f6c17-7902-48d3-a0f4-535e5b602d26 nodeName:}" failed. No retries permitted until 2026-09-08 01:58:52.56672434 +0000 UTC m=+4.090499400 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/917f6c17-7902-48d3-a0f4-535e5b602d26-original-pull-secret") pod "global-pull-secret-syncer-l7hw8" (UID: "917f6c17-7902-48d3-a0f4-535e5b602d26") : object "kube-system"/"original-pull-secret" not registered
Sep 08 01:58:51.667479 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:51.667421    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dt4k\" (UniqueName: \"kubernetes.io/projected/430aad96-9d26-4055-8133-b7f50b0f2978-kube-api-access-5dt4k\") pod \"network-check-target-2v24n\" (UID: \"430aad96-9d26-4055-8133-b7f50b0f2978\") " pod="openshift-network-diagnostics/network-check-target-2v24n"
Sep 08 01:58:51.667653 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:58:51.667601    2547 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 01:58:51.667653 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:58:51.667634    2547 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 01:58:51.667653 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:58:51.667649    2547 projected.go:196] Error preparing data for projected volume kube-api-access-5dt4k for pod openshift-network-diagnostics/network-check-target-2v24n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 01:58:51.667781 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:58:51.667729    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/430aad96-9d26-4055-8133-b7f50b0f2978-kube-api-access-5dt4k podName:430aad96-9d26-4055-8133-b7f50b0f2978 nodeName:}" failed. No retries permitted until 2026-09-08 01:58:52.667704055 +0000 UTC m=+4.191479139 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-5dt4k" (UniqueName: "kubernetes.io/projected/430aad96-9d26-4055-8133-b7f50b0f2978-kube-api-access-5dt4k") pod "network-check-target-2v24n" (UID: "430aad96-9d26-4055-8133-b7f50b0f2978") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 01:58:51.895538 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:51.895492    2547 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-07 01:53:51 +0000 UTC" deadline="2028-06-09 19:45:12.098320664 +0000 UTC"
Sep 08 01:58:51.895538 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:51.895525    2547 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15377h46m20.202799437s"
Sep 08 01:58:51.988592 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:51.988554    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jr45c"
Sep 08 01:58:51.988804 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:58:51.988777    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jr45c" podUID="b82c5af2-4c1c-41a9-8fa4-af8a85bcc8c6"
Sep 08 01:58:52.002692 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:52.002641    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lbtkf" event={"ID":"44506a48-1a0a-4143-8ced-0dea3455ad2d","Type":"ContainerStarted","Data":"001e0f2b5f02bce7a96a8eded1c219c2632c8cf342187e561404f1042117cbdb"}
Sep 08 01:58:52.265674 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:52.265443    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc793f939_4584_46af_a495_36b57bbdff41.slice/crio-516e095dc305f79ccedebf679730e94c7ac38093f1f6841bd92c87406c9bac16 WatchSource:0}: Error finding container 516e095dc305f79ccedebf679730e94c7ac38093f1f6841bd92c87406c9bac16: Status 404 returned error can't find the container with id 516e095dc305f79ccedebf679730e94c7ac38093f1f6841bd92c87406c9bac16
Sep 08 01:58:52.272098 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:52.272068    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae4a55d0_ad38_48a9_9a36_279949e2e4c5.slice/crio-a8d11b7f401035279d229c9840d9ad11cf00d61aa7fa74e416a92924d96220cf WatchSource:0}: Error finding container a8d11b7f401035279d229c9840d9ad11cf00d61aa7fa74e416a92924d96220cf: Status 404 returned error can't find the container with id a8d11b7f401035279d229c9840d9ad11cf00d61aa7fa74e416a92924d96220cf
Sep 08 01:58:52.273038 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:52.272999    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf873b12_c8cd_4c48_a016_0be5e88e9894.slice/crio-b736996495dcaf66c58c9ad6fbde732547917603c53acbe283057c70076deb31 WatchSource:0}: Error finding container b736996495dcaf66c58c9ad6fbde732547917603c53acbe283057c70076deb31: Status 404 returned error can't find the container with id b736996495dcaf66c58c9ad6fbde732547917603c53acbe283057c70076deb31
Sep 08 01:58:52.274263 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:52.274231    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b89e1b1_bcc0_4027_b778_19a7aba264a9.slice/crio-b7c2b80fecc33c7291389e2cedc8564ceea9fec436840d3e8b6b0532542b6344 WatchSource:0}: Error finding container b7c2b80fecc33c7291389e2cedc8564ceea9fec436840d3e8b6b0532542b6344: Status 404 returned error can't find the container with id b7c2b80fecc33c7291389e2cedc8564ceea9fec436840d3e8b6b0532542b6344
Sep 08 01:58:52.275999 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:52.275519    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d7adf2a_2cfe_40ff_90d3_4886a747463a.slice/crio-ed17098344d3b0ccca809c29b722c4c4951e20dcc79725ee5ca1a48875cf35ad WatchSource:0}: Error finding container ed17098344d3b0ccca809c29b722c4c4951e20dcc79725ee5ca1a48875cf35ad: Status 404 returned error can't find the container with id ed17098344d3b0ccca809c29b722c4c4951e20dcc79725ee5ca1a48875cf35ad
Sep 08 01:58:52.277420 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:52.277384    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ce55233_87a7_4f6d_88a0_79d5d8eb2e3c.slice/crio-825dc4fa0f3dfcd28cf266890e4491b9218a24bd1d0d8ca5906447bacc134ecb WatchSource:0}: Error finding container 825dc4fa0f3dfcd28cf266890e4491b9218a24bd1d0d8ca5906447bacc134ecb: Status 404 returned error can't find the container with id 825dc4fa0f3dfcd28cf266890e4491b9218a24bd1d0d8ca5906447bacc134ecb
Sep 08 01:58:52.279194 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:52.279148    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod275d1493_2484_4412_b30d_b2aa36723bef.slice/crio-783b30f4e2730448721bdc0f50da79f3066f73c11ae68c8d08e1a1de02f513e3 WatchSource:0}: Error finding container 783b30f4e2730448721bdc0f50da79f3066f73c11ae68c8d08e1a1de02f513e3: Status 404 returned error can't find the container with id 783b30f4e2730448721bdc0f50da79f3066f73c11ae68c8d08e1a1de02f513e3
Sep 08 01:58:52.283491 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:58:52.283464    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0536c61b_c71c_41d9_8b2c_18228a7ddff4.slice/crio-fc41032a1e2fc3f424e8165da0822db50f93111ab92ad253a736445cb3a1febc WatchSource:0}: Error finding container fc41032a1e2fc3f424e8165da0822db50f93111ab92ad253a736445cb3a1febc: Status 404 returned error can't find the container with id fc41032a1e2fc3f424e8165da0822db50f93111ab92ad253a736445cb3a1febc
Sep 08 01:58:52.575561 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:52.575527    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b82c5af2-4c1c-41a9-8fa4-af8a85bcc8c6-metrics-certs\") pod \"network-metrics-daemon-jr45c\" (UID: \"b82c5af2-4c1c-41a9-8fa4-af8a85bcc8c6\") " pod="openshift-multus/network-metrics-daemon-jr45c"
Sep 08 01:58:52.575728 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:52.575600    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/917f6c17-7902-48d3-a0f4-535e5b602d26-original-pull-secret\") pod \"global-pull-secret-syncer-l7hw8\" (UID: \"917f6c17-7902-48d3-a0f4-535e5b602d26\") " pod="kube-system/global-pull-secret-syncer-l7hw8"
Sep 08 01:58:52.575728 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:58:52.575691    2547 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 01:58:52.575809 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:58:52.575755    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b82c5af2-4c1c-41a9-8fa4-af8a85bcc8c6-metrics-certs podName:b82c5af2-4c1c-41a9-8fa4-af8a85bcc8c6 nodeName:}" failed. No retries permitted until 2026-09-08 01:58:54.575741609 +0000 UTC m=+6.099516675 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b82c5af2-4c1c-41a9-8fa4-af8a85bcc8c6-metrics-certs") pod "network-metrics-daemon-jr45c" (UID: "b82c5af2-4c1c-41a9-8fa4-af8a85bcc8c6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 01:58:52.575809 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:58:52.575691    2547 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 01:58:52.575896 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:58:52.575827    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/917f6c17-7902-48d3-a0f4-535e5b602d26-original-pull-secret podName:917f6c17-7902-48d3-a0f4-535e5b602d26 nodeName:}" failed. No retries permitted until 2026-09-08 01:58:54.575813348 +0000 UTC m=+6.099588423 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/917f6c17-7902-48d3-a0f4-535e5b602d26-original-pull-secret") pod "global-pull-secret-syncer-l7hw8" (UID: "917f6c17-7902-48d3-a0f4-535e5b602d26") : object "kube-system"/"original-pull-secret" not registered
Sep 08 01:58:52.676410 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:52.676372    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dt4k\" (UniqueName: \"kubernetes.io/projected/430aad96-9d26-4055-8133-b7f50b0f2978-kube-api-access-5dt4k\") pod \"network-check-target-2v24n\" (UID: \"430aad96-9d26-4055-8133-b7f50b0f2978\") " pod="openshift-network-diagnostics/network-check-target-2v24n"
Sep 08 01:58:52.676586 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:58:52.676534    2547 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 01:58:52.676586 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:58:52.676557    2547 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 01:58:52.676586 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:58:52.676572    2547 projected.go:196] Error preparing data for projected volume kube-api-access-5dt4k for pod openshift-network-diagnostics/network-check-target-2v24n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 01:58:52.676765 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:58:52.676634    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/430aad96-9d26-4055-8133-b7f50b0f2978-kube-api-access-5dt4k podName:430aad96-9d26-4055-8133-b7f50b0f2978 nodeName:}" failed. No retries permitted until 2026-09-08 01:58:54.676616496 +0000 UTC m=+6.200391565 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-5dt4k" (UniqueName: "kubernetes.io/projected/430aad96-9d26-4055-8133-b7f50b0f2978-kube-api-access-5dt4k") pod "network-check-target-2v24n" (UID: "430aad96-9d26-4055-8133-b7f50b0f2978") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 01:58:52.895892 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:52.895774    2547 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-07 01:53:51 +0000 UTC" deadline="2028-04-15 21:03:00.367890251 +0000 UTC"
Sep 08 01:58:52.895892 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:52.895847    2547 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14059h4m7.47204793s"
Sep 08 01:58:52.991455 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:52.990362    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l7hw8"
Sep 08 01:58:52.991455 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:58:52.990502    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l7hw8" podUID="917f6c17-7902-48d3-a0f4-535e5b602d26"
Sep 08 01:58:52.991455 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:52.990585    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2v24n"
Sep 08 01:58:52.991455 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:58:52.990657    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2v24n" podUID="430aad96-9d26-4055-8133-b7f50b0f2978"
Sep 08 01:58:53.010505 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:53.010342    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xskxl" event={"ID":"275d1493-2484-4412-b30d-b2aa36723bef","Type":"ContainerStarted","Data":"783b30f4e2730448721bdc0f50da79f3066f73c11ae68c8d08e1a1de02f513e3"}
Sep 08 01:58:53.014653 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:53.014580    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7wjjb" event={"ID":"0ce55233-87a7-4f6d-88a0-79d5d8eb2e3c","Type":"ContainerStarted","Data":"825dc4fa0f3dfcd28cf266890e4491b9218a24bd1d0d8ca5906447bacc134ecb"}
Sep 08 01:58:53.019223 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:53.019173    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-b7dzq" event={"ID":"4b89e1b1-bcc0-4027-b778-19a7aba264a9","Type":"ContainerStarted","Data":"b7c2b80fecc33c7291389e2cedc8564ceea9fec436840d3e8b6b0532542b6344"}
Sep 08 01:58:53.024244 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:53.024035    2547 generic.go:356] "Generic (PLEG): container finished" podID="3d670938dd837b2cf06ee8ff34788dde" containerID="d7f4f751347791759459b184147e094f9903f4c01fb0f8657557375c81e3a046" exitCode=0
Sep 08 01:58:53.024244 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:53.024137    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-78.ec2.internal" event={"ID":"3d670938dd837b2cf06ee8ff34788dde","Type":"ContainerDied","Data":"d7f4f751347791759459b184147e094f9903f4c01fb0f8657557375c81e3a046"}
Sep 08 01:58:53.028164 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:53.028136    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4s9cm" event={"ID":"0536c61b-c71c-41d9-8b2c-18228a7ddff4","Type":"ContainerStarted","Data":"fc41032a1e2fc3f424e8165da0822db50f93111ab92ad253a736445cb3a1febc"}
Sep 08 01:58:53.031193 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:53.031167    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b4ms7" event={"ID":"2d7adf2a-2cfe-40ff-90d3-4886a747463a","Type":"ContainerStarted","Data":"ed17098344d3b0ccca809c29b722c4c4951e20dcc79725ee5ca1a48875cf35ad"}
Sep 08 01:58:53.035878 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:53.035852    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-x72v8" event={"ID":"af873b12-c8cd-4c48-a016-0be5e88e9894","Type":"ContainerStarted","Data":"b736996495dcaf66c58c9ad6fbde732547917603c53acbe283057c70076deb31"}
Sep 08 01:58:53.045017 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:53.044942    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-pl5qh" event={"ID":"ae4a55d0-ad38-48a9-9a36-279949e2e4c5","Type":"ContainerStarted","Data":"a8d11b7f401035279d229c9840d9ad11cf00d61aa7fa74e416a92924d96220cf"}
Sep 08 01:58:53.048754 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:53.048718    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xr9c8" event={"ID":"c793f939-4584-46af-a495-36b57bbdff41","Type":"ContainerStarted","Data":"516e095dc305f79ccedebf679730e94c7ac38093f1f6841bd92c87406c9bac16"}
Sep 08 01:58:53.054912 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:53.054878    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-78.ec2.internal" event={"ID":"5cfe1f564aebe22c530741e9b588e31d","Type":"ContainerStarted","Data":"7656f5a2787dcf0df7417f711d41b4f9a61938f3a1fa1f4b0a9f75a9eaa492e9"}
Sep 08 01:58:53.988536 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:53.988000    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jr45c"
Sep 08 01:58:53.988536 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:58:53.988149    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jr45c" podUID="b82c5af2-4c1c-41a9-8fa4-af8a85bcc8c6"
Sep 08 01:58:54.596108 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:54.596065    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/917f6c17-7902-48d3-a0f4-535e5b602d26-original-pull-secret\") pod \"global-pull-secret-syncer-l7hw8\" (UID: \"917f6c17-7902-48d3-a0f4-535e5b602d26\") " pod="kube-system/global-pull-secret-syncer-l7hw8"
Sep 08 01:58:54.596309 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:54.596139    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b82c5af2-4c1c-41a9-8fa4-af8a85bcc8c6-metrics-certs\") pod \"network-metrics-daemon-jr45c\" (UID: \"b82c5af2-4c1c-41a9-8fa4-af8a85bcc8c6\") " pod="openshift-multus/network-metrics-daemon-jr45c"
Sep 08 01:58:54.596393 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:58:54.596312    2547 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 01:58:54.596393 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:58:54.596382    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b82c5af2-4c1c-41a9-8fa4-af8a85bcc8c6-metrics-certs podName:b82c5af2-4c1c-41a9-8fa4-af8a85bcc8c6 nodeName:}" failed. No retries permitted until 2026-09-08 01:58:58.596360378 +0000 UTC m=+10.120135444 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b82c5af2-4c1c-41a9-8fa4-af8a85bcc8c6-metrics-certs") pod "network-metrics-daemon-jr45c" (UID: "b82c5af2-4c1c-41a9-8fa4-af8a85bcc8c6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 01:58:54.596818 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:58:54.596797    2547 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 01:58:54.596924 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:58:54.596858    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/917f6c17-7902-48d3-a0f4-535e5b602d26-original-pull-secret podName:917f6c17-7902-48d3-a0f4-535e5b602d26 nodeName:}" failed. No retries permitted until 2026-09-08 01:58:58.596839265 +0000 UTC m=+10.120614328 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/917f6c17-7902-48d3-a0f4-535e5b602d26-original-pull-secret") pod "global-pull-secret-syncer-l7hw8" (UID: "917f6c17-7902-48d3-a0f4-535e5b602d26") : object "kube-system"/"original-pull-secret" not registered
Sep 08 01:58:54.697711 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:54.696909    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dt4k\" (UniqueName: \"kubernetes.io/projected/430aad96-9d26-4055-8133-b7f50b0f2978-kube-api-access-5dt4k\") pod \"network-check-target-2v24n\" (UID: \"430aad96-9d26-4055-8133-b7f50b0f2978\") " pod="openshift-network-diagnostics/network-check-target-2v24n"
Sep 08 01:58:54.697711 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:58:54.697097    2547 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 01:58:54.697711 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:58:54.697122    2547 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 01:58:54.697711 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:58:54.697135    2547 projected.go:196] Error preparing data for projected volume kube-api-access-5dt4k for pod openshift-network-diagnostics/network-check-target-2v24n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 01:58:54.697711 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:58:54.697217    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/430aad96-9d26-4055-8133-b7f50b0f2978-kube-api-access-5dt4k podName:430aad96-9d26-4055-8133-b7f50b0f2978 nodeName:}" failed. No retries permitted until 2026-09-08 01:58:58.69719902 +0000 UTC m=+10.220974095 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-5dt4k" (UniqueName: "kubernetes.io/projected/430aad96-9d26-4055-8133-b7f50b0f2978-kube-api-access-5dt4k") pod "network-check-target-2v24n" (UID: "430aad96-9d26-4055-8133-b7f50b0f2978") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 01:58:54.991291 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:54.990728    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l7hw8"
Sep 08 01:58:54.991291 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:58:54.990896    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l7hw8" podUID="917f6c17-7902-48d3-a0f4-535e5b602d26"
Sep 08 01:58:54.991914 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:54.991398    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2v24n"
Sep 08 01:58:54.991914 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:58:54.991496    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2v24n" podUID="430aad96-9d26-4055-8133-b7f50b0f2978"
Sep 08 01:58:55.988597 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:55.988556    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jr45c"
Sep 08 01:58:55.988805 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:58:55.988715    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jr45c" podUID="b82c5af2-4c1c-41a9-8fa4-af8a85bcc8c6"
Sep 08 01:58:56.989255 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:56.989224    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2v24n"
Sep 08 01:58:56.989717 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:58:56.989352    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2v24n" podUID="430aad96-9d26-4055-8133-b7f50b0f2978"
Sep 08 01:58:56.990271 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:56.989888    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l7hw8"
Sep 08 01:58:56.990271 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:58:56.989986    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l7hw8" podUID="917f6c17-7902-48d3-a0f4-535e5b602d26"
Sep 08 01:58:57.988246 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:57.988203    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jr45c"
Sep 08 01:58:57.988401 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:58:57.988360    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jr45c" podUID="b82c5af2-4c1c-41a9-8fa4-af8a85bcc8c6"
Sep 08 01:58:58.631260 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:58.631225    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b82c5af2-4c1c-41a9-8fa4-af8a85bcc8c6-metrics-certs\") pod \"network-metrics-daemon-jr45c\" (UID: \"b82c5af2-4c1c-41a9-8fa4-af8a85bcc8c6\") " pod="openshift-multus/network-metrics-daemon-jr45c"
Sep 08 01:58:58.631671 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:58.631292    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/917f6c17-7902-48d3-a0f4-535e5b602d26-original-pull-secret\") pod \"global-pull-secret-syncer-l7hw8\" (UID: \"917f6c17-7902-48d3-a0f4-535e5b602d26\") " pod="kube-system/global-pull-secret-syncer-l7hw8"
Sep 08 01:58:58.631671 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:58:58.631380    2547 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 01:58:58.631671 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:58:58.631385    2547 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 01:58:58.631671 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:58:58.631434    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/917f6c17-7902-48d3-a0f4-535e5b602d26-original-pull-secret podName:917f6c17-7902-48d3-a0f4-535e5b602d26 nodeName:}" failed. No retries permitted until 2026-09-08 01:59:06.631417094 +0000 UTC m=+18.155192168 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/917f6c17-7902-48d3-a0f4-535e5b602d26-original-pull-secret") pod "global-pull-secret-syncer-l7hw8" (UID: "917f6c17-7902-48d3-a0f4-535e5b602d26") : object "kube-system"/"original-pull-secret" not registered
Sep 08 01:58:58.631671 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:58:58.631454    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b82c5af2-4c1c-41a9-8fa4-af8a85bcc8c6-metrics-certs podName:b82c5af2-4c1c-41a9-8fa4-af8a85bcc8c6 nodeName:}" failed. No retries permitted until 2026-09-08 01:59:06.631443926 +0000 UTC m=+18.155218984 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b82c5af2-4c1c-41a9-8fa4-af8a85bcc8c6-metrics-certs") pod "network-metrics-daemon-jr45c" (UID: "b82c5af2-4c1c-41a9-8fa4-af8a85bcc8c6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 01:58:58.731930 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:58.731900    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dt4k\" (UniqueName: \"kubernetes.io/projected/430aad96-9d26-4055-8133-b7f50b0f2978-kube-api-access-5dt4k\") pod \"network-check-target-2v24n\" (UID: \"430aad96-9d26-4055-8133-b7f50b0f2978\") " pod="openshift-network-diagnostics/network-check-target-2v24n"
Sep 08 01:58:58.732091 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:58:58.732018    2547 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 01:58:58.732091 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:58:58.732035    2547 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 01:58:58.732091 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:58:58.732045    2547 projected.go:196] Error preparing data for projected volume kube-api-access-5dt4k for pod openshift-network-diagnostics/network-check-target-2v24n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 01:58:58.732292 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:58:58.732098    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/430aad96-9d26-4055-8133-b7f50b0f2978-kube-api-access-5dt4k podName:430aad96-9d26-4055-8133-b7f50b0f2978 nodeName:}" failed. No retries permitted until 2026-09-08 01:59:06.732080526 +0000 UTC m=+18.255855591 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-5dt4k" (UniqueName: "kubernetes.io/projected/430aad96-9d26-4055-8133-b7f50b0f2978-kube-api-access-5dt4k") pod "network-check-target-2v24n" (UID: "430aad96-9d26-4055-8133-b7f50b0f2978") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 01:58:58.989453 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:58.988808    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2v24n"
Sep 08 01:58:58.989453 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:58:58.988920    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2v24n" podUID="430aad96-9d26-4055-8133-b7f50b0f2978"
Sep 08 01:58:58.989453 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:58.989312    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l7hw8"
Sep 08 01:58:58.989453 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:58:58.989402    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l7hw8" podUID="917f6c17-7902-48d3-a0f4-535e5b602d26"
Sep 08 01:58:59.988071 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:58:59.988037    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jr45c"
Sep 08 01:58:59.988588 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:58:59.988181    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jr45c" podUID="b82c5af2-4c1c-41a9-8fa4-af8a85bcc8c6"
Sep 08 01:59:00.992375 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:00.992340    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l7hw8"
Sep 08 01:59:00.992813 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:00.992343    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2v24n"
Sep 08 01:59:00.992813 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:59:00.992464    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l7hw8" podUID="917f6c17-7902-48d3-a0f4-535e5b602d26"
Sep 08 01:59:00.992813 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:59:00.992568    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2v24n" podUID="430aad96-9d26-4055-8133-b7f50b0f2978"
Sep 08 01:59:01.988048 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:01.988010    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jr45c"
Sep 08 01:59:01.988247 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:59:01.988203    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jr45c" podUID="b82c5af2-4c1c-41a9-8fa4-af8a85bcc8c6"
Sep 08 01:59:02.987956 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:02.987923    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2v24n"
Sep 08 01:59:02.988443 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:02.987923    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l7hw8"
Sep 08 01:59:02.988443 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:59:02.988058    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2v24n" podUID="430aad96-9d26-4055-8133-b7f50b0f2978"
Sep 08 01:59:02.988443 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:59:02.988150    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l7hw8" podUID="917f6c17-7902-48d3-a0f4-535e5b602d26"
Sep 08 01:59:03.987962 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:03.987913    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jr45c"
Sep 08 01:59:03.988462 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:59:03.988059    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jr45c" podUID="b82c5af2-4c1c-41a9-8fa4-af8a85bcc8c6"
Sep 08 01:59:04.988281 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:04.988249    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l7hw8"
Sep 08 01:59:04.988670 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:04.988292    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2v24n"
Sep 08 01:59:04.988670 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:59:04.988351    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l7hw8" podUID="917f6c17-7902-48d3-a0f4-535e5b602d26"
Sep 08 01:59:04.988670 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:59:04.988492    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2v24n" podUID="430aad96-9d26-4055-8133-b7f50b0f2978"
Sep 08 01:59:05.988285 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:05.988234    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jr45c"
Sep 08 01:59:05.988705 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:59:05.988345    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jr45c" podUID="b82c5af2-4c1c-41a9-8fa4-af8a85bcc8c6"
Sep 08 01:59:06.690260 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:06.690217    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b82c5af2-4c1c-41a9-8fa4-af8a85bcc8c6-metrics-certs\") pod \"network-metrics-daemon-jr45c\" (UID: \"b82c5af2-4c1c-41a9-8fa4-af8a85bcc8c6\") " pod="openshift-multus/network-metrics-daemon-jr45c"
Sep 08 01:59:06.690538 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:06.690311    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/917f6c17-7902-48d3-a0f4-535e5b602d26-original-pull-secret\") pod \"global-pull-secret-syncer-l7hw8\" (UID: \"917f6c17-7902-48d3-a0f4-535e5b602d26\") " pod="kube-system/global-pull-secret-syncer-l7hw8"
Sep 08 01:59:06.690538 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:59:06.690390    2547 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 01:59:06.690538 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:59:06.690440    2547 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 01:59:06.690538 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:59:06.690474    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b82c5af2-4c1c-41a9-8fa4-af8a85bcc8c6-metrics-certs podName:b82c5af2-4c1c-41a9-8fa4-af8a85bcc8c6 nodeName:}" failed. No retries permitted until 2026-09-08 01:59:22.690452537 +0000 UTC m=+34.214227616 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b82c5af2-4c1c-41a9-8fa4-af8a85bcc8c6-metrics-certs") pod "network-metrics-daemon-jr45c" (UID: "b82c5af2-4c1c-41a9-8fa4-af8a85bcc8c6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 01:59:06.690538 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:59:06.690501    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/917f6c17-7902-48d3-a0f4-535e5b602d26-original-pull-secret podName:917f6c17-7902-48d3-a0f4-535e5b602d26 nodeName:}" failed. No retries permitted until 2026-09-08 01:59:22.690483122 +0000 UTC m=+34.214258187 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/917f6c17-7902-48d3-a0f4-535e5b602d26-original-pull-secret") pod "global-pull-secret-syncer-l7hw8" (UID: "917f6c17-7902-48d3-a0f4-535e5b602d26") : object "kube-system"/"original-pull-secret" not registered
Sep 08 01:59:06.791457 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:06.791423    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dt4k\" (UniqueName: \"kubernetes.io/projected/430aad96-9d26-4055-8133-b7f50b0f2978-kube-api-access-5dt4k\") pod \"network-check-target-2v24n\" (UID: \"430aad96-9d26-4055-8133-b7f50b0f2978\") " pod="openshift-network-diagnostics/network-check-target-2v24n"
Sep 08 01:59:06.791643 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:59:06.791571    2547 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 01:59:06.791643 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:59:06.791592    2547 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 01:59:06.791643 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:59:06.791606    2547 projected.go:196] Error preparing data for projected volume kube-api-access-5dt4k for pod openshift-network-diagnostics/network-check-target-2v24n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 01:59:06.791805 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:59:06.791668    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/430aad96-9d26-4055-8133-b7f50b0f2978-kube-api-access-5dt4k podName:430aad96-9d26-4055-8133-b7f50b0f2978 nodeName:}" failed. No retries permitted until 2026-09-08 01:59:22.791649995 +0000 UTC m=+34.315425057 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-5dt4k" (UniqueName: "kubernetes.io/projected/430aad96-9d26-4055-8133-b7f50b0f2978-kube-api-access-5dt4k") pod "network-check-target-2v24n" (UID: "430aad96-9d26-4055-8133-b7f50b0f2978") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 01:59:06.988631 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:06.988550    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2v24n"
Sep 08 01:59:06.989059 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:06.988564    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l7hw8"
Sep 08 01:59:06.989059 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:59:06.988676    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2v24n" podUID="430aad96-9d26-4055-8133-b7f50b0f2978"
Sep 08 01:59:06.989059 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:59:06.988765    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l7hw8" podUID="917f6c17-7902-48d3-a0f4-535e5b602d26"
Sep 08 01:59:07.988528 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:07.988489    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jr45c"
Sep 08 01:59:07.988727 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:59:07.988640    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jr45c" podUID="b82c5af2-4c1c-41a9-8fa4-af8a85bcc8c6"
Sep 08 01:59:08.989476 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:08.989443    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2v24n"
Sep 08 01:59:08.989932 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:59:08.989556    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2v24n" podUID="430aad96-9d26-4055-8133-b7f50b0f2978"
Sep 08 01:59:08.989932 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:08.989615    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l7hw8"
Sep 08 01:59:08.989932 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:59:08.989688    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l7hw8" podUID="917f6c17-7902-48d3-a0f4-535e5b602d26"
Sep 08 01:59:09.988323 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:09.987979    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jr45c"
Sep 08 01:59:09.988400 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:59:09.988386    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jr45c" podUID="b82c5af2-4c1c-41a9-8fa4-af8a85bcc8c6"
Sep 08 01:59:10.094172 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:10.094119    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lbtkf" event={"ID":"44506a48-1a0a-4143-8ced-0dea3455ad2d","Type":"ContainerStarted","Data":"fcffad2c039fbfb6a177eb505b159c84943f3aec492105fc49717f9ceb687f94"}
Sep 08 01:59:10.097348 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:10.097321    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xskxl" event={"ID":"275d1493-2484-4412-b30d-b2aa36723bef","Type":"ContainerStarted","Data":"28eb91981c890bcd344837aa4e9348e727c331f2aba1f16ec4a6faf270cd57cd"}
Sep 08 01:59:10.097462 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:10.097404    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xskxl" event={"ID":"275d1493-2484-4412-b30d-b2aa36723bef","Type":"ContainerStarted","Data":"f5bf384fc142d14db01914cd67881337c99b524d4a32b333dd7c74e05c60a9c4"}
Sep 08 01:59:10.097462 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:10.097443    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xskxl" event={"ID":"275d1493-2484-4412-b30d-b2aa36723bef","Type":"ContainerStarted","Data":"7b07255f7144eb273bf5d04faa78407fd7980aaf7b4421e3453b269f7a36db2f"}
Sep 08 01:59:10.097553 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:10.097475    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xskxl" event={"ID":"275d1493-2484-4412-b30d-b2aa36723bef","Type":"ContainerStarted","Data":"9d5f299b37b30809159c2384e9c5128f2ecaa21096bcab77f2974dc0bb8c3ec5"}
Sep 08 01:59:10.097553 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:10.097513    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xskxl" event={"ID":"275d1493-2484-4412-b30d-b2aa36723bef","Type":"ContainerStarted","Data":"3b4a6a40de610b251ac99b82d044dbd9f5e70a62d6310c917b7319b6d03cf3b9"}
Sep 08 01:59:10.097553 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:10.097546    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xskxl" event={"ID":"275d1493-2484-4412-b30d-b2aa36723bef","Type":"ContainerStarted","Data":"b35146593e7f7bcb8915d2dc77f98a68a97718fbe80cb793089248f17805fcb7"}
Sep 08 01:59:10.098767 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:10.098739    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7wjjb" event={"ID":"0ce55233-87a7-4f6d-88a0-79d5d8eb2e3c","Type":"ContainerStarted","Data":"fd9e4169c9a9b6952df14d278d10e39c689e488f737a996e83ea393ab58407c5"}
Sep 08 01:59:10.100686 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:10.100666    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-78.ec2.internal" event={"ID":"3d670938dd837b2cf06ee8ff34788dde","Type":"ContainerStarted","Data":"e024c1a87ea49ccbd37d1600a544daf78779ec0e954917a5c5df4c3f0e89f914"}
Sep 08 01:59:10.102358 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:10.102297    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4s9cm" event={"ID":"0536c61b-c71c-41d9-8b2c-18228a7ddff4","Type":"ContainerStarted","Data":"ffa116c2c33d0ffdbe277b52a867082290f6cef49d3ba58dddf3207600b71010"}
Sep 08 01:59:10.103896 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:10.103876    2547 generic.go:356] "Generic (PLEG): container finished" podID="2d7adf2a-2cfe-40ff-90d3-4886a747463a" containerID="70deb5d9470b29189da3444eed5e299c924bed3024eefb3091a980bb06bd09cd" exitCode=0
Sep 08 01:59:10.104005 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:10.103929    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b4ms7" event={"ID":"2d7adf2a-2cfe-40ff-90d3-4886a747463a","Type":"ContainerDied","Data":"70deb5d9470b29189da3444eed5e299c924bed3024eefb3091a980bb06bd09cd"}
Sep 08 01:59:10.105563 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:10.105542    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-x72v8" event={"ID":"af873b12-c8cd-4c48-a016-0be5e88e9894","Type":"ContainerStarted","Data":"0095bbba097d2b3ec6fc403999c2baf52f84a1d3b88d70bc55d5e5ac0cac50a7"}
Sep 08 01:59:10.110640 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:10.110564    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-pl5qh" event={"ID":"ae4a55d0-ad38-48a9-9a36-279949e2e4c5","Type":"ContainerStarted","Data":"ab95296e38047eb4dfae90465238e6590c6620b474f133d9e4f2ba9d1fe76100"}
Sep 08 01:59:10.112022 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:10.111990    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xr9c8" event={"ID":"c793f939-4584-46af-a495-36b57bbdff41","Type":"ContainerStarted","Data":"ebc4c46ffa99210a25c19b2bde8c258de8a9c6429e0a756b997cfe8dec259041"}
Sep 08 01:59:10.113997 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:10.113947    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-78.ec2.internal" podStartSLOduration=20.113936708 podStartE2EDuration="20.113936708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 01:58:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 01:58:53.069699029 +0000 UTC m=+4.593474115" watchObservedRunningTime="2026-09-08 01:59:10.113936708 +0000 UTC m=+21.637711797"
Sep 08 01:59:10.114200 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:10.114126    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-7wjjb" podStartSLOduration=3.6937160479999998 podStartE2EDuration="21.114122374s" totalImagesPullingTime="17.420406326s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 01:58:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 01:59:10.114031924 +0000 UTC m=+21.637807005" watchObservedRunningTime="2026-09-08 01:59:10.114122374 +0000 UTC m=+21.637897453"
Sep 08 01:59:10.126527 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:10.126478    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-4s9cm" podStartSLOduration=3.94700571 podStartE2EDuration="21.126463313s" totalImagesPullingTime="17.179457603s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 01:58:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 01:59:10.126100123 +0000 UTC m=+21.649875202" watchObservedRunningTime="2026-09-08 01:59:10.126463313 +0000 UTC m=+21.650238394"
Sep 08 01:59:10.174380 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:10.174333    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-xr9c8" podStartSLOduration=3.942833539 podStartE2EDuration="21.17431827s" totalImagesPullingTime="17.231484731s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 01:58:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 01:59:10.161019479 +0000 UTC m=+21.684794559" watchObservedRunningTime="2026-09-08 01:59:10.17431827 +0000 UTC m=+21.698093327"
Sep 08 01:59:10.174835 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:10.174812    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-pl5qh" podStartSLOduration=3.923578111 podStartE2EDuration="21.174806365s" totalImagesPullingTime="17.251228254s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 01:58:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 01:59:10.174513624 +0000 UTC m=+21.698288704" watchObservedRunningTime="2026-09-08 01:59:10.174806365 +0000 UTC m=+21.698581444"
Sep 08 01:59:10.191926 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:10.191884    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-78.ec2.internal" podStartSLOduration=20.191870547 podStartE2EDuration="20.191870547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 01:58:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 01:59:10.191654114 +0000 UTC m=+21.715429194" watchObservedRunningTime="2026-09-08 01:59:10.191870547 +0000 UTC m=+21.715645631"
Sep 08 01:59:10.197593 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:10.197574    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-xr9c8"
Sep 08 01:59:10.198070 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:10.198055    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-xr9c8"
Sep 08 01:59:10.209994 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:10.209957    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-x72v8" podStartSLOduration=3.959751752 podStartE2EDuration="21.209945428s" totalImagesPullingTime="17.250193676s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 01:58:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 01:59:10.209897182 +0000 UTC m=+21.733672261" watchObservedRunningTime="2026-09-08 01:59:10.209945428 +0000 UTC m=+21.733720508"
Sep 08 01:59:10.734736 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:10.734716    2547 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 08 01:59:10.988658 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:10.988413    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2v24n"
Sep 08 01:59:10.988828 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:59:10.988766    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2v24n" podUID="430aad96-9d26-4055-8133-b7f50b0f2978"
Sep 08 01:59:10.988965 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:10.988492    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l7hw8"
Sep 08 01:59:10.989077 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:59:10.989055    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l7hw8" podUID="917f6c17-7902-48d3-a0f4-535e5b602d26"
Sep 08 01:59:11.055094 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:11.054989    2547 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-08T01:59:10.734733076Z","UUID":"a0bbcb74-b18b-4a47-83b4-234972c11a8c","Handler":null,"Name":"","Endpoint":""}
Sep 08 01:59:11.056688 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:11.056655    2547 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 08 01:59:11.056688 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:11.056688    2547 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 08 01:59:11.117422 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:11.117209    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lbtkf" event={"ID":"44506a48-1a0a-4143-8ced-0dea3455ad2d","Type":"ContainerStarted","Data":"1922837d4ebb80dff73b329a1f4e2ae54feb3b4852047e13b9fab7518aae51cb"}
Sep 08 01:59:11.120578 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:11.119679    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-b7dzq" event={"ID":"4b89e1b1-bcc0-4027-b778-19a7aba264a9","Type":"ContainerStarted","Data":"393b78fe802a6182681aa85937916dc7dafeb94bbd107dbd0228b1c337d54c74"}
Sep 08 01:59:11.120578 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:11.120352    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-xr9c8"
Sep 08 01:59:11.121071 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:11.120799    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-xr9c8"
Sep 08 01:59:11.133658 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:11.133600    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-b7dzq" podStartSLOduration=4.910683195 podStartE2EDuration="22.133579534s" totalImagesPullingTime="17.222896339s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 01:58:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 01:59:11.132272379 +0000 UTC m=+22.656047460" watchObservedRunningTime="2026-09-08 01:59:11.133579534 +0000 UTC m=+22.657354616"
Sep 08 01:59:11.987861 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:11.987826    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jr45c"
Sep 08 01:59:11.988063 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:59:11.987956    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jr45c" podUID="b82c5af2-4c1c-41a9-8fa4-af8a85bcc8c6"
Sep 08 01:59:12.123860 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:12.123822    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lbtkf" event={"ID":"44506a48-1a0a-4143-8ced-0dea3455ad2d","Type":"ContainerStarted","Data":"c7afb41639a29a42304c9b6580fd45a1f7263e620edaa128554e5d67cc1ba3b2"}
Sep 08 01:59:12.138769 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:12.138725    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lbtkf" podStartSLOduration=7.648864085 podStartE2EDuration="23.138707272s" totalImagesPullingTime="15.489843187s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 01:58:49 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 01:59:12.138612617 +0000 UTC m=+23.662387698" watchObservedRunningTime="2026-09-08 01:59:12.138707272 +0000 UTC m=+23.662482352"
Sep 08 01:59:12.988894 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:12.988861    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2v24n"
Sep 08 01:59:12.989187 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:12.988861    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l7hw8"
Sep 08 01:59:12.989187 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:59:12.988979    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2v24n" podUID="430aad96-9d26-4055-8133-b7f50b0f2978"
Sep 08 01:59:12.989187 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:59:12.989062    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l7hw8" podUID="917f6c17-7902-48d3-a0f4-535e5b602d26"
Sep 08 01:59:13.130209 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:13.130167    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xskxl" event={"ID":"275d1493-2484-4412-b30d-b2aa36723bef","Type":"ContainerStarted","Data":"62754e27bfe1924e1cd076ca2c52fd375ae2f10fe24e7cbaea64cedc3e9987a5"}
Sep 08 01:59:13.988835 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:13.988797    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jr45c"
Sep 08 01:59:13.989021 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:59:13.988940    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jr45c" podUID="b82c5af2-4c1c-41a9-8fa4-af8a85bcc8c6"
Sep 08 01:59:14.988924 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:14.988730    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2v24n"
Sep 08 01:59:14.989438 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:14.988770    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l7hw8"
Sep 08 01:59:14.989438 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:59:14.989015    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2v24n" podUID="430aad96-9d26-4055-8133-b7f50b0f2978"
Sep 08 01:59:14.989438 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:59:14.989064    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l7hw8" podUID="917f6c17-7902-48d3-a0f4-535e5b602d26"
Sep 08 01:59:15.137528 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:15.137490    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xskxl" event={"ID":"275d1493-2484-4412-b30d-b2aa36723bef","Type":"ContainerStarted","Data":"fb2927a5c90cdeea784d3eca2a253323d6f008a960d89f04a53fb34080378f46"}
Sep 08 01:59:15.137747 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:15.137732    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xskxl"
Sep 08 01:59:15.139360 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:15.139338    2547 generic.go:356] "Generic (PLEG): container finished" podID="2d7adf2a-2cfe-40ff-90d3-4886a747463a" containerID="ca395fa89da72adfed2799e5972f4577f170489f4acbcbd810a50b3078e9789d" exitCode=0
Sep 08 01:59:15.139456 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:15.139371    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b4ms7" event={"ID":"2d7adf2a-2cfe-40ff-90d3-4886a747463a","Type":"ContainerDied","Data":"ca395fa89da72adfed2799e5972f4577f170489f4acbcbd810a50b3078e9789d"}
Sep 08 01:59:15.153486 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:15.153461    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xskxl"
Sep 08 01:59:15.159877 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:15.159836    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-xskxl" podStartSLOduration=8.903530633999999 podStartE2EDuration="26.159822617s" totalImagesPullingTime="17.256291983s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 01:58:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 01:59:15.158851952 +0000 UTC m=+26.682627033" watchObservedRunningTime="2026-09-08 01:59:15.159822617 +0000 UTC m=+26.683597694"
Sep 08 01:59:15.988075 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:15.988010    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jr45c"
Sep 08 01:59:15.988219 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:59:15.988192    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jr45c" podUID="b82c5af2-4c1c-41a9-8fa4-af8a85bcc8c6"
Sep 08 01:59:16.143092 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:16.143060    2547 generic.go:356] "Generic (PLEG): container finished" podID="2d7adf2a-2cfe-40ff-90d3-4886a747463a" containerID="c557604b3ea2ed4f9d303cfee415e6cfe448562bfc8279217b1693fae024221f" exitCode=0
Sep 08 01:59:16.143782 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:16.143111    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b4ms7" event={"ID":"2d7adf2a-2cfe-40ff-90d3-4886a747463a","Type":"ContainerDied","Data":"c557604b3ea2ed4f9d303cfee415e6cfe448562bfc8279217b1693fae024221f"}
Sep 08 01:59:16.143782 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:16.143646    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xskxl"
Sep 08 01:59:16.143782 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:16.143711    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xskxl"
Sep 08 01:59:16.159656 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:16.159504    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xskxl"
Sep 08 01:59:16.260119 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:16.260092    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jr45c"]
Sep 08 01:59:16.260268 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:16.260242    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jr45c"
Sep 08 01:59:16.260349 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:59:16.260327    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jr45c" podUID="b82c5af2-4c1c-41a9-8fa4-af8a85bcc8c6"
Sep 08 01:59:16.264031 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:16.264004    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-l7hw8"]
Sep 08 01:59:16.264173 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:16.264116    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l7hw8"
Sep 08 01:59:16.264254 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:59:16.264232    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l7hw8" podUID="917f6c17-7902-48d3-a0f4-535e5b602d26"
Sep 08 01:59:16.264791 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:16.264771    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2v24n"]
Sep 08 01:59:16.264886 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:16.264852    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2v24n"
Sep 08 01:59:16.264950 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:59:16.264934    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2v24n" podUID="430aad96-9d26-4055-8133-b7f50b0f2978"
Sep 08 01:59:17.147059 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:17.147029    2547 generic.go:356] "Generic (PLEG): container finished" podID="2d7adf2a-2cfe-40ff-90d3-4886a747463a" containerID="c2473ba36ccd9bddc480475ecbffe4721478f41bc9a344cd70ab363407b7d7d2" exitCode=0
Sep 08 01:59:17.147524 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:17.147118    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b4ms7" event={"ID":"2d7adf2a-2cfe-40ff-90d3-4886a747463a","Type":"ContainerDied","Data":"c2473ba36ccd9bddc480475ecbffe4721478f41bc9a344cd70ab363407b7d7d2"}
Sep 08 01:59:17.988275 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:17.988238    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jr45c"
Sep 08 01:59:17.988454 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:17.988395    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2v24n"
Sep 08 01:59:17.988454 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:17.988420    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l7hw8"
Sep 08 01:59:17.988558 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:59:17.988505    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2v24n" podUID="430aad96-9d26-4055-8133-b7f50b0f2978"
Sep 08 01:59:17.988558 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:59:17.988392    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jr45c" podUID="b82c5af2-4c1c-41a9-8fa4-af8a85bcc8c6"
Sep 08 01:59:17.988658 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:59:17.988594    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l7hw8" podUID="917f6c17-7902-48d3-a0f4-535e5b602d26"
Sep 08 01:59:19.988894 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:19.988850    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l7hw8"
Sep 08 01:59:19.988894 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:19.988890    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2v24n"
Sep 08 01:59:19.989454 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:19.988874    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jr45c"
Sep 08 01:59:19.989454 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:59:19.988985    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l7hw8" podUID="917f6c17-7902-48d3-a0f4-535e5b602d26"
Sep 08 01:59:19.989454 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:59:19.989078    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jr45c" podUID="b82c5af2-4c1c-41a9-8fa4-af8a85bcc8c6"
Sep 08 01:59:19.989454 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:59:19.989129    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2v24n" podUID="430aad96-9d26-4055-8133-b7f50b0f2978"
Sep 08 01:59:21.988368 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:21.988342    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2v24n"
Sep 08 01:59:21.991581 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:59:21.988863    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2v24n" podUID="430aad96-9d26-4055-8133-b7f50b0f2978"
Sep 08 01:59:21.991581 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:21.989529    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jr45c"
Sep 08 01:59:21.991581 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:59:21.989687    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jr45c" podUID="b82c5af2-4c1c-41a9-8fa4-af8a85bcc8c6"
Sep 08 01:59:21.991581 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:21.989739    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l7hw8"
Sep 08 01:59:21.991581 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:59:21.989837    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l7hw8" podUID="917f6c17-7902-48d3-a0f4-535e5b602d26"
Sep 08 01:59:22.161842 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:22.161811    2547 generic.go:356] "Generic (PLEG): container finished" podID="2d7adf2a-2cfe-40ff-90d3-4886a747463a" containerID="01471dce537150ab64370b635538e6d1ec205b42b0c39f28ef36e5bcc36f5725" exitCode=0
Sep 08 01:59:22.161974 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:22.161854    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b4ms7" event={"ID":"2d7adf2a-2cfe-40ff-90d3-4886a747463a","Type":"ContainerDied","Data":"01471dce537150ab64370b635538e6d1ec205b42b0c39f28ef36e5bcc36f5725"}
Sep 08 01:59:22.228127 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:22.228099    2547 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-78.ec2.internal" event="NodeReady"
Sep 08 01:59:22.228300 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:22.228285    2547 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 08 01:59:22.276414 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:22.276388    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-87nvd"]
Sep 08 01:59:22.293635 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:22.293613    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-q6gwz"]
Sep 08 01:59:22.293752 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:22.293735    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-87nvd"
Sep 08 01:59:22.295792 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:22.295766    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 08 01:59:22.295896 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:22.295820    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 08 01:59:22.295896 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:22.295820    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-r5fzh\""
Sep 08 01:59:22.295896 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:22.295866    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 08 01:59:22.310280 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:22.310260    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-68cff75bfd-p6b52"]
Sep 08 01:59:22.310404 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:22.310384    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-q6gwz"
Sep 08 01:59:22.312458 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:22.312433    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 08 01:59:22.312554 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:22.312478    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 08 01:59:22.312554 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:22.312486    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6zllf\""
Sep 08 01:59:22.325031 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:22.324993    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-87nvd"]
Sep 08 01:59:22.325158 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:22.325090    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-q6gwz"]
Sep 08 01:59:22.325217 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:22.325172    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68cff75bfd-p6b52"
Sep 08 01:59:22.325283 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:22.325249    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-68cff75bfd-p6b52"]
Sep 08 01:59:22.327468 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:22.327450    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 08 01:59:22.327546 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:22.327486    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 08 01:59:22.327759 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:22.327737    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 08 01:59:22.327875 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:22.327819    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-zntsx\""
Sep 08 01:59:22.331406 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:22.331384    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 08 01:59:22.400712 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:22.400677    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3ba60eb6-74cd-4416-9a15-d04ba2ef2b1e-config-volume\") pod \"dns-default-q6gwz\" (UID: \"3ba60eb6-74cd-4416-9a15-d04ba2ef2b1e\") " pod="openshift-dns/dns-default-q6gwz"
Sep 08 01:59:22.400882 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:22.400717    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqfvn\" (UniqueName: \"kubernetes.io/projected/3ba60eb6-74cd-4416-9a15-d04ba2ef2b1e-kube-api-access-wqfvn\") pod \"dns-default-q6gwz\" (UID: \"3ba60eb6-74cd-4416-9a15-d04ba2ef2b1e\") " pod="openshift-dns/dns-default-q6gwz"
Sep 08 01:59:22.400882 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:22.400744    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c7d9f993-10b4-4226-8193-a7599e0d3024-ca-trust-extracted\") pod \"image-registry-68cff75bfd-p6b52\" (UID: \"c7d9f993-10b4-4226-8193-a7599e0d3024\") " pod="openshift-image-registry/image-registry-68cff75bfd-p6b52"
Sep 08 01:59:22.400882 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:22.400766    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3ba60eb6-74cd-4416-9a15-d04ba2ef2b1e-metrics-tls\") pod \"dns-default-q6gwz\" (UID: \"3ba60eb6-74cd-4416-9a15-d04ba2ef2b1e\") " pod="openshift-dns/dns-default-q6gwz"
Sep 08 01:59:22.400882 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:22.400790    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c7d9f993-10b4-4226-8193-a7599e0d3024-image-registry-private-configuration\") pod \"image-registry-68cff75bfd-p6b52\" (UID: \"c7d9f993-10b4-4226-8193-a7599e0d3024\") " pod="openshift-image-registry/image-registry-68cff75bfd-p6b52"
Sep 08 01:59:22.400882 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:22.400814    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c7d9f993-10b4-4226-8193-a7599e0d3024-registry-tls\") pod \"image-registry-68cff75bfd-p6b52\" (UID: \"c7d9f993-10b4-4226-8193-a7599e0d3024\") " pod="openshift-image-registry/image-registry-68cff75bfd-p6b52"
Sep 08 01:59:22.400882 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:22.400837    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hssd\" (UniqueName: \"kubernetes.io/projected/444897cb-e238-4b58-a0ef-9adce681c307-kube-api-access-6hssd\") pod \"ingress-canary-87nvd\" (UID: \"444897cb-e238-4b58-a0ef-9adce681c307\") " pod="openshift-ingress-canary/ingress-canary-87nvd"
Sep 08 01:59:22.401095 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:22.400909    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c7d9f993-10b4-4226-8193-a7599e0d3024-registry-certificates\") pod \"image-registry-68cff75bfd-p6b52\" (UID: \"c7d9f993-10b4-4226-8193-a7599e0d3024\") " pod="openshift-image-registry/image-registry-68cff75bfd-p6b52"
Sep 08 01:59:22.401095 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:22.400935    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/444897cb-e238-4b58-a0ef-9adce681c307-cert\") pod \"ingress-canary-87nvd\" (UID: \"444897cb-e238-4b58-a0ef-9adce681c307\") " pod="openshift-ingress-canary/ingress-canary-87nvd"
Sep 08 01:59:22.401095 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:22.400967    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ba60eb6-74cd-4416-9a15-d04ba2ef2b1e-tmp-dir\") pod \"dns-default-q6gwz\" (UID: \"3ba60eb6-74cd-4416-9a15-d04ba2ef2b1e\") " pod="openshift-dns/dns-default-q6gwz"
Sep 08 01:59:22.401095 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:22.400995    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c7d9f993-10b4-4226-8193-a7599e0d3024-trusted-ca\") pod \"image-registry-68cff75bfd-p6b52\" (UID: \"c7d9f993-10b4-4226-8193-a7599e0d3024\") " pod="openshift-image-registry/image-registry-68cff75bfd-p6b52"
Sep 08 01:59:22.401095 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:22.401014    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c7d9f993-10b4-4226-8193-a7599e0d3024-bound-sa-token\") pod \"image-registry-68cff75bfd-p6b52\" (UID: \"c7d9f993-10b4-4226-8193-a7599e0d3024\") " pod="openshift-image-registry/image-registry-68cff75bfd-p6b52"
Sep 08 01:59:22.401095 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:22.401034    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c7d9f993-10b4-4226-8193-a7599e0d3024-installation-pull-secrets\") pod \"image-registry-68cff75bfd-p6b52\" (UID: \"c7d9f993-10b4-4226-8193-a7599e0d3024\") " pod="openshift-image-registry/image-registry-68cff75bfd-p6b52"
Sep 08 01:59:22.401095 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:22.401054    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkztm\" (UniqueName: \"kubernetes.io/projected/c7d9f993-10b4-4226-8193-a7599e0d3024-kube-api-access-pkztm\") pod \"image-registry-68cff75bfd-p6b52\" (UID: \"c7d9f993-10b4-4226-8193-a7599e0d3024\") " pod="openshift-image-registry/image-registry-68cff75bfd-p6b52"
Sep 08 01:59:22.501627 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:22.501533    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c7d9f993-10b4-4226-8193-a7599e0d3024-trusted-ca\") pod \"image-registry-68cff75bfd-p6b52\" (UID: \"c7d9f993-10b4-4226-8193-a7599e0d3024\") " pod="openshift-image-registry/image-registry-68cff75bfd-p6b52"
Sep 08 01:59:22.501627 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:22.501574    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c7d9f993-10b4-4226-8193-a7599e0d3024-bound-sa-token\") pod \"image-registry-68cff75bfd-p6b52\" (UID: \"c7d9f993-10b4-4226-8193-a7599e0d3024\") " pod="openshift-image-registry/image-registry-68cff75bfd-p6b52"
Sep 08 01:59:22.501627 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:22.501601    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c7d9f993-10b4-4226-8193-a7599e0d3024-installation-pull-secrets\") pod \"image-registry-68cff75bfd-p6b52\" (UID: \"c7d9f993-10b4-4226-8193-a7599e0d3024\") " pod="openshift-image-registry/image-registry-68cff75bfd-p6b52"
Sep 08 01:59:22.501627 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:22.501626    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkztm\" (UniqueName: \"kubernetes.io/projected/c7d9f993-10b4-4226-8193-a7599e0d3024-kube-api-access-pkztm\") pod \"image-registry-68cff75bfd-p6b52\" (UID: \"c7d9f993-10b4-4226-8193-a7599e0d3024\") " pod="openshift-image-registry/image-registry-68cff75bfd-p6b52"
Sep 08 01:59:22.501944 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:22.501679    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3ba60eb6-74cd-4416-9a15-d04ba2ef2b1e-config-volume\") pod \"dns-default-q6gwz\" (UID: \"3ba60eb6-74cd-4416-9a15-d04ba2ef2b1e\") " pod="openshift-dns/dns-default-q6gwz"
Sep 08 01:59:22.501944 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:22.501704    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqfvn\" (UniqueName: \"kubernetes.io/projected/3ba60eb6-74cd-4416-9a15-d04ba2ef2b1e-kube-api-access-wqfvn\") pod \"dns-default-q6gwz\" (UID: \"3ba60eb6-74cd-4416-9a15-d04ba2ef2b1e\") " pod="openshift-dns/dns-default-q6gwz"
Sep 08 01:59:22.501944 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:22.501726    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c7d9f993-10b4-4226-8193-a7599e0d3024-ca-trust-extracted\") pod \"image-registry-68cff75bfd-p6b52\" (UID: \"c7d9f993-10b4-4226-8193-a7599e0d3024\") " pod="openshift-image-registry/image-registry-68cff75bfd-p6b52"
Sep 08 01:59:22.501944 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:22.501749    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3ba60eb6-74cd-4416-9a15-d04ba2ef2b1e-metrics-tls\") pod \"dns-default-q6gwz\" (UID: \"3ba60eb6-74cd-4416-9a15-d04ba2ef2b1e\") " pod="openshift-dns/dns-default-q6gwz"
Sep 08 01:59:22.501944 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:59:22.501826    2547 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 01:59:22.501944 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:59:22.501879    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ba60eb6-74cd-4416-9a15-d04ba2ef2b1e-metrics-tls podName:3ba60eb6-74cd-4416-9a15-d04ba2ef2b1e nodeName:}" failed. No retries permitted until 2026-09-08 01:59:23.001861918 +0000 UTC m=+34.525636981 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3ba60eb6-74cd-4416-9a15-d04ba2ef2b1e-metrics-tls") pod "dns-default-q6gwz" (UID: "3ba60eb6-74cd-4416-9a15-d04ba2ef2b1e") : secret "dns-default-metrics-tls" not found
Sep 08 01:59:22.501944 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:22.501907    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c7d9f993-10b4-4226-8193-a7599e0d3024-image-registry-private-configuration\") pod \"image-registry-68cff75bfd-p6b52\" (UID: \"c7d9f993-10b4-4226-8193-a7599e0d3024\") " pod="openshift-image-registry/image-registry-68cff75bfd-p6b52"
Sep 08 01:59:22.502252 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:22.501953    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c7d9f993-10b4-4226-8193-a7599e0d3024-registry-tls\") pod \"image-registry-68cff75bfd-p6b52\" (UID: \"c7d9f993-10b4-4226-8193-a7599e0d3024\") " pod="openshift-image-registry/image-registry-68cff75bfd-p6b52"
Sep 08 01:59:22.502252 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:22.501978    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hssd\" (UniqueName: \"kubernetes.io/projected/444897cb-e238-4b58-a0ef-9adce681c307-kube-api-access-6hssd\") pod \"ingress-canary-87nvd\" (UID: \"444897cb-e238-4b58-a0ef-9adce681c307\") " pod="openshift-ingress-canary/ingress-canary-87nvd"
Sep 08 01:59:22.502252 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:22.502020    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c7d9f993-10b4-4226-8193-a7599e0d3024-registry-certificates\") pod \"image-registry-68cff75bfd-p6b52\" (UID: \"c7d9f993-10b4-4226-8193-a7599e0d3024\") " pod="openshift-image-registry/image-registry-68cff75bfd-p6b52"
Sep 08 01:59:22.502252 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:22.502045    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/444897cb-e238-4b58-a0ef-9adce681c307-cert\") pod \"ingress-canary-87nvd\" (UID: \"444897cb-e238-4b58-a0ef-9adce681c307\") " pod="openshift-ingress-canary/ingress-canary-87nvd"
Sep 08 01:59:22.502252 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:22.502088    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ba60eb6-74cd-4416-9a15-d04ba2ef2b1e-tmp-dir\") pod \"dns-default-q6gwz\" (UID: \"3ba60eb6-74cd-4416-9a15-d04ba2ef2b1e\") " pod="openshift-dns/dns-default-q6gwz"
Sep 08 01:59:22.502252 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:59:22.502175    2547 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 01:59:22.502252 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:59:22.502192    2547 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68cff75bfd-p6b52: secret "image-registry-tls" not found
Sep 08 01:59:22.502252 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:59:22.502238    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c7d9f993-10b4-4226-8193-a7599e0d3024-registry-tls podName:c7d9f993-10b4-4226-8193-a7599e0d3024 nodeName:}" failed. No retries permitted until 2026-09-08 01:59:23.002222538 +0000 UTC m=+34.525997614 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c7d9f993-10b4-4226-8193-a7599e0d3024-registry-tls") pod "image-registry-68cff75bfd-p6b52" (UID: "c7d9f993-10b4-4226-8193-a7599e0d3024") : secret "image-registry-tls" not found
Sep 08 01:59:22.502610 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:22.502399    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ba60eb6-74cd-4416-9a15-d04ba2ef2b1e-tmp-dir\") pod \"dns-default-q6gwz\" (UID: \"3ba60eb6-74cd-4416-9a15-d04ba2ef2b1e\") " pod="openshift-dns/dns-default-q6gwz"
Sep 08 01:59:22.502610 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:22.502401    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3ba60eb6-74cd-4416-9a15-d04ba2ef2b1e-config-volume\") pod \"dns-default-q6gwz\" (UID: \"3ba60eb6-74cd-4416-9a15-d04ba2ef2b1e\") " pod="openshift-dns/dns-default-q6gwz"
Sep 08 01:59:22.502690 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:22.502624    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c7d9f993-10b4-4226-8193-a7599e0d3024-ca-trust-extracted\") pod \"image-registry-68cff75bfd-p6b52\" (UID: \"c7d9f993-10b4-4226-8193-a7599e0d3024\") " pod="openshift-image-registry/image-registry-68cff75bfd-p6b52"
Sep 08 01:59:22.502811 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:22.502791    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c7d9f993-10b4-4226-8193-a7599e0d3024-trusted-ca\") pod \"image-registry-68cff75bfd-p6b52\" (UID: \"c7d9f993-10b4-4226-8193-a7599e0d3024\") " pod="openshift-image-registry/image-registry-68cff75bfd-p6b52"
Sep 08 01:59:22.503242 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:22.503218    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c7d9f993-10b4-4226-8193-a7599e0d3024-registry-certificates\") pod \"image-registry-68cff75bfd-p6b52\" (UID: \"c7d9f993-10b4-4226-8193-a7599e0d3024\") " pod="openshift-image-registry/image-registry-68cff75bfd-p6b52"
Sep 08 01:59:22.505991 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:22.505892    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c7d9f993-10b4-4226-8193-a7599e0d3024-installation-pull-secrets\") pod \"image-registry-68cff75bfd-p6b52\" (UID: \"c7d9f993-10b4-4226-8193-a7599e0d3024\") " pod="openshift-image-registry/image-registry-68cff75bfd-p6b52"
Sep 08 01:59:22.506055 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:22.505930    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c7d9f993-10b4-4226-8193-a7599e0d3024-image-registry-private-configuration\") pod \"image-registry-68cff75bfd-p6b52\" (UID: \"c7d9f993-10b4-4226-8193-a7599e0d3024\") " pod="openshift-image-registry/image-registry-68cff75bfd-p6b52"
Sep 08 01:59:22.506055 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:22.505963    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/444897cb-e238-4b58-a0ef-9adce681c307-cert\") pod \"ingress-canary-87nvd\" (UID: \"444897cb-e238-4b58-a0ef-9adce681c307\") " pod="openshift-ingress-canary/ingress-canary-87nvd"
Sep 08 01:59:22.511950 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:22.511928    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkztm\" (UniqueName: \"kubernetes.io/projected/c7d9f993-10b4-4226-8193-a7599e0d3024-kube-api-access-pkztm\") pod \"image-registry-68cff75bfd-p6b52\" (UID: \"c7d9f993-10b4-4226-8193-a7599e0d3024\") " pod="openshift-image-registry/image-registry-68cff75bfd-p6b52"
Sep 08 01:59:22.512871 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:22.512851    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hssd\" (UniqueName: \"kubernetes.io/projected/444897cb-e238-4b58-a0ef-9adce681c307-kube-api-access-6hssd\") pod \"ingress-canary-87nvd\" (UID: \"444897cb-e238-4b58-a0ef-9adce681c307\") " pod="openshift-ingress-canary/ingress-canary-87nvd"
Sep 08 01:59:22.513066 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:22.513049    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c7d9f993-10b4-4226-8193-a7599e0d3024-bound-sa-token\") pod \"image-registry-68cff75bfd-p6b52\" (UID: \"c7d9f993-10b4-4226-8193-a7599e0d3024\") " pod="openshift-image-registry/image-registry-68cff75bfd-p6b52"
Sep 08 01:59:22.513332 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:22.513312    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqfvn\" (UniqueName: \"kubernetes.io/projected/3ba60eb6-74cd-4416-9a15-d04ba2ef2b1e-kube-api-access-wqfvn\") pod \"dns-default-q6gwz\" (UID: \"3ba60eb6-74cd-4416-9a15-d04ba2ef2b1e\") " pod="openshift-dns/dns-default-q6gwz"
Sep 08 01:59:22.602867 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:22.602812    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-87nvd"
Sep 08 01:59:22.704054 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:22.704008    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/917f6c17-7902-48d3-a0f4-535e5b602d26-original-pull-secret\") pod \"global-pull-secret-syncer-l7hw8\" (UID: \"917f6c17-7902-48d3-a0f4-535e5b602d26\") " pod="kube-system/global-pull-secret-syncer-l7hw8"
Sep 08 01:59:22.704412 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:59:22.704207    2547 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 01:59:22.704412 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:22.704264    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b82c5af2-4c1c-41a9-8fa4-af8a85bcc8c6-metrics-certs\") pod \"network-metrics-daemon-jr45c\" (UID: \"b82c5af2-4c1c-41a9-8fa4-af8a85bcc8c6\") " pod="openshift-multus/network-metrics-daemon-jr45c"
Sep 08 01:59:22.704412 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:59:22.704280    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/917f6c17-7902-48d3-a0f4-535e5b602d26-original-pull-secret podName:917f6c17-7902-48d3-a0f4-535e5b602d26 nodeName:}" failed. No retries permitted until 2026-09-08 01:59:54.704261866 +0000 UTC m=+66.228036939 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/917f6c17-7902-48d3-a0f4-535e5b602d26-original-pull-secret") pod "global-pull-secret-syncer-l7hw8" (UID: "917f6c17-7902-48d3-a0f4-535e5b602d26") : object "kube-system"/"original-pull-secret" not registered
Sep 08 01:59:22.704412 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:59:22.704333    2547 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 01:59:22.704412 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:59:22.704369    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b82c5af2-4c1c-41a9-8fa4-af8a85bcc8c6-metrics-certs podName:b82c5af2-4c1c-41a9-8fa4-af8a85bcc8c6 nodeName:}" failed. No retries permitted until 2026-09-08 01:59:54.704355883 +0000 UTC m=+66.228130953 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b82c5af2-4c1c-41a9-8fa4-af8a85bcc8c6-metrics-certs") pod "network-metrics-daemon-jr45c" (UID: "b82c5af2-4c1c-41a9-8fa4-af8a85bcc8c6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 01:59:22.805296 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:22.805212    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dt4k\" (UniqueName: \"kubernetes.io/projected/430aad96-9d26-4055-8133-b7f50b0f2978-kube-api-access-5dt4k\") pod \"network-check-target-2v24n\" (UID: \"430aad96-9d26-4055-8133-b7f50b0f2978\") " pod="openshift-network-diagnostics/network-check-target-2v24n"
Sep 08 01:59:22.805439 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:59:22.805331    2547 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 01:59:22.805439 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:59:22.805345    2547 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 01:59:22.805439 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:59:22.805356    2547 projected.go:196] Error preparing data for projected volume kube-api-access-5dt4k for pod openshift-network-diagnostics/network-check-target-2v24n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 01:59:22.805439 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:59:22.805406    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/430aad96-9d26-4055-8133-b7f50b0f2978-kube-api-access-5dt4k podName:430aad96-9d26-4055-8133-b7f50b0f2978 nodeName:}" failed. No retries permitted until 2026-09-08 01:59:54.805394065 +0000 UTC m=+66.329169135 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-5dt4k" (UniqueName: "kubernetes.io/projected/430aad96-9d26-4055-8133-b7f50b0f2978-kube-api-access-5dt4k") pod "network-check-target-2v24n" (UID: "430aad96-9d26-4055-8133-b7f50b0f2978") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 01:59:22.914588 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:22.914551    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-87nvd"]
Sep 08 01:59:22.918332 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:59:22.918305    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod444897cb_e238_4b58_a0ef_9adce681c307.slice/crio-1eeb5273fb36ae9d579b1d9e74524078841caba0d721b1edb4d142c2c9151083 WatchSource:0}: Error finding container 1eeb5273fb36ae9d579b1d9e74524078841caba0d721b1edb4d142c2c9151083: Status 404 returned error can't find the container with id 1eeb5273fb36ae9d579b1d9e74524078841caba0d721b1edb4d142c2c9151083
Sep 08 01:59:23.006404 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:23.006364    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3ba60eb6-74cd-4416-9a15-d04ba2ef2b1e-metrics-tls\") pod \"dns-default-q6gwz\" (UID: \"3ba60eb6-74cd-4416-9a15-d04ba2ef2b1e\") " pod="openshift-dns/dns-default-q6gwz"
Sep 08 01:59:23.006404 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:23.006400    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c7d9f993-10b4-4226-8193-a7599e0d3024-registry-tls\") pod \"image-registry-68cff75bfd-p6b52\" (UID: \"c7d9f993-10b4-4226-8193-a7599e0d3024\") " pod="openshift-image-registry/image-registry-68cff75bfd-p6b52"
Sep 08 01:59:23.006947 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:59:23.006514    2547 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 01:59:23.006947 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:59:23.006578    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ba60eb6-74cd-4416-9a15-d04ba2ef2b1e-metrics-tls podName:3ba60eb6-74cd-4416-9a15-d04ba2ef2b1e nodeName:}" failed. No retries permitted until 2026-09-08 01:59:24.00655801 +0000 UTC m=+35.530333081 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3ba60eb6-74cd-4416-9a15-d04ba2ef2b1e-metrics-tls") pod "dns-default-q6gwz" (UID: "3ba60eb6-74cd-4416-9a15-d04ba2ef2b1e") : secret "dns-default-metrics-tls" not found
Sep 08 01:59:23.006947 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:59:23.006517    2547 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 01:59:23.006947 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:59:23.006623    2547 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68cff75bfd-p6b52: secret "image-registry-tls" not found
Sep 08 01:59:23.006947 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:59:23.006676    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c7d9f993-10b4-4226-8193-a7599e0d3024-registry-tls podName:c7d9f993-10b4-4226-8193-a7599e0d3024 nodeName:}" failed. No retries permitted until 2026-09-08 01:59:24.006662987 +0000 UTC m=+35.530438072 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c7d9f993-10b4-4226-8193-a7599e0d3024-registry-tls") pod "image-registry-68cff75bfd-p6b52" (UID: "c7d9f993-10b4-4226-8193-a7599e0d3024") : secret "image-registry-tls" not found
Sep 08 01:59:23.169106 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:23.169072    2547 generic.go:356] "Generic (PLEG): container finished" podID="2d7adf2a-2cfe-40ff-90d3-4886a747463a" containerID="a066fb69cbf0890d22d5fe1963073468d9bcb945c4dbbfbc4b07ac2e349e96ba" exitCode=0
Sep 08 01:59:23.169329 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:23.169162    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b4ms7" event={"ID":"2d7adf2a-2cfe-40ff-90d3-4886a747463a","Type":"ContainerDied","Data":"a066fb69cbf0890d22d5fe1963073468d9bcb945c4dbbfbc4b07ac2e349e96ba"}
Sep 08 01:59:23.170764 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:23.170741    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-87nvd" event={"ID":"444897cb-e238-4b58-a0ef-9adce681c307","Type":"ContainerStarted","Data":"1eeb5273fb36ae9d579b1d9e74524078841caba0d721b1edb4d142c2c9151083"}
Sep 08 01:59:23.473714 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:23.473624    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-2j82x"]
Sep 08 01:59:23.497660 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:23.497629    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-2j82x"]
Sep 08 01:59:23.497835 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:23.497720    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-2j82x"
Sep 08 01:59:23.499869 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:23.499845    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 08 01:59:23.500005 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:23.499912    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 08 01:59:23.500005 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:23.499968    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-2jbsk\""
Sep 08 01:59:23.611350 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:23.611313    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cg88l\" (UniqueName: \"kubernetes.io/projected/90cc711b-31b7-4417-8bb6-9e827e71bb28-kube-api-access-cg88l\") pod \"migrator-fdbb8fdff-2j82x\" (UID: \"90cc711b-31b7-4417-8bb6-9e827e71bb28\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-2j82x"
Sep 08 01:59:23.712624 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:23.712591    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg88l\" (UniqueName: \"kubernetes.io/projected/90cc711b-31b7-4417-8bb6-9e827e71bb28-kube-api-access-cg88l\") pod \"migrator-fdbb8fdff-2j82x\" (UID: \"90cc711b-31b7-4417-8bb6-9e827e71bb28\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-2j82x"
Sep 08 01:59:23.721010 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:23.720976    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg88l\" (UniqueName: \"kubernetes.io/projected/90cc711b-31b7-4417-8bb6-9e827e71bb28-kube-api-access-cg88l\") pod \"migrator-fdbb8fdff-2j82x\" (UID: \"90cc711b-31b7-4417-8bb6-9e827e71bb28\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-2j82x"
Sep 08 01:59:23.808407 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:23.808321    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-2j82x"
Sep 08 01:59:23.987890 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:23.987853    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l7hw8"
Sep 08 01:59:23.988074 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:23.987858    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jr45c"
Sep 08 01:59:23.988184 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:23.987860    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2v24n"
Sep 08 01:59:23.990311 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:23.990274    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 08 01:59:23.990498 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:23.990474    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 08 01:59:23.991079 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:23.991059    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 08 01:59:23.991079 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:23.991077    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-lxnlh\""
Sep 08 01:59:23.991252 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:23.991062    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-s2v8c\""
Sep 08 01:59:23.991252 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:23.991062    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 08 01:59:24.015959 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:24.015929    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3ba60eb6-74cd-4416-9a15-d04ba2ef2b1e-metrics-tls\") pod \"dns-default-q6gwz\" (UID: \"3ba60eb6-74cd-4416-9a15-d04ba2ef2b1e\") " pod="openshift-dns/dns-default-q6gwz"
Sep 08 01:59:24.016556 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:24.015972    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c7d9f993-10b4-4226-8193-a7599e0d3024-registry-tls\") pod \"image-registry-68cff75bfd-p6b52\" (UID: \"c7d9f993-10b4-4226-8193-a7599e0d3024\") " pod="openshift-image-registry/image-registry-68cff75bfd-p6b52"
Sep 08 01:59:24.016556 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:59:24.016078    2547 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 01:59:24.016556 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:59:24.016119    2547 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 01:59:24.016556 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:59:24.016154    2547 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68cff75bfd-p6b52: secret "image-registry-tls" not found
Sep 08 01:59:24.016556 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:59:24.016166    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ba60eb6-74cd-4416-9a15-d04ba2ef2b1e-metrics-tls podName:3ba60eb6-74cd-4416-9a15-d04ba2ef2b1e nodeName:}" failed. No retries permitted until 2026-09-08 01:59:26.016120667 +0000 UTC m=+37.539895725 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3ba60eb6-74cd-4416-9a15-d04ba2ef2b1e-metrics-tls") pod "dns-default-q6gwz" (UID: "3ba60eb6-74cd-4416-9a15-d04ba2ef2b1e") : secret "dns-default-metrics-tls" not found
Sep 08 01:59:24.016556 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:59:24.016203    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c7d9f993-10b4-4226-8193-a7599e0d3024-registry-tls podName:c7d9f993-10b4-4226-8193-a7599e0d3024 nodeName:}" failed. No retries permitted until 2026-09-08 01:59:26.016187226 +0000 UTC m=+37.539962284 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c7d9f993-10b4-4226-8193-a7599e0d3024-registry-tls") pod "image-registry-68cff75bfd-p6b52" (UID: "c7d9f993-10b4-4226-8193-a7599e0d3024") : secret "image-registry-tls" not found
Sep 08 01:59:24.104150 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:24.104058    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-2j82x"]
Sep 08 01:59:24.108053 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:59:24.108007    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90cc711b_31b7_4417_8bb6_9e827e71bb28.slice/crio-77a73850f63443315de96536ad8d3756dc58abea41b5839a940024a36a8166cc WatchSource:0}: Error finding container 77a73850f63443315de96536ad8d3756dc58abea41b5839a940024a36a8166cc: Status 404 returned error can't find the container with id 77a73850f63443315de96536ad8d3756dc58abea41b5839a940024a36a8166cc
Sep 08 01:59:24.178192 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:24.178152    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b4ms7" event={"ID":"2d7adf2a-2cfe-40ff-90d3-4886a747463a","Type":"ContainerStarted","Data":"f13487c3ab885cda835b25ce1b74adeb02ee48e150ca26b02851061dcfaa8e63"}
Sep 08 01:59:24.179351 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:24.179316    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-2j82x" event={"ID":"90cc711b-31b7-4417-8bb6-9e827e71bb28","Type":"ContainerStarted","Data":"77a73850f63443315de96536ad8d3756dc58abea41b5839a940024a36a8166cc"}
Sep 08 01:59:24.197752 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:24.197681    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-b4ms7" podStartSLOduration=7.911221602 podStartE2EDuration="35.197664707s" totalImagesPullingTime="27.286443105s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 01:58:49 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 01:59:24.196097689 +0000 UTC m=+35.719872774" watchObservedRunningTime="2026-09-08 01:59:24.197664707 +0000 UTC m=+35.721439788"
Sep 08 01:59:24.731536 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:24.731504    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-sts5m"]
Sep 08 01:59:24.757809 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:24.757774    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-sts5m"]
Sep 08 01:59:24.757976 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:24.757915    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-sts5m"
Sep 08 01:59:24.760329 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:24.760162    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 08 01:59:24.760329 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:24.760210    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 08 01:59:24.760329 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:24.760231    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 08 01:59:24.760329 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:24.760244    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xc9ld\""
Sep 08 01:59:24.760329 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:24.760247    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 08 01:59:24.821825 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:24.821789    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/278c1af4-cf3a-49c4-80b8-a40b5b6ddb72-crio-socket\") pod \"insights-runtime-extractor-sts5m\" (UID: \"278c1af4-cf3a-49c4-80b8-a40b5b6ddb72\") " pod="openshift-insights/insights-runtime-extractor-sts5m"
Sep 08 01:59:24.821825 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:24.821838    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/278c1af4-cf3a-49c4-80b8-a40b5b6ddb72-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sts5m\" (UID: \"278c1af4-cf3a-49c4-80b8-a40b5b6ddb72\") " pod="openshift-insights/insights-runtime-extractor-sts5m"
Sep 08 01:59:24.822033 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:24.821857    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/278c1af4-cf3a-49c4-80b8-a40b5b6ddb72-data-volume\") pod \"insights-runtime-extractor-sts5m\" (UID: \"278c1af4-cf3a-49c4-80b8-a40b5b6ddb72\") " pod="openshift-insights/insights-runtime-extractor-sts5m"
Sep 08 01:59:24.822033 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:24.821933    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcfgr\" (UniqueName: \"kubernetes.io/projected/278c1af4-cf3a-49c4-80b8-a40b5b6ddb72-kube-api-access-pcfgr\") pod \"insights-runtime-extractor-sts5m\" (UID: \"278c1af4-cf3a-49c4-80b8-a40b5b6ddb72\") " pod="openshift-insights/insights-runtime-extractor-sts5m"
Sep 08 01:59:24.822119 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:24.822034    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/278c1af4-cf3a-49c4-80b8-a40b5b6ddb72-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sts5m\" (UID: \"278c1af4-cf3a-49c4-80b8-a40b5b6ddb72\") " pod="openshift-insights/insights-runtime-extractor-sts5m"
Sep 08 01:59:24.922617 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:24.922582    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/278c1af4-cf3a-49c4-80b8-a40b5b6ddb72-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sts5m\" (UID: \"278c1af4-cf3a-49c4-80b8-a40b5b6ddb72\") " pod="openshift-insights/insights-runtime-extractor-sts5m"
Sep 08 01:59:24.922617 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:24.922624    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/278c1af4-cf3a-49c4-80b8-a40b5b6ddb72-data-volume\") pod \"insights-runtime-extractor-sts5m\" (UID: \"278c1af4-cf3a-49c4-80b8-a40b5b6ddb72\") " pod="openshift-insights/insights-runtime-extractor-sts5m"
Sep 08 01:59:24.922841 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:24.922643    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcfgr\" (UniqueName: \"kubernetes.io/projected/278c1af4-cf3a-49c4-80b8-a40b5b6ddb72-kube-api-access-pcfgr\") pod \"insights-runtime-extractor-sts5m\" (UID: \"278c1af4-cf3a-49c4-80b8-a40b5b6ddb72\") " pod="openshift-insights/insights-runtime-extractor-sts5m"
Sep 08 01:59:24.922841 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:59:24.922717    2547 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 01:59:24.922841 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:59:24.922819    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/278c1af4-cf3a-49c4-80b8-a40b5b6ddb72-kube-rbac-proxy-cm podName:278c1af4-cf3a-49c4-80b8-a40b5b6ddb72 nodeName:}" failed. No retries permitted until 2026-09-08 01:59:25.422795687 +0000 UTC m=+36.946570765 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/278c1af4-cf3a-49c4-80b8-a40b5b6ddb72-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sts5m" (UID: "278c1af4-cf3a-49c4-80b8-a40b5b6ddb72") : configmap "kube-rbac-proxy" not found
Sep 08 01:59:24.922992 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:24.922967    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/278c1af4-cf3a-49c4-80b8-a40b5b6ddb72-data-volume\") pod \"insights-runtime-extractor-sts5m\" (UID: \"278c1af4-cf3a-49c4-80b8-a40b5b6ddb72\") " pod="openshift-insights/insights-runtime-extractor-sts5m"
Sep 08 01:59:24.923182 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:24.923008    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/278c1af4-cf3a-49c4-80b8-a40b5b6ddb72-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sts5m\" (UID: \"278c1af4-cf3a-49c4-80b8-a40b5b6ddb72\") " pod="openshift-insights/insights-runtime-extractor-sts5m"
Sep 08 01:59:24.923182 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:59:24.923088    2547 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 01:59:24.923182 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:24.923090    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/278c1af4-cf3a-49c4-80b8-a40b5b6ddb72-crio-socket\") pod \"insights-runtime-extractor-sts5m\" (UID: \"278c1af4-cf3a-49c4-80b8-a40b5b6ddb72\") " pod="openshift-insights/insights-runtime-extractor-sts5m"
Sep 08 01:59:24.923182 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:59:24.923159    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/278c1af4-cf3a-49c4-80b8-a40b5b6ddb72-insights-runtime-extractor-tls podName:278c1af4-cf3a-49c4-80b8-a40b5b6ddb72 nodeName:}" failed. No retries permitted until 2026-09-08 01:59:25.423131225 +0000 UTC m=+36.946906287 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/278c1af4-cf3a-49c4-80b8-a40b5b6ddb72-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sts5m" (UID: "278c1af4-cf3a-49c4-80b8-a40b5b6ddb72") : secret "insights-runtime-extractor-tls" not found
Sep 08 01:59:24.923389 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:24.923290    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/278c1af4-cf3a-49c4-80b8-a40b5b6ddb72-crio-socket\") pod \"insights-runtime-extractor-sts5m\" (UID: \"278c1af4-cf3a-49c4-80b8-a40b5b6ddb72\") " pod="openshift-insights/insights-runtime-extractor-sts5m"
Sep 08 01:59:24.933719 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:24.933690    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcfgr\" (UniqueName: \"kubernetes.io/projected/278c1af4-cf3a-49c4-80b8-a40b5b6ddb72-kube-api-access-pcfgr\") pod \"insights-runtime-extractor-sts5m\" (UID: \"278c1af4-cf3a-49c4-80b8-a40b5b6ddb72\") " pod="openshift-insights/insights-runtime-extractor-sts5m"
Sep 08 01:59:25.428098 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:25.428059    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/278c1af4-cf3a-49c4-80b8-a40b5b6ddb72-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sts5m\" (UID: \"278c1af4-cf3a-49c4-80b8-a40b5b6ddb72\") " pod="openshift-insights/insights-runtime-extractor-sts5m"
Sep 08 01:59:25.428445 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:25.428198    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/278c1af4-cf3a-49c4-80b8-a40b5b6ddb72-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sts5m\" (UID: \"278c1af4-cf3a-49c4-80b8-a40b5b6ddb72\") " pod="openshift-insights/insights-runtime-extractor-sts5m"
Sep 08 01:59:25.428445 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:59:25.428219    2547 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 01:59:25.428445 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:59:25.428278    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/278c1af4-cf3a-49c4-80b8-a40b5b6ddb72-insights-runtime-extractor-tls podName:278c1af4-cf3a-49c4-80b8-a40b5b6ddb72 nodeName:}" failed. No retries permitted until 2026-09-08 01:59:26.428258993 +0000 UTC m=+37.952034065 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/278c1af4-cf3a-49c4-80b8-a40b5b6ddb72-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sts5m" (UID: "278c1af4-cf3a-49c4-80b8-a40b5b6ddb72") : secret "insights-runtime-extractor-tls" not found
Sep 08 01:59:25.428445 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:59:25.428306    2547 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 01:59:25.428445 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:59:25.428366    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/278c1af4-cf3a-49c4-80b8-a40b5b6ddb72-kube-rbac-proxy-cm podName:278c1af4-cf3a-49c4-80b8-a40b5b6ddb72 nodeName:}" failed. No retries permitted until 2026-09-08 01:59:26.428353349 +0000 UTC m=+37.952128429 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/278c1af4-cf3a-49c4-80b8-a40b5b6ddb72-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sts5m" (UID: "278c1af4-cf3a-49c4-80b8-a40b5b6ddb72") : configmap "kube-rbac-proxy" not found
Sep 08 01:59:25.972921 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:25.972889    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-75d99c484-77qtz"]
Sep 08 01:59:25.984458 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:25.984427    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-77qtz"
Sep 08 01:59:25.985006 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:25.984944    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-77qtz"]
Sep 08 01:59:25.986763 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:25.986743    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-bbhpn\""
Sep 08 01:59:25.987030 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:25.987009    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 08 01:59:25.987095 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:25.987010    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 08 01:59:25.987795 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:25.987597    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 08 01:59:25.987795 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:25.987604    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 08 01:59:26.031893 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:26.031859    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3ba60eb6-74cd-4416-9a15-d04ba2ef2b1e-metrics-tls\") pod \"dns-default-q6gwz\" (UID: \"3ba60eb6-74cd-4416-9a15-d04ba2ef2b1e\") " pod="openshift-dns/dns-default-q6gwz"
Sep 08 01:59:26.031893 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:26.031894    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c7d9f993-10b4-4226-8193-a7599e0d3024-registry-tls\") pod \"image-registry-68cff75bfd-p6b52\" (UID: \"c7d9f993-10b4-4226-8193-a7599e0d3024\") " pod="openshift-image-registry/image-registry-68cff75bfd-p6b52"
Sep 08 01:59:26.032129 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:59:26.032029    2547 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 01:59:26.032129 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:59:26.032054    2547 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 01:59:26.032129 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:59:26.032066    2547 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68cff75bfd-p6b52: secret "image-registry-tls" not found
Sep 08 01:59:26.032129 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:59:26.032105    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ba60eb6-74cd-4416-9a15-d04ba2ef2b1e-metrics-tls podName:3ba60eb6-74cd-4416-9a15-d04ba2ef2b1e nodeName:}" failed. No retries permitted until 2026-09-08 01:59:30.032083298 +0000 UTC m=+41.555858357 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3ba60eb6-74cd-4416-9a15-d04ba2ef2b1e-metrics-tls") pod "dns-default-q6gwz" (UID: "3ba60eb6-74cd-4416-9a15-d04ba2ef2b1e") : secret "dns-default-metrics-tls" not found
Sep 08 01:59:26.032129 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:59:26.032125    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c7d9f993-10b4-4226-8193-a7599e0d3024-registry-tls podName:c7d9f993-10b4-4226-8193-a7599e0d3024 nodeName:}" failed. No retries permitted until 2026-09-08 01:59:30.032115525 +0000 UTC m=+41.555890590 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c7d9f993-10b4-4226-8193-a7599e0d3024-registry-tls") pod "image-registry-68cff75bfd-p6b52" (UID: "c7d9f993-10b4-4226-8193-a7599e0d3024") : secret "image-registry-tls" not found
Sep 08 01:59:26.132729 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:26.132650    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/26d02cd5-09f4-46c2-912e-4fc9ef20599a-signing-cabundle\") pod \"service-ca-75d99c484-77qtz\" (UID: \"26d02cd5-09f4-46c2-912e-4fc9ef20599a\") " pod="openshift-service-ca/service-ca-75d99c484-77qtz"
Sep 08 01:59:26.132729 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:26.132692    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmdl4\" (UniqueName: \"kubernetes.io/projected/26d02cd5-09f4-46c2-912e-4fc9ef20599a-kube-api-access-kmdl4\") pod \"service-ca-75d99c484-77qtz\" (UID: \"26d02cd5-09f4-46c2-912e-4fc9ef20599a\") " pod="openshift-service-ca/service-ca-75d99c484-77qtz"
Sep 08 01:59:26.132943 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:26.132800    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/26d02cd5-09f4-46c2-912e-4fc9ef20599a-signing-key\") pod \"service-ca-75d99c484-77qtz\" (UID: \"26d02cd5-09f4-46c2-912e-4fc9ef20599a\") " pod="openshift-service-ca/service-ca-75d99c484-77qtz"
Sep 08 01:59:26.185613 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:26.185582    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-87nvd" event={"ID":"444897cb-e238-4b58-a0ef-9adce681c307","Type":"ContainerStarted","Data":"b03d8b2467764625bb60298fb8ae6b9fe27fdff92c0aaa0725d7b97dd825a9bd"}
Sep 08 01:59:26.199821 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:26.199770    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-87nvd" podStartSLOduration=1.697354994 podStartE2EDuration="4.199755059s" totalImagesPullingTime="2.502400065s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 01:59:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 01:59:26.199388547 +0000 UTC m=+37.723163640" watchObservedRunningTime="2026-09-08 01:59:26.199755059 +0000 UTC m=+37.723530139"
Sep 08 01:59:26.233775 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:26.233742    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/26d02cd5-09f4-46c2-912e-4fc9ef20599a-signing-key\") pod \"service-ca-75d99c484-77qtz\" (UID: \"26d02cd5-09f4-46c2-912e-4fc9ef20599a\") " pod="openshift-service-ca/service-ca-75d99c484-77qtz"
Sep 08 01:59:26.233931 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:26.233881    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/26d02cd5-09f4-46c2-912e-4fc9ef20599a-signing-cabundle\") pod \"service-ca-75d99c484-77qtz\" (UID: \"26d02cd5-09f4-46c2-912e-4fc9ef20599a\") " pod="openshift-service-ca/service-ca-75d99c484-77qtz"
Sep 08 01:59:26.233931 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:26.233906    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmdl4\" (UniqueName: \"kubernetes.io/projected/26d02cd5-09f4-46c2-912e-4fc9ef20599a-kube-api-access-kmdl4\") pod \"service-ca-75d99c484-77qtz\" (UID: \"26d02cd5-09f4-46c2-912e-4fc9ef20599a\") " pod="openshift-service-ca/service-ca-75d99c484-77qtz"
Sep 08 01:59:26.234814 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:26.234787    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/26d02cd5-09f4-46c2-912e-4fc9ef20599a-signing-cabundle\") pod \"service-ca-75d99c484-77qtz\" (UID: \"26d02cd5-09f4-46c2-912e-4fc9ef20599a\") " pod="openshift-service-ca/service-ca-75d99c484-77qtz"
Sep 08 01:59:26.236528 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:26.236509    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/26d02cd5-09f4-46c2-912e-4fc9ef20599a-signing-key\") pod \"service-ca-75d99c484-77qtz\" (UID: \"26d02cd5-09f4-46c2-912e-4fc9ef20599a\") " pod="openshift-service-ca/service-ca-75d99c484-77qtz"
Sep 08 01:59:26.241327 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:26.241301    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmdl4\" (UniqueName: \"kubernetes.io/projected/26d02cd5-09f4-46c2-912e-4fc9ef20599a-kube-api-access-kmdl4\") pod \"service-ca-75d99c484-77qtz\" (UID: \"26d02cd5-09f4-46c2-912e-4fc9ef20599a\") " pod="openshift-service-ca/service-ca-75d99c484-77qtz"
Sep 08 01:59:26.295762 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:26.295726    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-77qtz"
Sep 08 01:59:26.436191 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:26.436150    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/278c1af4-cf3a-49c4-80b8-a40b5b6ddb72-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sts5m\" (UID: \"278c1af4-cf3a-49c4-80b8-a40b5b6ddb72\") " pod="openshift-insights/insights-runtime-extractor-sts5m"
Sep 08 01:59:26.436602 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:59:26.436240    2547 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 01:59:26.436602 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:26.436259    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/278c1af4-cf3a-49c4-80b8-a40b5b6ddb72-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sts5m\" (UID: \"278c1af4-cf3a-49c4-80b8-a40b5b6ddb72\") " pod="openshift-insights/insights-runtime-extractor-sts5m"
Sep 08 01:59:26.436602 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:59:26.436303    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/278c1af4-cf3a-49c4-80b8-a40b5b6ddb72-insights-runtime-extractor-tls podName:278c1af4-cf3a-49c4-80b8-a40b5b6ddb72 nodeName:}" failed. No retries permitted until 2026-09-08 01:59:28.436288467 +0000 UTC m=+39.960063525 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/278c1af4-cf3a-49c4-80b8-a40b5b6ddb72-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sts5m" (UID: "278c1af4-cf3a-49c4-80b8-a40b5b6ddb72") : secret "insights-runtime-extractor-tls" not found
Sep 08 01:59:26.436602 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:59:26.436344    2547 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 01:59:26.436602 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:59:26.436400    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/278c1af4-cf3a-49c4-80b8-a40b5b6ddb72-kube-rbac-proxy-cm podName:278c1af4-cf3a-49c4-80b8-a40b5b6ddb72 nodeName:}" failed. No retries permitted until 2026-09-08 01:59:28.436384804 +0000 UTC m=+39.960159862 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/278c1af4-cf3a-49c4-80b8-a40b5b6ddb72-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sts5m" (UID: "278c1af4-cf3a-49c4-80b8-a40b5b6ddb72") : configmap "kube-rbac-proxy" not found
Sep 08 01:59:26.576095 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:26.575875    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-77qtz"]
Sep 08 01:59:26.579446 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:59:26.579422    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26d02cd5_09f4_46c2_912e_4fc9ef20599a.slice/crio-5fe8ec5d913fab9a4bab779b58d18d812e2948427f0344fd0eaf99d6fbf884ee WatchSource:0}: Error finding container 5fe8ec5d913fab9a4bab779b58d18d812e2948427f0344fd0eaf99d6fbf884ee: Status 404 returned error can't find the container with id 5fe8ec5d913fab9a4bab779b58d18d812e2948427f0344fd0eaf99d6fbf884ee
Sep 08 01:59:26.889293 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:26.889269    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4s9cm_0536c61b-c71c-41d9-8b2c-18228a7ddff4/dns-node-resolver/0.log"
Sep 08 01:59:27.191043 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:27.190962    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-2j82x" event={"ID":"90cc711b-31b7-4417-8bb6-9e827e71bb28","Type":"ContainerStarted","Data":"ea0ba6ab4eae955a722ffa16339188d2ba6ec14ec4ed2b70be27fd294109f326"}
Sep 08 01:59:27.191043 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:27.191029    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-2j82x" event={"ID":"90cc711b-31b7-4417-8bb6-9e827e71bb28","Type":"ContainerStarted","Data":"4b26096c9cc3468261bd349fd8af68c422fae3e06471e2d6c7538c6a5ce2921e"}
Sep 08 01:59:27.192298 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:27.192270    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-77qtz" event={"ID":"26d02cd5-09f4-46c2-912e-4fc9ef20599a","Type":"ContainerStarted","Data":"5fe8ec5d913fab9a4bab779b58d18d812e2948427f0344fd0eaf99d6fbf884ee"}
Sep 08 01:59:27.205757 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:27.205709    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-2j82x" podStartSLOduration=1.865039977 podStartE2EDuration="4.205695923s" totalImagesPullingTime="2.340655946s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 01:59:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 01:59:27.205274851 +0000 UTC m=+38.729049943" watchObservedRunningTime="2026-09-08 01:59:27.205695923 +0000 UTC m=+38.729471003"
Sep 08 01:59:28.091608 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:28.091581    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-pl5qh_ae4a55d0-ad38-48a9-9a36-279949e2e4c5/node-ca/0.log"
Sep 08 01:59:28.455753 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:28.455715    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/278c1af4-cf3a-49c4-80b8-a40b5b6ddb72-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sts5m\" (UID: \"278c1af4-cf3a-49c4-80b8-a40b5b6ddb72\") " pod="openshift-insights/insights-runtime-extractor-sts5m"
Sep 08 01:59:28.455907 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:28.455821    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/278c1af4-cf3a-49c4-80b8-a40b5b6ddb72-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sts5m\" (UID: \"278c1af4-cf3a-49c4-80b8-a40b5b6ddb72\") " pod="openshift-insights/insights-runtime-extractor-sts5m"
Sep 08 01:59:28.455907 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:59:28.455875    2547 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 01:59:28.455907 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:59:28.455904    2547 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 01:59:28.455993 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:59:28.455945    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/278c1af4-cf3a-49c4-80b8-a40b5b6ddb72-insights-runtime-extractor-tls podName:278c1af4-cf3a-49c4-80b8-a40b5b6ddb72 nodeName:}" failed. No retries permitted until 2026-09-08 01:59:32.455927419 +0000 UTC m=+43.979702492 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/278c1af4-cf3a-49c4-80b8-a40b5b6ddb72-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sts5m" (UID: "278c1af4-cf3a-49c4-80b8-a40b5b6ddb72") : secret "insights-runtime-extractor-tls" not found
Sep 08 01:59:28.455993 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:59:28.455959    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/278c1af4-cf3a-49c4-80b8-a40b5b6ddb72-kube-rbac-proxy-cm podName:278c1af4-cf3a-49c4-80b8-a40b5b6ddb72 nodeName:}" failed. No retries permitted until 2026-09-08 01:59:32.455953764 +0000 UTC m=+43.979728821 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/278c1af4-cf3a-49c4-80b8-a40b5b6ddb72-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sts5m" (UID: "278c1af4-cf3a-49c4-80b8-a40b5b6ddb72") : configmap "kube-rbac-proxy" not found
Sep 08 01:59:28.689321 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:28.689290    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-87nvd_444897cb-e238-4b58-a0ef-9adce681c307/serve-healthcheck-canary/0.log"
Sep 08 01:59:29.089673 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:29.089599    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-2j82x_90cc711b-31b7-4417-8bb6-9e827e71bb28/migrator/0.log"
Sep 08 01:59:29.199172 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:29.199122    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-77qtz" event={"ID":"26d02cd5-09f4-46c2-912e-4fc9ef20599a","Type":"ContainerStarted","Data":"840eb7cdb0b57d7795741cdeda5d65e77860a770ec061b95d3889e3b40c76123"}
Sep 08 01:59:29.214168 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:29.214103    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-75d99c484-77qtz" podStartSLOduration=2.001523222 podStartE2EDuration="4.214086094s" totalImagesPullingTime="2.212562872s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 01:59:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 01:59:29.213458782 +0000 UTC m=+40.737233875" watchObservedRunningTime="2026-09-08 01:59:29.214086094 +0000 UTC m=+40.737861174"
Sep 08 01:59:29.289097 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:29.289064    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-2j82x_90cc711b-31b7-4417-8bb6-9e827e71bb28/graceful-termination/0.log"
Sep 08 01:59:30.066925 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:30.066887    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3ba60eb6-74cd-4416-9a15-d04ba2ef2b1e-metrics-tls\") pod \"dns-default-q6gwz\" (UID: \"3ba60eb6-74cd-4416-9a15-d04ba2ef2b1e\") " pod="openshift-dns/dns-default-q6gwz"
Sep 08 01:59:30.066925 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:30.066921    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c7d9f993-10b4-4226-8193-a7599e0d3024-registry-tls\") pod \"image-registry-68cff75bfd-p6b52\" (UID: \"c7d9f993-10b4-4226-8193-a7599e0d3024\") " pod="openshift-image-registry/image-registry-68cff75bfd-p6b52"
Sep 08 01:59:30.067121 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:59:30.067056    2547 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 01:59:30.067121 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:59:30.067071    2547 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68cff75bfd-p6b52: secret "image-registry-tls" not found
Sep 08 01:59:30.067121 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:59:30.067054    2547 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 01:59:30.067121 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:59:30.067122    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c7d9f993-10b4-4226-8193-a7599e0d3024-registry-tls podName:c7d9f993-10b4-4226-8193-a7599e0d3024 nodeName:}" failed. No retries permitted until 2026-09-08 01:59:38.067101522 +0000 UTC m=+49.590876580 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c7d9f993-10b4-4226-8193-a7599e0d3024-registry-tls") pod "image-registry-68cff75bfd-p6b52" (UID: "c7d9f993-10b4-4226-8193-a7599e0d3024") : secret "image-registry-tls" not found
Sep 08 01:59:30.067264 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:59:30.067196    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ba60eb6-74cd-4416-9a15-d04ba2ef2b1e-metrics-tls podName:3ba60eb6-74cd-4416-9a15-d04ba2ef2b1e nodeName:}" failed. No retries permitted until 2026-09-08 01:59:38.067168476 +0000 UTC m=+49.590943534 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3ba60eb6-74cd-4416-9a15-d04ba2ef2b1e-metrics-tls") pod "dns-default-q6gwz" (UID: "3ba60eb6-74cd-4416-9a15-d04ba2ef2b1e") : secret "dns-default-metrics-tls" not found
Sep 08 01:59:32.488860 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:32.488815    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/278c1af4-cf3a-49c4-80b8-a40b5b6ddb72-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sts5m\" (UID: \"278c1af4-cf3a-49c4-80b8-a40b5b6ddb72\") " pod="openshift-insights/insights-runtime-extractor-sts5m"
Sep 08 01:59:32.489240 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:32.488902    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/278c1af4-cf3a-49c4-80b8-a40b5b6ddb72-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sts5m\" (UID: \"278c1af4-cf3a-49c4-80b8-a40b5b6ddb72\") " pod="openshift-insights/insights-runtime-extractor-sts5m"
Sep 08 01:59:32.489240 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:59:32.488969    2547 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 01:59:32.489240 ip-10-0-142-78 kubenswrapper[2547]: E0908 01:59:32.489053    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/278c1af4-cf3a-49c4-80b8-a40b5b6ddb72-insights-runtime-extractor-tls podName:278c1af4-cf3a-49c4-80b8-a40b5b6ddb72 nodeName:}" failed. No retries permitted until 2026-09-08 01:59:40.489035898 +0000 UTC m=+52.012810986 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/278c1af4-cf3a-49c4-80b8-a40b5b6ddb72-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sts5m" (UID: "278c1af4-cf3a-49c4-80b8-a40b5b6ddb72") : secret "insights-runtime-extractor-tls" not found
Sep 08 01:59:32.489382 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:32.489366    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/278c1af4-cf3a-49c4-80b8-a40b5b6ddb72-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sts5m\" (UID: \"278c1af4-cf3a-49c4-80b8-a40b5b6ddb72\") " pod="openshift-insights/insights-runtime-extractor-sts5m"
Sep 08 01:59:38.130803 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:38.130758    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3ba60eb6-74cd-4416-9a15-d04ba2ef2b1e-metrics-tls\") pod \"dns-default-q6gwz\" (UID: \"3ba60eb6-74cd-4416-9a15-d04ba2ef2b1e\") " pod="openshift-dns/dns-default-q6gwz"
Sep 08 01:59:38.130803 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:38.130800    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c7d9f993-10b4-4226-8193-a7599e0d3024-registry-tls\") pod \"image-registry-68cff75bfd-p6b52\" (UID: \"c7d9f993-10b4-4226-8193-a7599e0d3024\") " pod="openshift-image-registry/image-registry-68cff75bfd-p6b52"
Sep 08 01:59:38.133882 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:38.133858    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3ba60eb6-74cd-4416-9a15-d04ba2ef2b1e-metrics-tls\") pod \"dns-default-q6gwz\" (UID: \"3ba60eb6-74cd-4416-9a15-d04ba2ef2b1e\") " pod="openshift-dns/dns-default-q6gwz"
Sep 08 01:59:38.133997 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:38.133974    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c7d9f993-10b4-4226-8193-a7599e0d3024-registry-tls\") pod \"image-registry-68cff75bfd-p6b52\" (UID: \"c7d9f993-10b4-4226-8193-a7599e0d3024\") " pod="openshift-image-registry/image-registry-68cff75bfd-p6b52"
Sep 08 01:59:38.219014 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:38.218973    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-q6gwz"
Sep 08 01:59:38.233909 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:38.233871    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68cff75bfd-p6b52"
Sep 08 01:59:38.509188 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:38.509156    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-q6gwz"]
Sep 08 01:59:38.512560 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:59:38.512535    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ba60eb6_74cd_4416_9a15_d04ba2ef2b1e.slice/crio-7de9425e2a0977b33476b2a2ee710181f765efc6fbfcd222b1b8f59a39618b8b WatchSource:0}: Error finding container 7de9425e2a0977b33476b2a2ee710181f765efc6fbfcd222b1b8f59a39618b8b: Status 404 returned error can't find the container with id 7de9425e2a0977b33476b2a2ee710181f765efc6fbfcd222b1b8f59a39618b8b
Sep 08 01:59:38.534844 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:38.534812    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-68cff75bfd-p6b52"]
Sep 08 01:59:38.537914 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:59:38.537886    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7d9f993_10b4_4226_8193_a7599e0d3024.slice/crio-f144940c7d48fefc099c1ecf97efdd3f2526fe4a3aaece270839646d6807fb12 WatchSource:0}: Error finding container f144940c7d48fefc099c1ecf97efdd3f2526fe4a3aaece270839646d6807fb12: Status 404 returned error can't find the container with id f144940c7d48fefc099c1ecf97efdd3f2526fe4a3aaece270839646d6807fb12
Sep 08 01:59:39.224716 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:39.224680    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-q6gwz" event={"ID":"3ba60eb6-74cd-4416-9a15-d04ba2ef2b1e","Type":"ContainerStarted","Data":"7de9425e2a0977b33476b2a2ee710181f765efc6fbfcd222b1b8f59a39618b8b"}
Sep 08 01:59:39.226677 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:39.226644    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-68cff75bfd-p6b52" event={"ID":"c7d9f993-10b4-4226-8193-a7599e0d3024","Type":"ContainerStarted","Data":"79405026c9980ab78669ba6e905ae0c2b051d55e969a7ab47f3fb3cd9f71c113"}
Sep 08 01:59:39.226798 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:39.226689    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-68cff75bfd-p6b52" event={"ID":"c7d9f993-10b4-4226-8193-a7599e0d3024","Type":"ContainerStarted","Data":"f144940c7d48fefc099c1ecf97efdd3f2526fe4a3aaece270839646d6807fb12"}
Sep 08 01:59:39.226798 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:39.226767    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-68cff75bfd-p6b52"
Sep 08 01:59:39.245939 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:39.245877    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-68cff75bfd-p6b52" podStartSLOduration=52.245857392 podStartE2EDuration="52.245857392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 01:58:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 01:59:39.245050806 +0000 UTC m=+50.768825883" watchObservedRunningTime="2026-09-08 01:59:39.245857392 +0000 UTC m=+50.769632472"
Sep 08 01:59:40.234086 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:40.234048    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-q6gwz" event={"ID":"3ba60eb6-74cd-4416-9a15-d04ba2ef2b1e","Type":"ContainerStarted","Data":"06c0193d6ad9913f8c785193c76ba23da61e8ca5eddb6885fc85ac5d9089937e"}
Sep 08 01:59:40.553362 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:40.553321    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/278c1af4-cf3a-49c4-80b8-a40b5b6ddb72-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sts5m\" (UID: \"278c1af4-cf3a-49c4-80b8-a40b5b6ddb72\") " pod="openshift-insights/insights-runtime-extractor-sts5m"
Sep 08 01:59:40.555613 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:40.555582    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/278c1af4-cf3a-49c4-80b8-a40b5b6ddb72-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sts5m\" (UID: \"278c1af4-cf3a-49c4-80b8-a40b5b6ddb72\") " pod="openshift-insights/insights-runtime-extractor-sts5m"
Sep 08 01:59:40.668550 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:40.668514    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-sts5m"
Sep 08 01:59:40.940756 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:40.940717    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-sts5m"]
Sep 08 01:59:40.944152 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:59:40.944108    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod278c1af4_cf3a_49c4_80b8_a40b5b6ddb72.slice/crio-77bc7d575a73c3b2a5065b963c51275b94750193bf71ceda17af5d0e106b389b WatchSource:0}: Error finding container 77bc7d575a73c3b2a5065b963c51275b94750193bf71ceda17af5d0e106b389b: Status 404 returned error can't find the container with id 77bc7d575a73c3b2a5065b963c51275b94750193bf71ceda17af5d0e106b389b
Sep 08 01:59:41.239121 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:41.239024    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-q6gwz" event={"ID":"3ba60eb6-74cd-4416-9a15-d04ba2ef2b1e","Type":"ContainerStarted","Data":"1fde5c0192e027ada31845ed0094e354c5391dfeaa4cbc3452fbd79ee9e382cd"}
Sep 08 01:59:41.239541 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:41.239154    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-q6gwz"
Sep 08 01:59:41.240609 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:41.240588    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sts5m" event={"ID":"278c1af4-cf3a-49c4-80b8-a40b5b6ddb72","Type":"ContainerStarted","Data":"972ddf6fd5e70994b805d034aa2ff6450754afa012057fced606f1c163036268"}
Sep 08 01:59:41.240702 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:41.240633    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sts5m" event={"ID":"278c1af4-cf3a-49c4-80b8-a40b5b6ddb72","Type":"ContainerStarted","Data":"77bc7d575a73c3b2a5065b963c51275b94750193bf71ceda17af5d0e106b389b"}
Sep 08 01:59:41.256285 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:41.256243    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-q6gwz" podStartSLOduration=17.683369807 podStartE2EDuration="19.256228614s" totalImagesPullingTime="1.572858807s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 01:59:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 01:59:41.255363223 +0000 UTC m=+52.779138300" watchObservedRunningTime="2026-09-08 01:59:41.256228614 +0000 UTC m=+52.780003694"
Sep 08 01:59:42.246622 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:42.246575    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sts5m" event={"ID":"278c1af4-cf3a-49c4-80b8-a40b5b6ddb72","Type":"ContainerStarted","Data":"afb4f220af78bec7c9d477cd9c7fb1dfa29f7cc85093299daecbfebea7e86da2"}
Sep 08 01:59:43.251466 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:43.251430    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sts5m" event={"ID":"278c1af4-cf3a-49c4-80b8-a40b5b6ddb72","Type":"ContainerStarted","Data":"9d85c3f3d63c802ad8c23290da505d54238745c76f9f6a5ecad207f6a667cea0"}
Sep 08 01:59:43.268468 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:43.268417    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-sts5m" podStartSLOduration=17.240690506 podStartE2EDuration="19.268402893s" totalImagesPullingTime="2.027712387s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 01:59:24 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 01:59:43.267924334 +0000 UTC m=+54.791699413" watchObservedRunningTime="2026-09-08 01:59:43.268402893 +0000 UTC m=+54.792177973"
Sep 08 01:59:45.575354 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:45.575317    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-68cff75bfd-p6b52"]
Sep 08 01:59:45.589081 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:45.589037    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-98c5bd764-42nk6"]
Sep 08 01:59:45.610927 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:45.610892    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-85d98c998f-6qfk4"]
Sep 08 01:59:45.611074 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:45.611014    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-98c5bd764-42nk6"
Sep 08 01:59:45.613366 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:45.613343    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 08 01:59:45.613505 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:45.613407    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 08 01:59:45.613505 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:45.613418    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 08 01:59:45.613505 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:45.613461    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 08 01:59:45.625602 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:45.625569    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-756d6fd785-lknpg"]
Sep 08 01:59:45.625729 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:45.625687    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-6qfk4"
Sep 08 01:59:45.627946 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:45.627877    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 08 01:59:45.628150 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:45.628121    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 08 01:59:45.628256 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:45.628121    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-qcj4g\""
Sep 08 01:59:45.647682 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:45.647648    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66984d74b8-hdhcn"]
Sep 08 01:59:45.647830 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:45.647784    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-756d6fd785-lknpg"
Sep 08 01:59:45.650187 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:45.650160    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 08 01:59:45.650334 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:45.650245    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 08 01:59:45.650410 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:45.650394    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 08 01:59:45.650485 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:45.650468    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 08 01:59:45.663571 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:45.663547    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-98c5bd764-42nk6"]
Sep 08 01:59:45.663698 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:45.663591    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66984d74b8-hdhcn"]
Sep 08 01:59:45.663698 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:45.663602    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-756d6fd785-lknpg"]
Sep 08 01:59:45.663698 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:45.663614    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-6qfk4"]
Sep 08 01:59:45.663698 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:45.663655    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66984d74b8-hdhcn"
Sep 08 01:59:45.665689 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:45.665669    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-474sj\""
Sep 08 01:59:45.665787 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:45.665669    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 08 01:59:45.688315 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:45.688225    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/d4f5555d-308d-4c51-91d8-8738f39caae5-klusterlet-config\") pod \"klusterlet-addon-workmgr-98c5bd764-42nk6\" (UID: \"d4f5555d-308d-4c51-91d8-8738f39caae5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-98c5bd764-42nk6"
Sep 08 01:59:45.688315 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:45.688306    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dlch\" (UniqueName: \"kubernetes.io/projected/7ad15f65-08f1-4e92-a163-93131006559a-kube-api-access-5dlch\") pod \"downloads-85d98c998f-6qfk4\" (UID: \"7ad15f65-08f1-4e92-a163-93131006559a\") " pod="openshift-console/downloads-85d98c998f-6qfk4"
Sep 08 01:59:45.688554 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:45.688341    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c65t\" (UniqueName: \"kubernetes.io/projected/3108a751-775f-48dc-93ff-2d43c4dda80f-kube-api-access-8c65t\") pod \"managed-serviceaccount-addon-agent-66984d74b8-hdhcn\" (UID: \"3108a751-775f-48dc-93ff-2d43c4dda80f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66984d74b8-hdhcn"
Sep 08 01:59:45.688554 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:45.688378    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d4f5555d-308d-4c51-91d8-8738f39caae5-tmp\") pod \"klusterlet-addon-workmgr-98c5bd764-42nk6\" (UID: \"d4f5555d-308d-4c51-91d8-8738f39caae5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-98c5bd764-42nk6"
Sep 08 01:59:45.688554 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:45.688465    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6vml\" (UniqueName: \"kubernetes.io/projected/d4f5555d-308d-4c51-91d8-8738f39caae5-kube-api-access-j6vml\") pod \"klusterlet-addon-workmgr-98c5bd764-42nk6\" (UID: \"d4f5555d-308d-4c51-91d8-8738f39caae5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-98c5bd764-42nk6"
Sep 08 01:59:45.688707 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:45.688551    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3108a751-775f-48dc-93ff-2d43c4dda80f-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-66984d74b8-hdhcn\" (UID: \"3108a751-775f-48dc-93ff-2d43c4dda80f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66984d74b8-hdhcn"
Sep 08 01:59:45.689131 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:45.689113    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-db5d55684-xlxmr"]
Sep 08 01:59:45.711409 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:45.711376    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-db5d55684-xlxmr"]
Sep 08 01:59:45.711560 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:45.711479    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-db5d55684-xlxmr"
Sep 08 01:59:45.790065 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:45.790034    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d4f5555d-308d-4c51-91d8-8738f39caae5-tmp\") pod \"klusterlet-addon-workmgr-98c5bd764-42nk6\" (UID: \"d4f5555d-308d-4c51-91d8-8738f39caae5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-98c5bd764-42nk6"
Sep 08 01:59:45.790281 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:45.790076    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/919a89ec-bec1-4c20-85a0-3e711dd71f3e-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-756d6fd785-lknpg\" (UID: \"919a89ec-bec1-4c20-85a0-3e711dd71f3e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-756d6fd785-lknpg"
Sep 08 01:59:45.790281 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:45.790099    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9ca7a4bc-6b17-4220-95e1-b28e01db798e-registry-certificates\") pod \"image-registry-db5d55684-xlxmr\" (UID: \"9ca7a4bc-6b17-4220-95e1-b28e01db798e\") " pod="openshift-image-registry/image-registry-db5d55684-xlxmr"
Sep 08 01:59:45.790281 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:45.790165    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/919a89ec-bec1-4c20-85a0-3e711dd71f3e-hub\") pod \"cluster-proxy-proxy-agent-756d6fd785-lknpg\" (UID: \"919a89ec-bec1-4c20-85a0-3e711dd71f3e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-756d6fd785-lknpg"
Sep 08 01:59:45.790281 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:45.790190    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6vml\" (UniqueName: \"kubernetes.io/projected/d4f5555d-308d-4c51-91d8-8738f39caae5-kube-api-access-j6vml\") pod \"klusterlet-addon-workmgr-98c5bd764-42nk6\" (UID: \"d4f5555d-308d-4c51-91d8-8738f39caae5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-98c5bd764-42nk6"
Sep 08 01:59:45.790281 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:45.790216    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/919a89ec-bec1-4c20-85a0-3e711dd71f3e-ca\") pod \"cluster-proxy-proxy-agent-756d6fd785-lknpg\" (UID: \"919a89ec-bec1-4c20-85a0-3e711dd71f3e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-756d6fd785-lknpg"
Sep 08 01:59:45.790281 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:45.790247    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9ca7a4bc-6b17-4220-95e1-b28e01db798e-ca-trust-extracted\") pod \"image-registry-db5d55684-xlxmr\" (UID: \"9ca7a4bc-6b17-4220-95e1-b28e01db798e\") " pod="openshift-image-registry/image-registry-db5d55684-xlxmr"
Sep 08 01:59:45.790583 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:45.790321    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3108a751-775f-48dc-93ff-2d43c4dda80f-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-66984d74b8-hdhcn\" (UID: \"3108a751-775f-48dc-93ff-2d43c4dda80f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66984d74b8-hdhcn"
Sep 08 01:59:45.790583 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:45.790361    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l55tc\" (UniqueName: \"kubernetes.io/projected/9ca7a4bc-6b17-4220-95e1-b28e01db798e-kube-api-access-l55tc\") pod \"image-registry-db5d55684-xlxmr\" (UID: \"9ca7a4bc-6b17-4220-95e1-b28e01db798e\") " pod="openshift-image-registry/image-registry-db5d55684-xlxmr"
Sep 08 01:59:45.790583 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:45.790378    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nth8f\" (UniqueName: \"kubernetes.io/projected/919a89ec-bec1-4c20-85a0-3e711dd71f3e-kube-api-access-nth8f\") pod \"cluster-proxy-proxy-agent-756d6fd785-lknpg\" (UID: \"919a89ec-bec1-4c20-85a0-3e711dd71f3e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-756d6fd785-lknpg"
Sep 08 01:59:45.790583 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:45.790397    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9ca7a4bc-6b17-4220-95e1-b28e01db798e-image-registry-private-configuration\") pod \"image-registry-db5d55684-xlxmr\" (UID: \"9ca7a4bc-6b17-4220-95e1-b28e01db798e\") " pod="openshift-image-registry/image-registry-db5d55684-xlxmr"
Sep 08 01:59:45.790583 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:45.790417    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9ca7a4bc-6b17-4220-95e1-b28e01db798e-trusted-ca\") pod \"image-registry-db5d55684-xlxmr\" (UID: \"9ca7a4bc-6b17-4220-95e1-b28e01db798e\") " pod="openshift-image-registry/image-registry-db5d55684-xlxmr"
Sep 08 01:59:45.790583 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:45.790443    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/d4f5555d-308d-4c51-91d8-8738f39caae5-klusterlet-config\") pod \"klusterlet-addon-workmgr-98c5bd764-42nk6\" (UID: \"d4f5555d-308d-4c51-91d8-8738f39caae5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-98c5bd764-42nk6"
Sep 08 01:59:45.790583 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:45.790521    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d4f5555d-308d-4c51-91d8-8738f39caae5-tmp\") pod \"klusterlet-addon-workmgr-98c5bd764-42nk6\" (UID: \"d4f5555d-308d-4c51-91d8-8738f39caae5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-98c5bd764-42nk6"
Sep 08 01:59:45.790583 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:45.790538    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dlch\" (UniqueName: \"kubernetes.io/projected/7ad15f65-08f1-4e92-a163-93131006559a-kube-api-access-5dlch\") pod \"downloads-85d98c998f-6qfk4\" (UID: \"7ad15f65-08f1-4e92-a163-93131006559a\") " pod="openshift-console/downloads-85d98c998f-6qfk4"
Sep 08 01:59:45.790583 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:45.790571    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/919a89ec-bec1-4c20-85a0-3e711dd71f3e-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-756d6fd785-lknpg\" (UID: \"919a89ec-bec1-4c20-85a0-3e711dd71f3e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-756d6fd785-lknpg"
Sep 08 01:59:45.790980 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:45.790603    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9ca7a4bc-6b17-4220-95e1-b28e01db798e-registry-tls\") pod \"image-registry-db5d55684-xlxmr\" (UID: \"9ca7a4bc-6b17-4220-95e1-b28e01db798e\") " pod="openshift-image-registry/image-registry-db5d55684-xlxmr"
Sep 08 01:59:45.790980 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:45.790642    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c65t\" (UniqueName: \"kubernetes.io/projected/3108a751-775f-48dc-93ff-2d43c4dda80f-kube-api-access-8c65t\") pod \"managed-serviceaccount-addon-agent-66984d74b8-hdhcn\" (UID: \"3108a751-775f-48dc-93ff-2d43c4dda80f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66984d74b8-hdhcn"
Sep 08 01:59:45.790980 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:45.790671    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/919a89ec-bec1-4c20-85a0-3e711dd71f3e-additional-service-ca\") pod \"cluster-proxy-proxy-agent-756d6fd785-lknpg\" (UID: \"919a89ec-bec1-4c20-85a0-3e711dd71f3e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-756d6fd785-lknpg"
Sep 08 01:59:45.790980 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:45.790705    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9ca7a4bc-6b17-4220-95e1-b28e01db798e-installation-pull-secrets\") pod \"image-registry-db5d55684-xlxmr\" (UID: \"9ca7a4bc-6b17-4220-95e1-b28e01db798e\") " pod="openshift-image-registry/image-registry-db5d55684-xlxmr"
Sep 08 01:59:45.790980 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:45.790758    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9ca7a4bc-6b17-4220-95e1-b28e01db798e-bound-sa-token\") pod \"image-registry-db5d55684-xlxmr\" (UID: \"9ca7a4bc-6b17-4220-95e1-b28e01db798e\") " pod="openshift-image-registry/image-registry-db5d55684-xlxmr"
Sep 08 01:59:45.793110 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:45.793088    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/d4f5555d-308d-4c51-91d8-8738f39caae5-klusterlet-config\") pod \"klusterlet-addon-workmgr-98c5bd764-42nk6\" (UID: \"d4f5555d-308d-4c51-91d8-8738f39caae5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-98c5bd764-42nk6"
Sep 08 01:59:45.793286 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:45.793269    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3108a751-775f-48dc-93ff-2d43c4dda80f-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-66984d74b8-hdhcn\" (UID: \"3108a751-775f-48dc-93ff-2d43c4dda80f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66984d74b8-hdhcn"
Sep 08 01:59:45.804422 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:45.804399    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dlch\" (UniqueName: \"kubernetes.io/projected/7ad15f65-08f1-4e92-a163-93131006559a-kube-api-access-5dlch\") pod \"downloads-85d98c998f-6qfk4\" (UID: \"7ad15f65-08f1-4e92-a163-93131006559a\") " pod="openshift-console/downloads-85d98c998f-6qfk4"
Sep 08 01:59:45.804809 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:45.804789    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6vml\" (UniqueName: \"kubernetes.io/projected/d4f5555d-308d-4c51-91d8-8738f39caae5-kube-api-access-j6vml\") pod \"klusterlet-addon-workmgr-98c5bd764-42nk6\" (UID: \"d4f5555d-308d-4c51-91d8-8738f39caae5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-98c5bd764-42nk6"
Sep 08 01:59:45.805084 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:45.805063    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c65t\" (UniqueName: \"kubernetes.io/projected/3108a751-775f-48dc-93ff-2d43c4dda80f-kube-api-access-8c65t\") pod \"managed-serviceaccount-addon-agent-66984d74b8-hdhcn\" (UID: \"3108a751-775f-48dc-93ff-2d43c4dda80f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66984d74b8-hdhcn"
Sep 08 01:59:45.891320 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:45.891288    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/919a89ec-bec1-4c20-85a0-3e711dd71f3e-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-756d6fd785-lknpg\" (UID: \"919a89ec-bec1-4c20-85a0-3e711dd71f3e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-756d6fd785-lknpg"
Sep 08 01:59:45.891476 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:45.891327    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9ca7a4bc-6b17-4220-95e1-b28e01db798e-registry-certificates\") pod \"image-registry-db5d55684-xlxmr\" (UID: \"9ca7a4bc-6b17-4220-95e1-b28e01db798e\") " pod="openshift-image-registry/image-registry-db5d55684-xlxmr"
Sep 08 01:59:45.891476 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:45.891342    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/919a89ec-bec1-4c20-85a0-3e711dd71f3e-hub\") pod \"cluster-proxy-proxy-agent-756d6fd785-lknpg\" (UID: \"919a89ec-bec1-4c20-85a0-3e711dd71f3e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-756d6fd785-lknpg"
Sep 08 01:59:45.891476 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:45.891367    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/919a89ec-bec1-4c20-85a0-3e711dd71f3e-ca\") pod \"cluster-proxy-proxy-agent-756d6fd785-lknpg\" (UID: \"919a89ec-bec1-4c20-85a0-3e711dd71f3e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-756d6fd785-lknpg"
Sep 08 01:59:45.891476 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:45.891422    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9ca7a4bc-6b17-4220-95e1-b28e01db798e-ca-trust-extracted\") pod \"image-registry-db5d55684-xlxmr\" (UID: \"9ca7a4bc-6b17-4220-95e1-b28e01db798e\") " pod="openshift-image-registry/image-registry-db5d55684-xlxmr"
Sep 08 01:59:45.891679 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:45.891475    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l55tc\" (UniqueName: \"kubernetes.io/projected/9ca7a4bc-6b17-4220-95e1-b28e01db798e-kube-api-access-l55tc\") pod \"image-registry-db5d55684-xlxmr\" (UID: \"9ca7a4bc-6b17-4220-95e1-b28e01db798e\") " pod="openshift-image-registry/image-registry-db5d55684-xlxmr"
Sep 08 01:59:45.891679 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:45.891498    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nth8f\" (UniqueName: \"kubernetes.io/projected/919a89ec-bec1-4c20-85a0-3e711dd71f3e-kube-api-access-nth8f\") pod \"cluster-proxy-proxy-agent-756d6fd785-lknpg\" (UID: \"919a89ec-bec1-4c20-85a0-3e711dd71f3e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-756d6fd785-lknpg"
Sep 08 01:59:45.891679 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:45.891521    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9ca7a4bc-6b17-4220-95e1-b28e01db798e-image-registry-private-configuration\") pod \"image-registry-db5d55684-xlxmr\" (UID: \"9ca7a4bc-6b17-4220-95e1-b28e01db798e\") " pod="openshift-image-registry/image-registry-db5d55684-xlxmr"
Sep 08 01:59:45.891679 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:45.891547    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9ca7a4bc-6b17-4220-95e1-b28e01db798e-trusted-ca\") pod \"image-registry-db5d55684-xlxmr\" (UID: \"9ca7a4bc-6b17-4220-95e1-b28e01db798e\") " pod="openshift-image-registry/image-registry-db5d55684-xlxmr"
Sep 08 01:59:45.891679 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:45.891608    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/919a89ec-bec1-4c20-85a0-3e711dd71f3e-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-756d6fd785-lknpg\" (UID: \"919a89ec-bec1-4c20-85a0-3e711dd71f3e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-756d6fd785-lknpg"
Sep 08 01:59:45.891679 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:45.891631    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9ca7a4bc-6b17-4220-95e1-b28e01db798e-registry-tls\") pod \"image-registry-db5d55684-xlxmr\" (UID: \"9ca7a4bc-6b17-4220-95e1-b28e01db798e\") " pod="openshift-image-registry/image-registry-db5d55684-xlxmr"
Sep 08 01:59:45.891679 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:45.891659    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/919a89ec-bec1-4c20-85a0-3e711dd71f3e-additional-service-ca\") pod \"cluster-proxy-proxy-agent-756d6fd785-lknpg\" (UID: \"919a89ec-bec1-4c20-85a0-3e711dd71f3e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-756d6fd785-lknpg"
Sep 08 01:59:45.892051 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:45.891693    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9ca7a4bc-6b17-4220-95e1-b28e01db798e-installation-pull-secrets\") pod \"image-registry-db5d55684-xlxmr\" (UID: \"9ca7a4bc-6b17-4220-95e1-b28e01db798e\") " pod="openshift-image-registry/image-registry-db5d55684-xlxmr"
Sep 08 01:59:45.892051 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:45.891724    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9ca7a4bc-6b17-4220-95e1-b28e01db798e-bound-sa-token\") pod \"image-registry-db5d55684-xlxmr\" (UID: \"9ca7a4bc-6b17-4220-95e1-b28e01db798e\") " pod="openshift-image-registry/image-registry-db5d55684-xlxmr"
Sep 08 01:59:45.892051 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:45.891926    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9ca7a4bc-6b17-4220-95e1-b28e01db798e-ca-trust-extracted\") pod \"image-registry-db5d55684-xlxmr\" (UID: \"9ca7a4bc-6b17-4220-95e1-b28e01db798e\") " pod="openshift-image-registry/image-registry-db5d55684-xlxmr"
Sep 08 01:59:45.892391 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:45.892274    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9ca7a4bc-6b17-4220-95e1-b28e01db798e-registry-certificates\") pod \"image-registry-db5d55684-xlxmr\" (UID: \"9ca7a4bc-6b17-4220-95e1-b28e01db798e\") " pod="openshift-image-registry/image-registry-db5d55684-xlxmr"
Sep 08 01:59:45.892949 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:45.892919    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/919a89ec-bec1-4c20-85a0-3e711dd71f3e-additional-service-ca\") pod \"cluster-proxy-proxy-agent-756d6fd785-lknpg\" (UID: \"919a89ec-bec1-4c20-85a0-3e711dd71f3e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-756d6fd785-lknpg"
Sep 08 01:59:45.893330 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:45.893286    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9ca7a4bc-6b17-4220-95e1-b28e01db798e-trusted-ca\") pod \"image-registry-db5d55684-xlxmr\" (UID: \"9ca7a4bc-6b17-4220-95e1-b28e01db798e\") " pod="openshift-image-registry/image-registry-db5d55684-xlxmr"
Sep 08 01:59:45.894540 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:45.894468    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/919a89ec-bec1-4c20-85a0-3e711dd71f3e-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-756d6fd785-lknpg\" (UID: \"919a89ec-bec1-4c20-85a0-3e711dd71f3e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-756d6fd785-lknpg"
Sep 08 01:59:45.894653 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:45.894611    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/919a89ec-bec1-4c20-85a0-3e711dd71f3e-hub\") pod \"cluster-proxy-proxy-agent-756d6fd785-lknpg\" (UID: \"919a89ec-bec1-4c20-85a0-3e711dd71f3e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-756d6fd785-lknpg"
Sep 08 01:59:45.894715 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:45.894698    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9ca7a4bc-6b17-4220-95e1-b28e01db798e-image-registry-private-configuration\") pod \"image-registry-db5d55684-xlxmr\" (UID: \"9ca7a4bc-6b17-4220-95e1-b28e01db798e\") " pod="openshift-image-registry/image-registry-db5d55684-xlxmr"
Sep 08 01:59:45.894984 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:45.894955    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9ca7a4bc-6b17-4220-95e1-b28e01db798e-installation-pull-secrets\") pod \"image-registry-db5d55684-xlxmr\" (UID: \"9ca7a4bc-6b17-4220-95e1-b28e01db798e\") " pod="openshift-image-registry/image-registry-db5d55684-xlxmr"
Sep 08 01:59:45.895092 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:45.894955    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/919a89ec-bec1-4c20-85a0-3e711dd71f3e-ca\") pod \"cluster-proxy-proxy-agent-756d6fd785-lknpg\" (UID: \"919a89ec-bec1-4c20-85a0-3e711dd71f3e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-756d6fd785-lknpg"
Sep 08 01:59:45.895371 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:45.895352    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9ca7a4bc-6b17-4220-95e1-b28e01db798e-registry-tls\") pod \"image-registry-db5d55684-xlxmr\" (UID: \"9ca7a4bc-6b17-4220-95e1-b28e01db798e\") " pod="openshift-image-registry/image-registry-db5d55684-xlxmr"
Sep 08 01:59:45.895439 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:45.895430    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/919a89ec-bec1-4c20-85a0-3e711dd71f3e-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-756d6fd785-lknpg\" (UID: \"919a89ec-bec1-4c20-85a0-3e711dd71f3e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-756d6fd785-lknpg"
Sep 08 01:59:45.899927 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:45.899896    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nth8f\" (UniqueName: \"kubernetes.io/projected/919a89ec-bec1-4c20-85a0-3e711dd71f3e-kube-api-access-nth8f\") pod \"cluster-proxy-proxy-agent-756d6fd785-lknpg\" (UID: \"919a89ec-bec1-4c20-85a0-3e711dd71f3e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-756d6fd785-lknpg"
Sep 08 01:59:45.901034 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:45.901007    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l55tc\" (UniqueName: \"kubernetes.io/projected/9ca7a4bc-6b17-4220-95e1-b28e01db798e-kube-api-access-l55tc\") pod \"image-registry-db5d55684-xlxmr\" (UID: \"9ca7a4bc-6b17-4220-95e1-b28e01db798e\") " pod="openshift-image-registry/image-registry-db5d55684-xlxmr"
Sep 08 01:59:45.901294 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:45.901277    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9ca7a4bc-6b17-4220-95e1-b28e01db798e-bound-sa-token\") pod \"image-registry-db5d55684-xlxmr\" (UID: \"9ca7a4bc-6b17-4220-95e1-b28e01db798e\") " pod="openshift-image-registry/image-registry-db5d55684-xlxmr"
Sep 08 01:59:45.921858 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:45.921820    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-98c5bd764-42nk6"
Sep 08 01:59:45.935151 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:45.935099    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-6qfk4"
Sep 08 01:59:45.969597 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:45.969569    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-756d6fd785-lknpg"
Sep 08 01:59:45.976009 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:45.975986    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66984d74b8-hdhcn"
Sep 08 01:59:46.028057 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:46.027974    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-db5d55684-xlxmr"
Sep 08 01:59:46.231444 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:46.231372    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-98c5bd764-42nk6"]
Sep 08 01:59:46.234866 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:59:46.234837    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4f5555d_308d_4c51_91d8_8738f39caae5.slice/crio-6dfe6904c79066cc165ee192c6e1e44a8db91209cbf32446737a66b841240d1f WatchSource:0}: Error finding container 6dfe6904c79066cc165ee192c6e1e44a8db91209cbf32446737a66b841240d1f: Status 404 returned error can't find the container with id 6dfe6904c79066cc165ee192c6e1e44a8db91209cbf32446737a66b841240d1f
Sep 08 01:59:46.260239 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:46.260209    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-6qfk4"]
Sep 08 01:59:46.262352 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:46.262328    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-98c5bd764-42nk6" event={"ID":"d4f5555d-308d-4c51-91d8-8738f39caae5","Type":"ContainerStarted","Data":"6dfe6904c79066cc165ee192c6e1e44a8db91209cbf32446737a66b841240d1f"}
Sep 08 01:59:46.262867 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:59:46.262848    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ad15f65_08f1_4e92_a163_93131006559a.slice/crio-87a277c531090e10a21ff9cb20ef8ea62e0d7e0e54ff3b6435f560d25f78f139 WatchSource:0}: Error finding container 87a277c531090e10a21ff9cb20ef8ea62e0d7e0e54ff3b6435f560d25f78f139: Status 404 returned error can't find the container with id 87a277c531090e10a21ff9cb20ef8ea62e0d7e0e54ff3b6435f560d25f78f139
Sep 08 01:59:46.277723 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:46.277699    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-756d6fd785-lknpg"]
Sep 08 01:59:46.280608 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:59:46.280578    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod919a89ec_bec1_4c20_85a0_3e711dd71f3e.slice/crio-df4b3e1641ed430ae7b49d2423d7d3474d66db506e9f097bc510af4ca181ea97 WatchSource:0}: Error finding container df4b3e1641ed430ae7b49d2423d7d3474d66db506e9f097bc510af4ca181ea97: Status 404 returned error can't find the container with id df4b3e1641ed430ae7b49d2423d7d3474d66db506e9f097bc510af4ca181ea97
Sep 08 01:59:46.310458 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:46.310369    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66984d74b8-hdhcn"]
Sep 08 01:59:46.312810 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:59:46.312790    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3108a751_775f_48dc_93ff_2d43c4dda80f.slice/crio-efb4d9db31c5f21b6dc8bc5fcc72d153705260f012fd7429772a8ac0d941660b WatchSource:0}: Error finding container efb4d9db31c5f21b6dc8bc5fcc72d153705260f012fd7429772a8ac0d941660b: Status 404 returned error can't find the container with id efb4d9db31c5f21b6dc8bc5fcc72d153705260f012fd7429772a8ac0d941660b
Sep 08 01:59:46.338715 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:46.338687    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-db5d55684-xlxmr"]
Sep 08 01:59:46.341390 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:59:46.341365    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ca7a4bc_6b17_4220_95e1_b28e01db798e.slice/crio-e3babed30e9f7912fb701a847dba036add71daafecbf734d99003a0f7046e51a WatchSource:0}: Error finding container e3babed30e9f7912fb701a847dba036add71daafecbf734d99003a0f7046e51a: Status 404 returned error can't find the container with id e3babed30e9f7912fb701a847dba036add71daafecbf734d99003a0f7046e51a
Sep 08 01:59:47.271026 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:47.270987    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-db5d55684-xlxmr" event={"ID":"9ca7a4bc-6b17-4220-95e1-b28e01db798e","Type":"ContainerStarted","Data":"46618a793bd161099da68670fb79273c41faa70fd11476647e9f31552ca54195"}
Sep 08 01:59:47.271026 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:47.271053    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-db5d55684-xlxmr" event={"ID":"9ca7a4bc-6b17-4220-95e1-b28e01db798e","Type":"ContainerStarted","Data":"e3babed30e9f7912fb701a847dba036add71daafecbf734d99003a0f7046e51a"}
Sep 08 01:59:47.271904 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:47.271848    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-db5d55684-xlxmr"
Sep 08 01:59:47.276548 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:47.276474    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66984d74b8-hdhcn" event={"ID":"3108a751-775f-48dc-93ff-2d43c4dda80f","Type":"ContainerStarted","Data":"efb4d9db31c5f21b6dc8bc5fcc72d153705260f012fd7429772a8ac0d941660b"}
Sep 08 01:59:47.279271 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:47.279209    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-756d6fd785-lknpg" event={"ID":"919a89ec-bec1-4c20-85a0-3e711dd71f3e","Type":"ContainerStarted","Data":"df4b3e1641ed430ae7b49d2423d7d3474d66db506e9f097bc510af4ca181ea97"}
Sep 08 01:59:47.281384 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:47.281307    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-6qfk4" event={"ID":"7ad15f65-08f1-4e92-a163-93131006559a","Type":"ContainerStarted","Data":"87a277c531090e10a21ff9cb20ef8ea62e0d7e0e54ff3b6435f560d25f78f139"}
Sep 08 01:59:48.165810 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:48.165779    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xskxl"
Sep 08 01:59:48.195480 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:48.195165    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-db5d55684-xlxmr" podStartSLOduration=3.195124531 podStartE2EDuration="3.195124531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 01:59:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 01:59:47.290030085 +0000 UTC m=+58.813805167" watchObservedRunningTime="2026-09-08 01:59:48.195124531 +0000 UTC m=+59.718899612"
Sep 08 01:59:51.249434 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:51.249398    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-q6gwz"
Sep 08 01:59:54.321075 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:54.321010    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66984d74b8-hdhcn" event={"ID":"3108a751-775f-48dc-93ff-2d43c4dda80f","Type":"ContainerStarted","Data":"a7b5018eabe64b208a34cbba9bb6b4073865b186f60ab5a67e6ef091933f483d"}
Sep 08 01:59:54.322952 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:54.322930    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-756d6fd785-lknpg" event={"ID":"919a89ec-bec1-4c20-85a0-3e711dd71f3e","Type":"ContainerStarted","Data":"66140a4c025e0c6cc065f7e1ba7b021ca6dfee74237e036da7a28bd2f271e633"}
Sep 08 01:59:54.323044 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:54.322962    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-756d6fd785-lknpg" event={"ID":"919a89ec-bec1-4c20-85a0-3e711dd71f3e","Type":"ContainerStarted","Data":"dc2eaa2e206ea19dc676e67fa87938255805c30a56851a023cbfa2e349484557"}
Sep 08 01:59:54.325770 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:54.325719    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-98c5bd764-42nk6" event={"ID":"d4f5555d-308d-4c51-91d8-8738f39caae5","Type":"ContainerStarted","Data":"5be6b38885d9a19051eea305dbef2f10b4e633cb7950067ef2eca68b6a7e66d8"}
Sep 08 01:59:54.326184 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:54.326162    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-98c5bd764-42nk6"
Sep 08 01:59:54.328384 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:54.328367    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-98c5bd764-42nk6"
Sep 08 01:59:54.334239 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:54.334193    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66984d74b8-hdhcn" podStartSLOduration=1.5557273999999999 podStartE2EDuration="9.334176794s" totalImagesPullingTime="7.778449394s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 01:59:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 01:59:54.333579214 +0000 UTC m=+65.857354314" watchObservedRunningTime="2026-09-08 01:59:54.334176794 +0000 UTC m=+65.857951876"
Sep 08 01:59:54.349831 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:54.349783    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-98c5bd764-42nk6" podStartSLOduration=1.468987767 podStartE2EDuration="9.349763286s" totalImagesPullingTime="7.880775519s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 01:59:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 01:59:54.349705427 +0000 UTC m=+65.873480507" watchObservedRunningTime="2026-09-08 01:59:54.349763286 +0000 UTC m=+65.873538442"
Sep 08 01:59:54.786481 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:54.786431    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b82c5af2-4c1c-41a9-8fa4-af8a85bcc8c6-metrics-certs\") pod \"network-metrics-daemon-jr45c\" (UID: \"b82c5af2-4c1c-41a9-8fa4-af8a85bcc8c6\") " pod="openshift-multus/network-metrics-daemon-jr45c"
Sep 08 01:59:54.786703 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:54.786520    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/917f6c17-7902-48d3-a0f4-535e5b602d26-original-pull-secret\") pod \"global-pull-secret-syncer-l7hw8\" (UID: \"917f6c17-7902-48d3-a0f4-535e5b602d26\") " pod="kube-system/global-pull-secret-syncer-l7hw8"
Sep 08 01:59:54.788743 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:54.788714    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 08 01:59:54.789422 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:54.789403    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 08 01:59:54.799600 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:54.799576    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/917f6c17-7902-48d3-a0f4-535e5b602d26-original-pull-secret\") pod \"global-pull-secret-syncer-l7hw8\" (UID: \"917f6c17-7902-48d3-a0f4-535e5b602d26\") " pod="kube-system/global-pull-secret-syncer-l7hw8"
Sep 08 01:59:54.799720 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:54.799638    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b82c5af2-4c1c-41a9-8fa4-af8a85bcc8c6-metrics-certs\") pod \"network-metrics-daemon-jr45c\" (UID: \"b82c5af2-4c1c-41a9-8fa4-af8a85bcc8c6\") " pod="openshift-multus/network-metrics-daemon-jr45c"
Sep 08 01:59:54.887204 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:54.887166    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dt4k\" (UniqueName: \"kubernetes.io/projected/430aad96-9d26-4055-8133-b7f50b0f2978-kube-api-access-5dt4k\") pod \"network-check-target-2v24n\" (UID: \"430aad96-9d26-4055-8133-b7f50b0f2978\") " pod="openshift-network-diagnostics/network-check-target-2v24n"
Sep 08 01:59:54.889592 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:54.889568    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 08 01:59:54.900345 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:54.900315    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l7hw8"
Sep 08 01:59:54.900345 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:54.900333    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 08 01:59:54.910409 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:54.910382    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-s2v8c\""
Sep 08 01:59:54.911330 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:54.911310    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dt4k\" (UniqueName: \"kubernetes.io/projected/430aad96-9d26-4055-8133-b7f50b0f2978-kube-api-access-5dt4k\") pod \"network-check-target-2v24n\" (UID: \"430aad96-9d26-4055-8133-b7f50b0f2978\") " pod="openshift-network-diagnostics/network-check-target-2v24n"
Sep 08 01:59:54.915918 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:54.915897    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-lxnlh\""
Sep 08 01:59:54.918498 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:54.918479    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jr45c"
Sep 08 01:59:54.924261 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:54.924227    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2v24n"
Sep 08 01:59:55.182413 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:55.182286    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-l7hw8"]
Sep 08 01:59:55.186290 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:59:55.186259    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod917f6c17_7902_48d3_a0f4_535e5b602d26.slice/crio-38fa30a545723c1ed48f7558cd698769eeccfb42b6f5e64aa2bfc8d19a1625b4 WatchSource:0}: Error finding container 38fa30a545723c1ed48f7558cd698769eeccfb42b6f5e64aa2bfc8d19a1625b4: Status 404 returned error can't find the container with id 38fa30a545723c1ed48f7558cd698769eeccfb42b6f5e64aa2bfc8d19a1625b4
Sep 08 01:59:55.205793 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:55.205765    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jr45c"]
Sep 08 01:59:55.208518 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:59:55.208496    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb82c5af2_4c1c_41a9_8fa4_af8a85bcc8c6.slice/crio-07eeaabb153f1af39592bdd20e2bcbc2c497d4998869428110054c6f6804f32c WatchSource:0}: Error finding container 07eeaabb153f1af39592bdd20e2bcbc2c497d4998869428110054c6f6804f32c: Status 404 returned error can't find the container with id 07eeaabb153f1af39592bdd20e2bcbc2c497d4998869428110054c6f6804f32c
Sep 08 01:59:55.212409 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:55.212031    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2v24n"]
Sep 08 01:59:55.216170 ip-10-0-142-78 kubenswrapper[2547]: W0908 01:59:55.216120    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod430aad96_9d26_4055_8133_b7f50b0f2978.slice/crio-f265fffbd248999cf7c458705f237bb322925c02d3c366406d1cce10061ce2d5 WatchSource:0}: Error finding container f265fffbd248999cf7c458705f237bb322925c02d3c366406d1cce10061ce2d5: Status 404 returned error can't find the container with id f265fffbd248999cf7c458705f237bb322925c02d3c366406d1cce10061ce2d5
Sep 08 01:59:55.331023 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:55.330983    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-l7hw8" event={"ID":"917f6c17-7902-48d3-a0f4-535e5b602d26","Type":"ContainerStarted","Data":"38fa30a545723c1ed48f7558cd698769eeccfb42b6f5e64aa2bfc8d19a1625b4"}
Sep 08 01:59:55.332405 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:55.332377    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2v24n" event={"ID":"430aad96-9d26-4055-8133-b7f50b0f2978","Type":"ContainerStarted","Data":"f265fffbd248999cf7c458705f237bb322925c02d3c366406d1cce10061ce2d5"}
Sep 08 01:59:55.333796 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:55.333774    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jr45c" event={"ID":"b82c5af2-4c1c-41a9-8fa4-af8a85bcc8c6","Type":"ContainerStarted","Data":"07eeaabb153f1af39592bdd20e2bcbc2c497d4998869428110054c6f6804f32c"}
Sep 08 01:59:55.336098 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:55.336073    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-756d6fd785-lknpg" event={"ID":"919a89ec-bec1-4c20-85a0-3e711dd71f3e","Type":"ContainerStarted","Data":"41ffbf0936c5c9af81fdc5e9de82e7e23cedb8e3ac6daff5c289098463c863a3"}
Sep 08 01:59:55.353127 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:55.353075    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-756d6fd785-lknpg" podStartSLOduration=2.52331316 podStartE2EDuration="10.353057272s" totalImagesPullingTime="7.829744112s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 01:59:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 01:59:55.352664283 +0000 UTC m=+66.876439363" watchObservedRunningTime="2026-09-08 01:59:55.353057272 +0000 UTC m=+66.876832355"
Sep 08 01:59:55.582117 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:55.581985    2547 patch_prober.go:28] interesting pod/image-registry-68cff75bfd-p6b52 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 08 01:59:55.582117 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:55.582053    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-68cff75bfd-p6b52" podUID="c7d9f993-10b4-4226-8193-a7599e0d3024" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 01:59:57.353285 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:57.353062    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jr45c" event={"ID":"b82c5af2-4c1c-41a9-8fa4-af8a85bcc8c6","Type":"ContainerStarted","Data":"6e7b268d8fefd5fb7cf4a06894746fcd66455f22e8a30c0af6c0784f4336e9cb"}
Sep 08 01:59:57.353285 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:57.353245    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jr45c" event={"ID":"b82c5af2-4c1c-41a9-8fa4-af8a85bcc8c6","Type":"ContainerStarted","Data":"bc8a2cb22f8130514b1e8cfcca904e8dfe3f32384d3d995982bec2a29eb4429f"}
Sep 08 01:59:57.372228 ip-10-0-142-78 kubenswrapper[2547]: I0908 01:59:57.372039    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-jr45c" podStartSLOduration=67.084984541 podStartE2EDuration="1m8.372016316s" totalImagesPullingTime="1.287031775s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 01:58:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 01:59:57.370892027 +0000 UTC m=+68.894667131" watchObservedRunningTime="2026-09-08 01:59:57.372016316 +0000 UTC m=+68.895791388"
Sep 08 02:00:00.075806 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:00:00.075771    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-5ngwm"]
Sep 08 02:00:00.105112 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:00:00.105080    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5ngwm"
Sep 08 02:00:00.107341 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:00:00.107307    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 08 02:00:00.107481 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:00:00.107391    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-9gddq\""
Sep 08 02:00:00.107481 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:00:00.107408    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 08 02:00:00.107481 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:00:00.107471    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 08 02:00:00.111167 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:00:00.109178    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 08 02:00:00.111167 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:00:00.109256    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 08 02:00:00.111167 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:00:00.109324    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 08 02:00:00.232423 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:00:00.232385    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/167f56fa-baa2-41bf-8a91-0b63f41213a6-metrics-client-ca\") pod \"node-exporter-5ngwm\" (UID: \"167f56fa-baa2-41bf-8a91-0b63f41213a6\") " pod="openshift-monitoring/node-exporter-5ngwm"
Sep 08 02:00:00.232605 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:00:00.232458    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/167f56fa-baa2-41bf-8a91-0b63f41213a6-node-exporter-tls\") pod \"node-exporter-5ngwm\" (UID: \"167f56fa-baa2-41bf-8a91-0b63f41213a6\") " pod="openshift-monitoring/node-exporter-5ngwm"
Sep 08 02:00:00.232605 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:00:00.232501    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/167f56fa-baa2-41bf-8a91-0b63f41213a6-node-exporter-accelerators-collector-config\") pod \"node-exporter-5ngwm\" (UID: \"167f56fa-baa2-41bf-8a91-0b63f41213a6\") " pod="openshift-monitoring/node-exporter-5ngwm"
Sep 08 02:00:00.232605 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:00:00.232524    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/167f56fa-baa2-41bf-8a91-0b63f41213a6-root\") pod \"node-exporter-5ngwm\" (UID: \"167f56fa-baa2-41bf-8a91-0b63f41213a6\") " pod="openshift-monitoring/node-exporter-5ngwm"
Sep 08 02:00:00.232605 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:00:00.232552    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/167f56fa-baa2-41bf-8a91-0b63f41213a6-sys\") pod \"node-exporter-5ngwm\" (UID: \"167f56fa-baa2-41bf-8a91-0b63f41213a6\") " pod="openshift-monitoring/node-exporter-5ngwm"
Sep 08 02:00:00.232605 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:00:00.232577    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/167f56fa-baa2-41bf-8a91-0b63f41213a6-node-exporter-textfile\") pod \"node-exporter-5ngwm\" (UID: \"167f56fa-baa2-41bf-8a91-0b63f41213a6\") " pod="openshift-monitoring/node-exporter-5ngwm"
Sep 08 02:00:00.232605 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:00:00.232594    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwm8x\" (UniqueName: \"kubernetes.io/projected/167f56fa-baa2-41bf-8a91-0b63f41213a6-kube-api-access-mwm8x\") pod \"node-exporter-5ngwm\" (UID: \"167f56fa-baa2-41bf-8a91-0b63f41213a6\") " pod="openshift-monitoring/node-exporter-5ngwm"
Sep 08 02:00:00.232838 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:00:00.232620    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/167f56fa-baa2-41bf-8a91-0b63f41213a6-node-exporter-wtmp\") pod \"node-exporter-5ngwm\" (UID: \"167f56fa-baa2-41bf-8a91-0b63f41213a6\") " pod="openshift-monitoring/node-exporter-5ngwm"
Sep 08 02:00:00.232838 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:00:00.232643    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/167f56fa-baa2-41bf-8a91-0b63f41213a6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5ngwm\" (UID: \"167f56fa-baa2-41bf-8a91-0b63f41213a6\") " pod="openshift-monitoring/node-exporter-5ngwm"
Sep 08 02:00:00.333383 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:00:00.333292    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/167f56fa-baa2-41bf-8a91-0b63f41213a6-node-exporter-tls\") pod \"node-exporter-5ngwm\" (UID: \"167f56fa-baa2-41bf-8a91-0b63f41213a6\") " pod="openshift-monitoring/node-exporter-5ngwm"
Sep 08 02:00:00.333383 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:00:00.333343    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/167f56fa-baa2-41bf-8a91-0b63f41213a6-node-exporter-accelerators-collector-config\") pod \"node-exporter-5ngwm\" (UID: \"167f56fa-baa2-41bf-8a91-0b63f41213a6\") " pod="openshift-monitoring/node-exporter-5ngwm"
Sep 08 02:00:00.333383 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:00:00.333369    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/167f56fa-baa2-41bf-8a91-0b63f41213a6-root\") pod \"node-exporter-5ngwm\" (UID: \"167f56fa-baa2-41bf-8a91-0b63f41213a6\") " pod="openshift-monitoring/node-exporter-5ngwm"
Sep 08 02:00:00.333655 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:00:00.333425    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/167f56fa-baa2-41bf-8a91-0b63f41213a6-root\") pod \"node-exporter-5ngwm\" (UID: \"167f56fa-baa2-41bf-8a91-0b63f41213a6\") " pod="openshift-monitoring/node-exporter-5ngwm"
Sep 08 02:00:00.333655 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:00:00.333460    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/167f56fa-baa2-41bf-8a91-0b63f41213a6-sys\") pod \"node-exporter-5ngwm\" (UID: \"167f56fa-baa2-41bf-8a91-0b63f41213a6\") " pod="openshift-monitoring/node-exporter-5ngwm"
Sep 08 02:00:00.333655 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:00:00.333492    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/167f56fa-baa2-41bf-8a91-0b63f41213a6-node-exporter-textfile\") pod \"node-exporter-5ngwm\" (UID: \"167f56fa-baa2-41bf-8a91-0b63f41213a6\") " pod="openshift-monitoring/node-exporter-5ngwm"
Sep 08 02:00:00.333655 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:00:00.333516    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwm8x\" (UniqueName: \"kubernetes.io/projected/167f56fa-baa2-41bf-8a91-0b63f41213a6-kube-api-access-mwm8x\") pod \"node-exporter-5ngwm\" (UID: \"167f56fa-baa2-41bf-8a91-0b63f41213a6\") " pod="openshift-monitoring/node-exporter-5ngwm"
Sep 08 02:00:00.333655 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:00:00.333581    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/167f56fa-baa2-41bf-8a91-0b63f41213a6-sys\") pod \"node-exporter-5ngwm\" (UID: \"167f56fa-baa2-41bf-8a91-0b63f41213a6\") " pod="openshift-monitoring/node-exporter-5ngwm"
Sep 08 02:00:00.333902 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:00:00.333754    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/167f56fa-baa2-41bf-8a91-0b63f41213a6-node-exporter-wtmp\") pod \"node-exporter-5ngwm\" (UID: \"167f56fa-baa2-41bf-8a91-0b63f41213a6\") " pod="openshift-monitoring/node-exporter-5ngwm"
Sep 08 02:00:00.333902 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:00:00.333787    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/167f56fa-baa2-41bf-8a91-0b63f41213a6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5ngwm\" (UID: \"167f56fa-baa2-41bf-8a91-0b63f41213a6\") " pod="openshift-monitoring/node-exporter-5ngwm"
Sep 08 02:00:00.333902 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:00:00.333822    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/167f56fa-baa2-41bf-8a91-0b63f41213a6-metrics-client-ca\") pod \"node-exporter-5ngwm\" (UID: \"167f56fa-baa2-41bf-8a91-0b63f41213a6\") " pod="openshift-monitoring/node-exporter-5ngwm"
Sep 08 02:00:00.334051 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:00:00.333912    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/167f56fa-baa2-41bf-8a91-0b63f41213a6-node-exporter-wtmp\") pod \"node-exporter-5ngwm\" (UID: \"167f56fa-baa2-41bf-8a91-0b63f41213a6\") " pod="openshift-monitoring/node-exporter-5ngwm"
Sep 08 02:00:00.334051 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:00:00.333944    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/167f56fa-baa2-41bf-8a91-0b63f41213a6-node-exporter-textfile\") pod \"node-exporter-5ngwm\" (UID: \"167f56fa-baa2-41bf-8a91-0b63f41213a6\") " pod="openshift-monitoring/node-exporter-5ngwm"
Sep 08 02:00:00.334051 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:00:00.333997    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/167f56fa-baa2-41bf-8a91-0b63f41213a6-node-exporter-accelerators-collector-config\") pod \"node-exporter-5ngwm\" (UID: \"167f56fa-baa2-41bf-8a91-0b63f41213a6\") " pod="openshift-monitoring/node-exporter-5ngwm"
Sep 08 02:00:00.334365 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:00:00.334333    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/167f56fa-baa2-41bf-8a91-0b63f41213a6-metrics-client-ca\") pod \"node-exporter-5ngwm\" (UID: \"167f56fa-baa2-41bf-8a91-0b63f41213a6\") " pod="openshift-monitoring/node-exporter-5ngwm"
Sep 08 02:00:00.336068 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:00:00.336045    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/167f56fa-baa2-41bf-8a91-0b63f41213a6-node-exporter-tls\") pod \"node-exporter-5ngwm\" (UID: \"167f56fa-baa2-41bf-8a91-0b63f41213a6\") " pod="openshift-monitoring/node-exporter-5ngwm"
Sep 08 02:00:00.336377 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:00:00.336355    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/167f56fa-baa2-41bf-8a91-0b63f41213a6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5ngwm\" (UID: \"167f56fa-baa2-41bf-8a91-0b63f41213a6\") " pod="openshift-monitoring/node-exporter-5ngwm"
Sep 08 02:00:00.345120 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:00:00.345097    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwm8x\" (UniqueName: \"kubernetes.io/projected/167f56fa-baa2-41bf-8a91-0b63f41213a6-kube-api-access-mwm8x\") pod \"node-exporter-5ngwm\" (UID: \"167f56fa-baa2-41bf-8a91-0b63f41213a6\") " pod="openshift-monitoring/node-exporter-5ngwm"
Sep 08 02:00:00.423160 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:00:00.423100    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5ngwm"
Sep 08 02:00:05.581895 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:00:05.581863    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-68cff75bfd-p6b52"
Sep 08 02:00:08.169351 ip-10-0-142-78 kubenswrapper[2547]: W0908 02:00:08.169312    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod167f56fa_baa2_41bf_8a91_0b63f41213a6.slice/crio-6de10512dd889a40f8508a5aec682179e706b5f7b32a301833d2c66ce928a0bf WatchSource:0}: Error finding container 6de10512dd889a40f8508a5aec682179e706b5f7b32a301833d2c66ce928a0bf: Status 404 returned error can't find the container with id 6de10512dd889a40f8508a5aec682179e706b5f7b32a301833d2c66ce928a0bf
Sep 08 02:00:08.395942 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:00:08.395902    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2v24n" event={"ID":"430aad96-9d26-4055-8133-b7f50b0f2978","Type":"ContainerStarted","Data":"426f6fecad43d0510d0b52f451b564fa1a0b47dc4a7afd3d94dfbc600cabaa68"}
Sep 08 02:00:08.396131 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:00:08.396038    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-2v24n"
Sep 08 02:00:08.397499 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:00:08.397471    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5ngwm" event={"ID":"167f56fa-baa2-41bf-8a91-0b63f41213a6","Type":"ContainerStarted","Data":"6de10512dd889a40f8508a5aec682179e706b5f7b32a301833d2c66ce928a0bf"}
Sep 08 02:00:08.403777 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:00:08.403749    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-6qfk4" event={"ID":"7ad15f65-08f1-4e92-a163-93131006559a","Type":"ContainerStarted","Data":"930a5d1bbbfca04eac640561d69319c6b391ee1350ab26865661921a4bab1e25"}
Sep 08 02:00:08.403991 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:00:08.403967    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-85d98c998f-6qfk4"
Sep 08 02:00:08.405558 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:00:08.405534    2547 patch_prober.go:28] interesting pod/downloads-85d98c998f-6qfk4 container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.133.0.15:8080/\": dial tcp 10.133.0.15:8080: connect: connection refused" start-of-body=
Sep 08 02:00:08.405673 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:00:08.405587    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-85d98c998f-6qfk4" podUID="7ad15f65-08f1-4e92-a163-93131006559a" containerName="download-server" probeResult="failure" output="Get \"http://10.133.0.15:8080/\": dial tcp 10.133.0.15:8080: connect: connection refused"
Sep 08 02:00:08.405673 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:00:08.405620    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-l7hw8" event={"ID":"917f6c17-7902-48d3-a0f4-535e5b602d26","Type":"ContainerStarted","Data":"5b29b15e1c8abeea8f5fb6be91e1a9081f58b2911452317432737a967f81b466"}
Sep 08 02:00:08.410500 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:00:08.410457    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-2v24n" podStartSLOduration=66.462736461 podStartE2EDuration="1m19.410444685s" totalImagesPullingTime="12.947708224s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 01:58:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:00:08.409245632 +0000 UTC m=+79.933020716" watchObservedRunningTime="2026-09-08 02:00:08.410444685 +0000 UTC m=+79.934219765"
Sep 08 02:00:08.423992 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:00:08.423940    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-85d98c998f-6qfk4" podStartSLOduration=1.5223690680000002 podStartE2EDuration="23.423924291s" totalImagesPullingTime="21.901555223s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 01:59:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:00:08.422526987 +0000 UTC m=+79.946302070" watchObservedRunningTime="2026-09-08 02:00:08.423924291 +0000 UTC m=+79.947699427"
Sep 08 02:00:08.437655 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:00:08.437600    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-l7hw8" podStartSLOduration=66.445168226 podStartE2EDuration="1m19.437585768s" totalImagesPullingTime="12.992417542s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 01:58:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:00:08.436114616 +0000 UTC m=+79.959889698" watchObservedRunningTime="2026-09-08 02:00:08.437585768 +0000 UTC m=+79.961360848"
Sep 08 02:00:09.298129 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:00:09.298091    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-db5d55684-xlxmr"
Sep 08 02:00:09.420875 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:00:09.420840    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-85d98c998f-6qfk4"
Sep 08 02:00:10.417193 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:00:10.417152    2547 generic.go:356] "Generic (PLEG): container finished" podID="167f56fa-baa2-41bf-8a91-0b63f41213a6" containerID="f0bb8b568388fe320bc454a063b04836975b703764b3333361b5cae6bbfc0021" exitCode=0
Sep 08 02:00:10.417682 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:00:10.417257    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5ngwm" event={"ID":"167f56fa-baa2-41bf-8a91-0b63f41213a6","Type":"ContainerDied","Data":"f0bb8b568388fe320bc454a063b04836975b703764b3333361b5cae6bbfc0021"}
Sep 08 02:00:10.595557 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:00:10.595507    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-68cff75bfd-p6b52" podUID="c7d9f993-10b4-4226-8193-a7599e0d3024" containerName="registry" containerID="cri-o://79405026c9980ab78669ba6e905ae0c2b051d55e969a7ab47f3fb3cd9f71c113" gracePeriod=30
Sep 08 02:00:10.900730 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:00:10.900691    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68cff75bfd-p6b52"
Sep 08 02:00:11.024122 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:00:11.024031    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c7d9f993-10b4-4226-8193-a7599e0d3024-trusted-ca\") pod \"c7d9f993-10b4-4226-8193-a7599e0d3024\" (UID: \"c7d9f993-10b4-4226-8193-a7599e0d3024\") "
Sep 08 02:00:11.024122 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:00:11.024112    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c7d9f993-10b4-4226-8193-a7599e0d3024-image-registry-private-configuration\") pod \"c7d9f993-10b4-4226-8193-a7599e0d3024\" (UID: \"c7d9f993-10b4-4226-8193-a7599e0d3024\") "
Sep 08 02:00:11.024362 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:00:11.024169    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c7d9f993-10b4-4226-8193-a7599e0d3024-bound-sa-token\") pod \"c7d9f993-10b4-4226-8193-a7599e0d3024\" (UID: \"c7d9f993-10b4-4226-8193-a7599e0d3024\") "
Sep 08 02:00:11.024362 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:00:11.024219    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c7d9f993-10b4-4226-8193-a7599e0d3024-registry-certificates\") pod \"c7d9f993-10b4-4226-8193-a7599e0d3024\" (UID: \"c7d9f993-10b4-4226-8193-a7599e0d3024\") "
Sep 08 02:00:11.024362 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:00:11.024276    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pkztm\" (UniqueName: \"kubernetes.io/projected/c7d9f993-10b4-4226-8193-a7599e0d3024-kube-api-access-pkztm\") pod \"c7d9f993-10b4-4226-8193-a7599e0d3024\" (UID: \"c7d9f993-10b4-4226-8193-a7599e0d3024\") "
Sep 08 02:00:11.024362 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:00:11.024308    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c7d9f993-10b4-4226-8193-a7599e0d3024-installation-pull-secrets\") pod \"c7d9f993-10b4-4226-8193-a7599e0d3024\" (UID: \"c7d9f993-10b4-4226-8193-a7599e0d3024\") "
Sep 08 02:00:11.024362 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:00:11.024348    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c7d9f993-10b4-4226-8193-a7599e0d3024-ca-trust-extracted\") pod \"c7d9f993-10b4-4226-8193-a7599e0d3024\" (UID: \"c7d9f993-10b4-4226-8193-a7599e0d3024\") "
Sep 08 02:00:11.024594 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:00:11.024375    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c7d9f993-10b4-4226-8193-a7599e0d3024-registry-tls\") pod \"c7d9f993-10b4-4226-8193-a7599e0d3024\" (UID: \"c7d9f993-10b4-4226-8193-a7599e0d3024\") "
Sep 08 02:00:11.024594 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:00:11.024533    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7d9f993-10b4-4226-8193-a7599e0d3024-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "c7d9f993-10b4-4226-8193-a7599e0d3024" (UID: "c7d9f993-10b4-4226-8193-a7599e0d3024"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 02:00:11.027871 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:00:11.024993    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7d9f993-10b4-4226-8193-a7599e0d3024-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "c7d9f993-10b4-4226-8193-a7599e0d3024" (UID: "c7d9f993-10b4-4226-8193-a7599e0d3024"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 02:00:11.027871 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:00:11.027198    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7d9f993-10b4-4226-8193-a7599e0d3024-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "c7d9f993-10b4-4226-8193-a7599e0d3024" (UID: "c7d9f993-10b4-4226-8193-a7599e0d3024"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 02:00:11.027871 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:00:11.027474    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7d9f993-10b4-4226-8193-a7599e0d3024-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "c7d9f993-10b4-4226-8193-a7599e0d3024" (UID: "c7d9f993-10b4-4226-8193-a7599e0d3024"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 02:00:11.027871 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:00:11.027536    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7d9f993-10b4-4226-8193-a7599e0d3024-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "c7d9f993-10b4-4226-8193-a7599e0d3024" (UID: "c7d9f993-10b4-4226-8193-a7599e0d3024"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 02:00:11.028527 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:00:11.028020    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7d9f993-10b4-4226-8193-a7599e0d3024-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "c7d9f993-10b4-4226-8193-a7599e0d3024" (UID: "c7d9f993-10b4-4226-8193-a7599e0d3024"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 02:00:11.028672 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:00:11.028642    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7d9f993-10b4-4226-8193-a7599e0d3024-kube-api-access-pkztm" (OuterVolumeSpecName: "kube-api-access-pkztm") pod "c7d9f993-10b4-4226-8193-a7599e0d3024" (UID: "c7d9f993-10b4-4226-8193-a7599e0d3024"). InnerVolumeSpecName "kube-api-access-pkztm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 02:00:11.045878 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:00:11.045732    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7d9f993-10b4-4226-8193-a7599e0d3024-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "c7d9f993-10b4-4226-8193-a7599e0d3024" (UID: "c7d9f993-10b4-4226-8193-a7599e0d3024"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 02:00:11.125843 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:00:11.125803    2547 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c7d9f993-10b4-4226-8193-a7599e0d3024-registry-certificates\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:00:11.125843 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:00:11.125838    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkztm\" (UniqueName: \"kubernetes.io/projected/c7d9f993-10b4-4226-8193-a7599e0d3024-kube-api-access-pkztm\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:00:11.125843 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:00:11.125851    2547 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c7d9f993-10b4-4226-8193-a7599e0d3024-installation-pull-secrets\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:00:11.126084 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:00:11.125863    2547 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c7d9f993-10b4-4226-8193-a7599e0d3024-ca-trust-extracted\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:00:11.126084 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:00:11.125875    2547 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c7d9f993-10b4-4226-8193-a7599e0d3024-registry-tls\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:00:11.126084 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:00:11.125888    2547 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c7d9f993-10b4-4226-8193-a7599e0d3024-trusted-ca\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:00:11.126084 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:00:11.125900    2547 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c7d9f993-10b4-4226-8193-a7599e0d3024-image-registry-private-configuration\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:00:11.126084 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:00:11.125913    2547 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c7d9f993-10b4-4226-8193-a7599e0d3024-bound-sa-token\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:00:11.423288 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:00:11.423254    2547 generic.go:356] "Generic (PLEG): container finished" podID="c7d9f993-10b4-4226-8193-a7599e0d3024" containerID="79405026c9980ab78669ba6e905ae0c2b051d55e969a7ab47f3fb3cd9f71c113" exitCode=0
Sep 08 02:00:11.423761 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:00:11.423328    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68cff75bfd-p6b52"
Sep 08 02:00:11.423761 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:00:11.423356    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-68cff75bfd-p6b52" event={"ID":"c7d9f993-10b4-4226-8193-a7599e0d3024","Type":"ContainerDied","Data":"79405026c9980ab78669ba6e905ae0c2b051d55e969a7ab47f3fb3cd9f71c113"}
Sep 08 02:00:11.423761 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:00:11.423418    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-68cff75bfd-p6b52" event={"ID":"c7d9f993-10b4-4226-8193-a7599e0d3024","Type":"ContainerDied","Data":"f144940c7d48fefc099c1ecf97efdd3f2526fe4a3aaece270839646d6807fb12"}
Sep 08 02:00:11.423761 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:00:11.423471    2547 scope.go:117] "RemoveContainer" containerID="79405026c9980ab78669ba6e905ae0c2b051d55e969a7ab47f3fb3cd9f71c113"
Sep 08 02:00:11.426296 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:00:11.426269    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5ngwm" event={"ID":"167f56fa-baa2-41bf-8a91-0b63f41213a6","Type":"ContainerStarted","Data":"907b843fa705e86de24c3b39b3bedf021aa65ce9a3d7cfa29132883d34c6d1c9"}
Sep 08 02:00:11.426419 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:00:11.426326    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5ngwm" event={"ID":"167f56fa-baa2-41bf-8a91-0b63f41213a6","Type":"ContainerStarted","Data":"97987699a27417ffe509275f65b20de5a5c96b1a90c62cd4ba678bab9c7dae90"}
Sep 08 02:00:11.442768 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:00:11.442742    2547 scope.go:117] "RemoveContainer" containerID="79405026c9980ab78669ba6e905ae0c2b051d55e969a7ab47f3fb3cd9f71c113"
Sep 08 02:00:11.443259 ip-10-0-142-78 kubenswrapper[2547]: E0908 02:00:11.443223    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79405026c9980ab78669ba6e905ae0c2b051d55e969a7ab47f3fb3cd9f71c113\": container with ID starting with 79405026c9980ab78669ba6e905ae0c2b051d55e969a7ab47f3fb3cd9f71c113 not found: ID does not exist" containerID="79405026c9980ab78669ba6e905ae0c2b051d55e969a7ab47f3fb3cd9f71c113"
Sep 08 02:00:11.443362 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:00:11.443267    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79405026c9980ab78669ba6e905ae0c2b051d55e969a7ab47f3fb3cd9f71c113"} err="failed to get container status \"79405026c9980ab78669ba6e905ae0c2b051d55e969a7ab47f3fb3cd9f71c113\": rpc error: code = NotFound desc = could not find container \"79405026c9980ab78669ba6e905ae0c2b051d55e969a7ab47f3fb3cd9f71c113\": container with ID starting with 79405026c9980ab78669ba6e905ae0c2b051d55e969a7ab47f3fb3cd9f71c113 not found: ID does not exist"
Sep 08 02:00:11.445120 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:00:11.445070    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-5ngwm" podStartSLOduration=9.795037341 podStartE2EDuration="11.44505662s" totalImagesPullingTime="1.557295894s" totalInitContainerRuntime="92.723385ms" isStatefulPod=true podCreationTimestamp="2026-09-08 02:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:00:11.44365184 +0000 UTC m=+82.967426923" watchObservedRunningTime="2026-09-08 02:00:11.44505662 +0000 UTC m=+82.968831700"
Sep 08 02:00:11.459051 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:00:11.459010    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-68cff75bfd-p6b52"]
Sep 08 02:00:11.460910 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:00:11.460886    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-68cff75bfd-p6b52"]
Sep 08 02:00:12.994009 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:00:12.993970    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7d9f993-10b4-4226-8193-a7599e0d3024" path="/var/lib/kubelet/pods/c7d9f993-10b4-4226-8193-a7599e0d3024/volumes"
Sep 08 02:00:39.416783 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:00:39.416753    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-2v24n"
Sep 08 02:03:28.141787 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:03:28.141752    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6566bd6ff-rxqm5"]
Sep 08 02:03:28.142234 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:03:28.141991    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7d9f993-10b4-4226-8193-a7599e0d3024" containerName="registry"
Sep 08 02:03:28.142234 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:03:28.142002    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7d9f993-10b4-4226-8193-a7599e0d3024" containerName="registry"
Sep 08 02:03:28.142234 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:03:28.142056    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7d9f993-10b4-4226-8193-a7599e0d3024" containerName="registry"
Sep 08 02:03:28.143748 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:03:28.143731    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6566bd6ff-rxqm5"
Sep 08 02:03:28.147801 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:03:28.147780    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 08 02:03:28.147926 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:03:28.147800    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 08 02:03:28.147926 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:03:28.147828    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 08 02:03:28.148314 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:03:28.148298    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-dzwkl\""
Sep 08 02:03:28.151711 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:03:28.151692    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 08 02:03:28.167064 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:03:28.167029    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6566bd6ff-rxqm5"]
Sep 08 02:03:28.278180 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:03:28.278122    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/277e92d6-e780-4011-a698-6f00a205e02b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6566bd6ff-rxqm5\" (UID: \"277e92d6-e780-4011-a698-6f00a205e02b\") " pod="opendatahub/opendatahub-operator-controller-manager-6566bd6ff-rxqm5"
Sep 08 02:03:28.278180 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:03:28.278192    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/277e92d6-e780-4011-a698-6f00a205e02b-manifests\") pod \"opendatahub-operator-controller-manager-6566bd6ff-rxqm5\" (UID: \"277e92d6-e780-4011-a698-6f00a205e02b\") " pod="opendatahub/opendatahub-operator-controller-manager-6566bd6ff-rxqm5"
Sep 08 02:03:28.278399 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:03:28.278262    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/277e92d6-e780-4011-a698-6f00a205e02b-tmp\") pod \"opendatahub-operator-controller-manager-6566bd6ff-rxqm5\" (UID: \"277e92d6-e780-4011-a698-6f00a205e02b\") " pod="opendatahub/opendatahub-operator-controller-manager-6566bd6ff-rxqm5"
Sep 08 02:03:28.278399 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:03:28.278292    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/277e92d6-e780-4011-a698-6f00a205e02b-webhook-cert\") pod \"opendatahub-operator-controller-manager-6566bd6ff-rxqm5\" (UID: \"277e92d6-e780-4011-a698-6f00a205e02b\") " pod="opendatahub/opendatahub-operator-controller-manager-6566bd6ff-rxqm5"
Sep 08 02:03:28.278399 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:03:28.278319    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8k8x\" (UniqueName: \"kubernetes.io/projected/277e92d6-e780-4011-a698-6f00a205e02b-kube-api-access-h8k8x\") pod \"opendatahub-operator-controller-manager-6566bd6ff-rxqm5\" (UID: \"277e92d6-e780-4011-a698-6f00a205e02b\") " pod="opendatahub/opendatahub-operator-controller-manager-6566bd6ff-rxqm5"
Sep 08 02:03:28.379297 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:03:28.379248    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/277e92d6-e780-4011-a698-6f00a205e02b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6566bd6ff-rxqm5\" (UID: \"277e92d6-e780-4011-a698-6f00a205e02b\") " pod="opendatahub/opendatahub-operator-controller-manager-6566bd6ff-rxqm5"
Sep 08 02:03:28.379297 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:03:28.379304    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/277e92d6-e780-4011-a698-6f00a205e02b-manifests\") pod \"opendatahub-operator-controller-manager-6566bd6ff-rxqm5\" (UID: \"277e92d6-e780-4011-a698-6f00a205e02b\") " pod="opendatahub/opendatahub-operator-controller-manager-6566bd6ff-rxqm5"
Sep 08 02:03:28.379555 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:03:28.379326    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/277e92d6-e780-4011-a698-6f00a205e02b-tmp\") pod \"opendatahub-operator-controller-manager-6566bd6ff-rxqm5\" (UID: \"277e92d6-e780-4011-a698-6f00a205e02b\") " pod="opendatahub/opendatahub-operator-controller-manager-6566bd6ff-rxqm5"
Sep 08 02:03:28.379555 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:03:28.379349    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/277e92d6-e780-4011-a698-6f00a205e02b-webhook-cert\") pod \"opendatahub-operator-controller-manager-6566bd6ff-rxqm5\" (UID: \"277e92d6-e780-4011-a698-6f00a205e02b\") " pod="opendatahub/opendatahub-operator-controller-manager-6566bd6ff-rxqm5"
Sep 08 02:03:28.379555 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:03:28.379371    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8k8x\" (UniqueName: \"kubernetes.io/projected/277e92d6-e780-4011-a698-6f00a205e02b-kube-api-access-h8k8x\") pod \"opendatahub-operator-controller-manager-6566bd6ff-rxqm5\" (UID: \"277e92d6-e780-4011-a698-6f00a205e02b\") " pod="opendatahub/opendatahub-operator-controller-manager-6566bd6ff-rxqm5"
Sep 08 02:03:28.379859 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:03:28.379828    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/277e92d6-e780-4011-a698-6f00a205e02b-manifests\") pod \"opendatahub-operator-controller-manager-6566bd6ff-rxqm5\" (UID: \"277e92d6-e780-4011-a698-6f00a205e02b\") " pod="opendatahub/opendatahub-operator-controller-manager-6566bd6ff-rxqm5"
Sep 08 02:03:28.381600 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:03:28.381574    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/277e92d6-e780-4011-a698-6f00a205e02b-tmp\") pod \"opendatahub-operator-controller-manager-6566bd6ff-rxqm5\" (UID: \"277e92d6-e780-4011-a698-6f00a205e02b\") " pod="opendatahub/opendatahub-operator-controller-manager-6566bd6ff-rxqm5"
Sep 08 02:03:28.381892 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:03:28.381875    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/277e92d6-e780-4011-a698-6f00a205e02b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6566bd6ff-rxqm5\" (UID: \"277e92d6-e780-4011-a698-6f00a205e02b\") " pod="opendatahub/opendatahub-operator-controller-manager-6566bd6ff-rxqm5"
Sep 08 02:03:28.381932 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:03:28.381904    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/277e92d6-e780-4011-a698-6f00a205e02b-webhook-cert\") pod \"opendatahub-operator-controller-manager-6566bd6ff-rxqm5\" (UID: \"277e92d6-e780-4011-a698-6f00a205e02b\") " pod="opendatahub/opendatahub-operator-controller-manager-6566bd6ff-rxqm5"
Sep 08 02:03:28.406186 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:03:28.406101    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8k8x\" (UniqueName: \"kubernetes.io/projected/277e92d6-e780-4011-a698-6f00a205e02b-kube-api-access-h8k8x\") pod \"opendatahub-operator-controller-manager-6566bd6ff-rxqm5\" (UID: \"277e92d6-e780-4011-a698-6f00a205e02b\") " pod="opendatahub/opendatahub-operator-controller-manager-6566bd6ff-rxqm5"
Sep 08 02:03:28.453612 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:03:28.453578    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6566bd6ff-rxqm5"
Sep 08 02:03:28.734585 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:03:28.734472    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6566bd6ff-rxqm5"]
Sep 08 02:03:28.739276 ip-10-0-142-78 kubenswrapper[2547]: W0908 02:03:28.739248    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod277e92d6_e780_4011_a698_6f00a205e02b.slice/crio-ac0ecca1440595d4eb51ffa979926e3d07538c46cab7855172d924b86cdf18ea WatchSource:0}: Error finding container ac0ecca1440595d4eb51ffa979926e3d07538c46cab7855172d924b86cdf18ea: Status 404 returned error can't find the container with id ac0ecca1440595d4eb51ffa979926e3d07538c46cab7855172d924b86cdf18ea
Sep 08 02:03:29.089741 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:03:29.089656    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6566bd6ff-rxqm5" event={"ID":"277e92d6-e780-4011-a698-6f00a205e02b","Type":"ContainerStarted","Data":"ac0ecca1440595d4eb51ffa979926e3d07538c46cab7855172d924b86cdf18ea"}
Sep 08 02:03:32.103685 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:03:32.103585    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6566bd6ff-rxqm5" event={"ID":"277e92d6-e780-4011-a698-6f00a205e02b","Type":"ContainerStarted","Data":"463bb4f5de25a79fcc34b985fb67c34b0768f25815ac00f6abf0571f557e1832"}
Sep 08 02:03:33.109236 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:03:33.109131    2547 generic.go:356] "Generic (PLEG): container finished" podID="277e92d6-e780-4011-a698-6f00a205e02b" containerID="463bb4f5de25a79fcc34b985fb67c34b0768f25815ac00f6abf0571f557e1832" exitCode=0
Sep 08 02:03:33.109236 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:03:33.109205    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6566bd6ff-rxqm5" event={"ID":"277e92d6-e780-4011-a698-6f00a205e02b","Type":"ContainerDied","Data":"463bb4f5de25a79fcc34b985fb67c34b0768f25815ac00f6abf0571f557e1832"}
Sep 08 02:03:34.114963 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:03:34.114921    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6566bd6ff-rxqm5" event={"ID":"277e92d6-e780-4011-a698-6f00a205e02b","Type":"ContainerStarted","Data":"38edacea1368f57b4234b5648ea5b51f0a668dbe9f9026766c72f7d89cedc6ea"}
Sep 08 02:03:34.115370 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:03:34.115025    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6566bd6ff-rxqm5"
Sep 08 02:03:34.137495 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:03:34.137443    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6566bd6ff-rxqm5" podStartSLOduration=1.7917155710000001 podStartE2EDuration="6.137425471s" totalImagesPullingTime="3.343306309s" totalInitContainerRuntime="1.002403591s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:03:28 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:03:34.136687943 +0000 UTC m=+285.660463034" watchObservedRunningTime="2026-09-08 02:03:34.137425471 +0000 UTC m=+285.661200550"
Sep 08 02:03:45.121259 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:03:45.121225    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6566bd6ff-rxqm5"
Sep 08 02:03:47.753264 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:03:47.753224    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-9c6fc765d-j4v4c"]
Sep 08 02:03:47.758863 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:03:47.758842    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-9c6fc765d-j4v4c"
Sep 08 02:03:47.777561 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:03:47.777529    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-vkdjb\""
Sep 08 02:03:47.777561 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:03:47.777544    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 08 02:03:47.779655 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:03:47.779638    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 08 02:03:47.780400 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:03:47.780384    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 08 02:03:47.784470 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:03:47.784452    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 08 02:03:47.797047 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:03:47.797020    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-9c6fc765d-j4v4c"]
Sep 08 02:03:47.815152 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:03:47.815102    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21eff83a-e938-4010-b027-0088de0fc9c8-tls-certs\") pod \"kube-auth-proxy-9c6fc765d-j4v4c\" (UID: \"21eff83a-e938-4010-b027-0088de0fc9c8\") " pod="openshift-ingress/kube-auth-proxy-9c6fc765d-j4v4c"
Sep 08 02:03:47.815293 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:03:47.815160    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5742\" (UniqueName: \"kubernetes.io/projected/21eff83a-e938-4010-b027-0088de0fc9c8-kube-api-access-g5742\") pod \"kube-auth-proxy-9c6fc765d-j4v4c\" (UID: \"21eff83a-e938-4010-b027-0088de0fc9c8\") " pod="openshift-ingress/kube-auth-proxy-9c6fc765d-j4v4c"
Sep 08 02:03:47.815293 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:03:47.815190    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/21eff83a-e938-4010-b027-0088de0fc9c8-tmp\") pod \"kube-auth-proxy-9c6fc765d-j4v4c\" (UID: \"21eff83a-e938-4010-b027-0088de0fc9c8\") " pod="openshift-ingress/kube-auth-proxy-9c6fc765d-j4v4c"
Sep 08 02:03:47.916196 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:03:47.916126    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21eff83a-e938-4010-b027-0088de0fc9c8-tls-certs\") pod \"kube-auth-proxy-9c6fc765d-j4v4c\" (UID: \"21eff83a-e938-4010-b027-0088de0fc9c8\") " pod="openshift-ingress/kube-auth-proxy-9c6fc765d-j4v4c"
Sep 08 02:03:47.916196 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:03:47.916196    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5742\" (UniqueName: \"kubernetes.io/projected/21eff83a-e938-4010-b027-0088de0fc9c8-kube-api-access-g5742\") pod \"kube-auth-proxy-9c6fc765d-j4v4c\" (UID: \"21eff83a-e938-4010-b027-0088de0fc9c8\") " pod="openshift-ingress/kube-auth-proxy-9c6fc765d-j4v4c"
Sep 08 02:03:47.916388 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:03:47.916218    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/21eff83a-e938-4010-b027-0088de0fc9c8-tmp\") pod \"kube-auth-proxy-9c6fc765d-j4v4c\" (UID: \"21eff83a-e938-4010-b027-0088de0fc9c8\") " pod="openshift-ingress/kube-auth-proxy-9c6fc765d-j4v4c"
Sep 08 02:03:47.918480 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:03:47.918450    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/21eff83a-e938-4010-b027-0088de0fc9c8-tmp\") pod \"kube-auth-proxy-9c6fc765d-j4v4c\" (UID: \"21eff83a-e938-4010-b027-0088de0fc9c8\") " pod="openshift-ingress/kube-auth-proxy-9c6fc765d-j4v4c"
Sep 08 02:03:47.918622 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:03:47.918602    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21eff83a-e938-4010-b027-0088de0fc9c8-tls-certs\") pod \"kube-auth-proxy-9c6fc765d-j4v4c\" (UID: \"21eff83a-e938-4010-b027-0088de0fc9c8\") " pod="openshift-ingress/kube-auth-proxy-9c6fc765d-j4v4c"
Sep 08 02:03:47.933812 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:03:47.933781    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5742\" (UniqueName: \"kubernetes.io/projected/21eff83a-e938-4010-b027-0088de0fc9c8-kube-api-access-g5742\") pod \"kube-auth-proxy-9c6fc765d-j4v4c\" (UID: \"21eff83a-e938-4010-b027-0088de0fc9c8\") " pod="openshift-ingress/kube-auth-proxy-9c6fc765d-j4v4c"
Sep 08 02:03:48.067723 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:03:48.067638    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-9c6fc765d-j4v4c"
Sep 08 02:03:48.347184 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:03:48.347092    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-9c6fc765d-j4v4c"]
Sep 08 02:03:48.350034 ip-10-0-142-78 kubenswrapper[2547]: W0908 02:03:48.350011    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21eff83a_e938_4010_b027_0088de0fc9c8.slice/crio-8cf9e322af7513af17801eed25a2bfcfe2ad1da6cdc1ddf71033d257365dca30 WatchSource:0}: Error finding container 8cf9e322af7513af17801eed25a2bfcfe2ad1da6cdc1ddf71033d257365dca30: Status 404 returned error can't find the container with id 8cf9e322af7513af17801eed25a2bfcfe2ad1da6cdc1ddf71033d257365dca30
Sep 08 02:03:48.861784 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:03:48.861753    2547 kubelet.go:1648] "Image garbage collection succeeded"
Sep 08 02:03:49.171125 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:03:49.171081    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-9c6fc765d-j4v4c" event={"ID":"21eff83a-e938-4010-b027-0088de0fc9c8","Type":"ContainerStarted","Data":"8cf9e322af7513af17801eed25a2bfcfe2ad1da6cdc1ddf71033d257365dca30"}
Sep 08 02:03:51.159573 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:03:51.159541    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 08 02:03:52.183321 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:03:52.183280    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-9c6fc765d-j4v4c" event={"ID":"21eff83a-e938-4010-b027-0088de0fc9c8","Type":"ContainerStarted","Data":"4ea29d5b22103f2b4d853faaa923db0b7a67d8141c48f0a850b85a80c841156b"}
Sep 08 02:03:52.200560 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:03:52.200503    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-9c6fc765d-j4v4c" podStartSLOduration=2.394783674 podStartE2EDuration="5.200486355s" totalImagesPullingTime="2.805702681s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:03:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:03:52.199085925 +0000 UTC m=+303.722861006" watchObservedRunningTime="2026-09-08 02:03:52.200486355 +0000 UTC m=+303.724261466"
Sep 08 02:03:59.214407 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:03:59.214368    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-4fwjq"]
Sep 08 02:03:59.219019 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:03:59.218994    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-4fwjq"
Sep 08 02:03:59.226244 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:03:59.226220    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-q9hbq\""
Sep 08 02:03:59.226363 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:03:59.226269    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Sep 08 02:03:59.235346 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:03:59.235317    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-4fwjq"]
Sep 08 02:03:59.302896 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:03:59.302861    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4c7rx\" (UniqueName: \"kubernetes.io/projected/6aea39a7-0b3c-4f02-b5cc-b86f82d41c9c-kube-api-access-4c7rx\") pod \"model-serving-api-54d89bd5fc-4fwjq\" (UID: \"6aea39a7-0b3c-4f02-b5cc-b86f82d41c9c\") " pod="opendatahub/model-serving-api-54d89bd5fc-4fwjq"
Sep 08 02:03:59.303068 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:03:59.302908    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6aea39a7-0b3c-4f02-b5cc-b86f82d41c9c-tls-certs\") pod \"model-serving-api-54d89bd5fc-4fwjq\" (UID: \"6aea39a7-0b3c-4f02-b5cc-b86f82d41c9c\") " pod="opendatahub/model-serving-api-54d89bd5fc-4fwjq"
Sep 08 02:03:59.403496 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:03:59.403452    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4c7rx\" (UniqueName: \"kubernetes.io/projected/6aea39a7-0b3c-4f02-b5cc-b86f82d41c9c-kube-api-access-4c7rx\") pod \"model-serving-api-54d89bd5fc-4fwjq\" (UID: \"6aea39a7-0b3c-4f02-b5cc-b86f82d41c9c\") " pod="opendatahub/model-serving-api-54d89bd5fc-4fwjq"
Sep 08 02:03:59.403496 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:03:59.403498    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6aea39a7-0b3c-4f02-b5cc-b86f82d41c9c-tls-certs\") pod \"model-serving-api-54d89bd5fc-4fwjq\" (UID: \"6aea39a7-0b3c-4f02-b5cc-b86f82d41c9c\") " pod="opendatahub/model-serving-api-54d89bd5fc-4fwjq"
Sep 08 02:03:59.405939 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:03:59.405914    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6aea39a7-0b3c-4f02-b5cc-b86f82d41c9c-tls-certs\") pod \"model-serving-api-54d89bd5fc-4fwjq\" (UID: \"6aea39a7-0b3c-4f02-b5cc-b86f82d41c9c\") " pod="opendatahub/model-serving-api-54d89bd5fc-4fwjq"
Sep 08 02:03:59.414925 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:03:59.414892    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4c7rx\" (UniqueName: \"kubernetes.io/projected/6aea39a7-0b3c-4f02-b5cc-b86f82d41c9c-kube-api-access-4c7rx\") pod \"model-serving-api-54d89bd5fc-4fwjq\" (UID: \"6aea39a7-0b3c-4f02-b5cc-b86f82d41c9c\") " pod="opendatahub/model-serving-api-54d89bd5fc-4fwjq"
Sep 08 02:03:59.528672 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:03:59.528576    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-4fwjq"
Sep 08 02:03:59.820598 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:03:59.820514    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-4fwjq"]
Sep 08 02:03:59.823856 ip-10-0-142-78 kubenswrapper[2547]: W0908 02:03:59.823829    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6aea39a7_0b3c_4f02_b5cc_b86f82d41c9c.slice/crio-95f80b80080cee5c4c19a2c6501f573fa8f25189271e7bfb0e8965ba068f5292 WatchSource:0}: Error finding container 95f80b80080cee5c4c19a2c6501f573fa8f25189271e7bfb0e8965ba068f5292: Status 404 returned error can't find the container with id 95f80b80080cee5c4c19a2c6501f573fa8f25189271e7bfb0e8965ba068f5292
Sep 08 02:03:59.826156 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:03:59.826126    2547 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 02:04:00.211696 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:04:00.211657    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-4fwjq" event={"ID":"6aea39a7-0b3c-4f02-b5cc-b86f82d41c9c","Type":"ContainerStarted","Data":"95f80b80080cee5c4c19a2c6501f573fa8f25189271e7bfb0e8965ba068f5292"}
Sep 08 02:04:02.220480 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:04:02.220438    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-4fwjq" event={"ID":"6aea39a7-0b3c-4f02-b5cc-b86f82d41c9c","Type":"ContainerStarted","Data":"0fd102dba8e0d39615d414c382f35808c3a1f0edde05205087ee623a3f4fce3d"}
Sep 08 02:04:02.220866 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:04:02.220579    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-4fwjq"
Sep 08 02:04:02.239274 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:04:02.239216    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-4fwjq" podStartSLOduration=1.340454819 podStartE2EDuration="3.239195978s" totalImagesPullingTime="1.898741159s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:03:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:04:02.238867794 +0000 UTC m=+313.762642875" watchObservedRunningTime="2026-09-08 02:04:02.239195978 +0000 UTC m=+313.762971078"
Sep 08 02:04:05.709171 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:04:05.709122    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5777c545f5-f4wtw"]
Sep 08 02:04:05.712398 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:04:05.712381    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5777c545f5-f4wtw"
Sep 08 02:04:05.714776 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:04:05.714753    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 08 02:04:05.714865 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:04:05.714807    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 08 02:04:05.715634 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:04:05.715611    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-nwh77\""
Sep 08 02:04:05.715732 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:04:05.715658    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 08 02:04:05.715732 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:04:05.715666    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 08 02:04:05.715732 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:04:05.715658    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 08 02:04:05.726632 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:04:05.726605    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5777c545f5-f4wtw"]
Sep 08 02:04:05.753953 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:04:05.753922    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e95368d2-40b8-48c9-a354-f5e274c7f359-cert\") pod \"lws-controller-manager-5777c545f5-f4wtw\" (UID: \"e95368d2-40b8-48c9-a354-f5e274c7f359\") " pod="openshift-lws-operator/lws-controller-manager-5777c545f5-f4wtw"
Sep 08 02:04:05.754120 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:04:05.753959    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e95368d2-40b8-48c9-a354-f5e274c7f359-manager-config\") pod \"lws-controller-manager-5777c545f5-f4wtw\" (UID: \"e95368d2-40b8-48c9-a354-f5e274c7f359\") " pod="openshift-lws-operator/lws-controller-manager-5777c545f5-f4wtw"
Sep 08 02:04:05.754120 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:04:05.754003    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqs2k\" (UniqueName: \"kubernetes.io/projected/e95368d2-40b8-48c9-a354-f5e274c7f359-kube-api-access-sqs2k\") pod \"lws-controller-manager-5777c545f5-f4wtw\" (UID: \"e95368d2-40b8-48c9-a354-f5e274c7f359\") " pod="openshift-lws-operator/lws-controller-manager-5777c545f5-f4wtw"
Sep 08 02:04:05.754120 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:04:05.754058    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e95368d2-40b8-48c9-a354-f5e274c7f359-metrics-cert\") pod \"lws-controller-manager-5777c545f5-f4wtw\" (UID: \"e95368d2-40b8-48c9-a354-f5e274c7f359\") " pod="openshift-lws-operator/lws-controller-manager-5777c545f5-f4wtw"
Sep 08 02:04:05.854613 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:04:05.854562    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqs2k\" (UniqueName: \"kubernetes.io/projected/e95368d2-40b8-48c9-a354-f5e274c7f359-kube-api-access-sqs2k\") pod \"lws-controller-manager-5777c545f5-f4wtw\" (UID: \"e95368d2-40b8-48c9-a354-f5e274c7f359\") " pod="openshift-lws-operator/lws-controller-manager-5777c545f5-f4wtw"
Sep 08 02:04:05.854613 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:04:05.854620    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e95368d2-40b8-48c9-a354-f5e274c7f359-metrics-cert\") pod \"lws-controller-manager-5777c545f5-f4wtw\" (UID: \"e95368d2-40b8-48c9-a354-f5e274c7f359\") " pod="openshift-lws-operator/lws-controller-manager-5777c545f5-f4wtw"
Sep 08 02:04:05.854848 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:04:05.854654    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e95368d2-40b8-48c9-a354-f5e274c7f359-cert\") pod \"lws-controller-manager-5777c545f5-f4wtw\" (UID: \"e95368d2-40b8-48c9-a354-f5e274c7f359\") " pod="openshift-lws-operator/lws-controller-manager-5777c545f5-f4wtw"
Sep 08 02:04:05.854848 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:04:05.854671    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e95368d2-40b8-48c9-a354-f5e274c7f359-manager-config\") pod \"lws-controller-manager-5777c545f5-f4wtw\" (UID: \"e95368d2-40b8-48c9-a354-f5e274c7f359\") " pod="openshift-lws-operator/lws-controller-manager-5777c545f5-f4wtw"
Sep 08 02:04:05.855320 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:04:05.855302    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e95368d2-40b8-48c9-a354-f5e274c7f359-manager-config\") pod \"lws-controller-manager-5777c545f5-f4wtw\" (UID: \"e95368d2-40b8-48c9-a354-f5e274c7f359\") " pod="openshift-lws-operator/lws-controller-manager-5777c545f5-f4wtw"
Sep 08 02:04:05.857334 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:04:05.857311    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e95368d2-40b8-48c9-a354-f5e274c7f359-metrics-cert\") pod \"lws-controller-manager-5777c545f5-f4wtw\" (UID: \"e95368d2-40b8-48c9-a354-f5e274c7f359\") " pod="openshift-lws-operator/lws-controller-manager-5777c545f5-f4wtw"
Sep 08 02:04:05.857334 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:04:05.857326    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e95368d2-40b8-48c9-a354-f5e274c7f359-cert\") pod \"lws-controller-manager-5777c545f5-f4wtw\" (UID: \"e95368d2-40b8-48c9-a354-f5e274c7f359\") " pod="openshift-lws-operator/lws-controller-manager-5777c545f5-f4wtw"
Sep 08 02:04:05.865975 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:04:05.865950    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqs2k\" (UniqueName: \"kubernetes.io/projected/e95368d2-40b8-48c9-a354-f5e274c7f359-kube-api-access-sqs2k\") pod \"lws-controller-manager-5777c545f5-f4wtw\" (UID: \"e95368d2-40b8-48c9-a354-f5e274c7f359\") " pod="openshift-lws-operator/lws-controller-manager-5777c545f5-f4wtw"
Sep 08 02:04:06.021154 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:04:06.021026    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5777c545f5-f4wtw"
Sep 08 02:04:06.302913 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:04:06.302832    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5777c545f5-f4wtw"]
Sep 08 02:04:06.306213 ip-10-0-142-78 kubenswrapper[2547]: W0908 02:04:06.306179    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode95368d2_40b8_48c9_a354_f5e274c7f359.slice/crio-d5b18a76c9017183833d906dcc63ffe6d2211f8b378023d11b711cdd722ac2ee WatchSource:0}: Error finding container d5b18a76c9017183833d906dcc63ffe6d2211f8b378023d11b711cdd722ac2ee: Status 404 returned error can't find the container with id d5b18a76c9017183833d906dcc63ffe6d2211f8b378023d11b711cdd722ac2ee
Sep 08 02:04:07.241018 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:04:07.240980    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5777c545f5-f4wtw" event={"ID":"e95368d2-40b8-48c9-a354-f5e274c7f359","Type":"ContainerStarted","Data":"d5b18a76c9017183833d906dcc63ffe6d2211f8b378023d11b711cdd722ac2ee"}
Sep 08 02:04:09.250177 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:04:09.250123    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5777c545f5-f4wtw" event={"ID":"e95368d2-40b8-48c9-a354-f5e274c7f359","Type":"ContainerStarted","Data":"e6217ecb8d21a23ba53f2b8440199c86c73e32cc78b0cbcf6a75210a43a86031"}
Sep 08 02:04:09.250595 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:04:09.250258    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5777c545f5-f4wtw"
Sep 08 02:04:09.290612 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:04:09.290551    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5777c545f5-f4wtw" podStartSLOduration=1.989065642 podStartE2EDuration="4.290534561s" totalImagesPullingTime="2.301468919s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:04:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:04:09.28630724 +0000 UTC m=+320.810082320" watchObservedRunningTime="2026-09-08 02:04:09.290534561 +0000 UTC m=+320.814309640"
Sep 08 02:04:13.228441 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:04:13.228408    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-4fwjq"
Sep 08 02:04:20.256667 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:04:20.256634    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5777c545f5-f4wtw"
Sep 08 02:07:07.506695 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:07:07.506655    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6cb48d57d-6l8qp"]
Sep 08 02:07:07.509104 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:07:07.509077    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6cb48d57d-6l8qp"
Sep 08 02:07:07.511468 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:07:07.511446    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 08 02:07:07.511468 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:07:07.511464    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 08 02:07:07.512172 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:07:07.512154    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-dc7lc\""
Sep 08 02:07:07.512172 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:07:07.512161    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 08 02:07:07.521068 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:07:07.521040    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6cb48d57d-6l8qp"]
Sep 08 02:07:07.642857 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:07:07.642807    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncr9f\" (UniqueName: \"kubernetes.io/projected/1983102f-b0ac-4e96-a32d-4f909739926d-kube-api-access-ncr9f\") pod \"maas-controller-6cb48d57d-6l8qp\" (UID: \"1983102f-b0ac-4e96-a32d-4f909739926d\") " pod="opendatahub/maas-controller-6cb48d57d-6l8qp"
Sep 08 02:07:07.642857 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:07:07.642860    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1983102f-b0ac-4e96-a32d-4f909739926d-metrics-certs\") pod \"maas-controller-6cb48d57d-6l8qp\" (UID: \"1983102f-b0ac-4e96-a32d-4f909739926d\") " pod="opendatahub/maas-controller-6cb48d57d-6l8qp"
Sep 08 02:07:07.643088 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:07:07.642881    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1983102f-b0ac-4e96-a32d-4f909739926d-webhook-cert\") pod \"maas-controller-6cb48d57d-6l8qp\" (UID: \"1983102f-b0ac-4e96-a32d-4f909739926d\") " pod="opendatahub/maas-controller-6cb48d57d-6l8qp"
Sep 08 02:07:07.743524 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:07:07.743489    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncr9f\" (UniqueName: \"kubernetes.io/projected/1983102f-b0ac-4e96-a32d-4f909739926d-kube-api-access-ncr9f\") pod \"maas-controller-6cb48d57d-6l8qp\" (UID: \"1983102f-b0ac-4e96-a32d-4f909739926d\") " pod="opendatahub/maas-controller-6cb48d57d-6l8qp"
Sep 08 02:07:07.743524 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:07:07.743529    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1983102f-b0ac-4e96-a32d-4f909739926d-metrics-certs\") pod \"maas-controller-6cb48d57d-6l8qp\" (UID: \"1983102f-b0ac-4e96-a32d-4f909739926d\") " pod="opendatahub/maas-controller-6cb48d57d-6l8qp"
Sep 08 02:07:07.743783 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:07:07.743547    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1983102f-b0ac-4e96-a32d-4f909739926d-webhook-cert\") pod \"maas-controller-6cb48d57d-6l8qp\" (UID: \"1983102f-b0ac-4e96-a32d-4f909739926d\") " pod="opendatahub/maas-controller-6cb48d57d-6l8qp"
Sep 08 02:07:07.743783 ip-10-0-142-78 kubenswrapper[2547]: E0908 02:07:07.743655    2547 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found
Sep 08 02:07:07.743783 ip-10-0-142-78 kubenswrapper[2547]: E0908 02:07:07.743712    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1983102f-b0ac-4e96-a32d-4f909739926d-webhook-cert podName:1983102f-b0ac-4e96-a32d-4f909739926d nodeName:}" failed. No retries permitted until 2026-09-08 02:07:08.243697561 +0000 UTC m=+499.767472620 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/1983102f-b0ac-4e96-a32d-4f909739926d-webhook-cert") pod "maas-controller-6cb48d57d-6l8qp" (UID: "1983102f-b0ac-4e96-a32d-4f909739926d") : secret "maas-controller-webhook-cert" not found
Sep 08 02:07:07.746105 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:07:07.746071    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1983102f-b0ac-4e96-a32d-4f909739926d-metrics-certs\") pod \"maas-controller-6cb48d57d-6l8qp\" (UID: \"1983102f-b0ac-4e96-a32d-4f909739926d\") " pod="opendatahub/maas-controller-6cb48d57d-6l8qp"
Sep 08 02:07:07.751836 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:07:07.751801    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncr9f\" (UniqueName: \"kubernetes.io/projected/1983102f-b0ac-4e96-a32d-4f909739926d-kube-api-access-ncr9f\") pod \"maas-controller-6cb48d57d-6l8qp\" (UID: \"1983102f-b0ac-4e96-a32d-4f909739926d\") " pod="opendatahub/maas-controller-6cb48d57d-6l8qp"
Sep 08 02:07:08.247080 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:07:08.247030    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1983102f-b0ac-4e96-a32d-4f909739926d-webhook-cert\") pod \"maas-controller-6cb48d57d-6l8qp\" (UID: \"1983102f-b0ac-4e96-a32d-4f909739926d\") " pod="opendatahub/maas-controller-6cb48d57d-6l8qp"
Sep 08 02:07:08.249511 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:07:08.249485    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1983102f-b0ac-4e96-a32d-4f909739926d-webhook-cert\") pod \"maas-controller-6cb48d57d-6l8qp\" (UID: \"1983102f-b0ac-4e96-a32d-4f909739926d\") " pod="opendatahub/maas-controller-6cb48d57d-6l8qp"
Sep 08 02:07:08.421196 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:07:08.421155    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6cb48d57d-6l8qp"
Sep 08 02:07:08.693123 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:07:08.693090    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6cb48d57d-6l8qp"]
Sep 08 02:07:08.696639 ip-10-0-142-78 kubenswrapper[2547]: W0908 02:07:08.696611    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1983102f_b0ac_4e96_a32d_4f909739926d.slice/crio-1f3bf541658e5b2ea62baab6279642f19756d021a849e67d73a49c4544cafd9a WatchSource:0}: Error finding container 1f3bf541658e5b2ea62baab6279642f19756d021a849e67d73a49c4544cafd9a: Status 404 returned error can't find the container with id 1f3bf541658e5b2ea62baab6279642f19756d021a849e67d73a49c4544cafd9a
Sep 08 02:07:08.946752 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:07:08.946653    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6cb48d57d-6l8qp" event={"ID":"1983102f-b0ac-4e96-a32d-4f909739926d","Type":"ContainerStarted","Data":"1f3bf541658e5b2ea62baab6279642f19756d021a849e67d73a49c4544cafd9a"}
Sep 08 02:07:11.961897 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:07:11.961857    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6cb48d57d-6l8qp" event={"ID":"1983102f-b0ac-4e96-a32d-4f909739926d","Type":"ContainerStarted","Data":"8ea44ac3a721cf61b1068b9900ff13df143871db85e92b87e8761e3fa3ed003a"}
Sep 08 02:07:11.962405 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:07:11.961991    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6cb48d57d-6l8qp"
Sep 08 02:07:11.980513 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:07:11.980456    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6cb48d57d-6l8qp" podStartSLOduration=2.350357556 podStartE2EDuration="4.980439292s" totalImagesPullingTime="2.630081736s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:07:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:07:11.979333898 +0000 UTC m=+503.503108998" watchObservedRunningTime="2026-09-08 02:07:11.980439292 +0000 UTC m=+503.504214370"
Sep 08 02:07:22.968844 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:07:22.968809    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6cb48d57d-6l8qp"
Sep 08 02:07:23.015619 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:07:23.015576    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6cb48d57d-6l8qp"]
Sep 08 02:07:23.015813 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:07:23.015752    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6cb48d57d-6l8qp" podUID="1983102f-b0ac-4e96-a32d-4f909739926d" containerName="manager" containerID="cri-o://8ea44ac3a721cf61b1068b9900ff13df143871db85e92b87e8761e3fa3ed003a" gracePeriod=30
Sep 08 02:07:23.253318 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:07:23.253291    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6cb48d57d-6l8qp"
Sep 08 02:07:23.372128 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:07:23.372096    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1983102f-b0ac-4e96-a32d-4f909739926d-metrics-certs\") pod \"1983102f-b0ac-4e96-a32d-4f909739926d\" (UID: \"1983102f-b0ac-4e96-a32d-4f909739926d\") "
Sep 08 02:07:23.372326 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:07:23.372189    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1983102f-b0ac-4e96-a32d-4f909739926d-webhook-cert\") pod \"1983102f-b0ac-4e96-a32d-4f909739926d\" (UID: \"1983102f-b0ac-4e96-a32d-4f909739926d\") "
Sep 08 02:07:23.372326 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:07:23.372227    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ncr9f\" (UniqueName: \"kubernetes.io/projected/1983102f-b0ac-4e96-a32d-4f909739926d-kube-api-access-ncr9f\") pod \"1983102f-b0ac-4e96-a32d-4f909739926d\" (UID: \"1983102f-b0ac-4e96-a32d-4f909739926d\") "
Sep 08 02:07:23.374316 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:07:23.374287    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1983102f-b0ac-4e96-a32d-4f909739926d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "1983102f-b0ac-4e96-a32d-4f909739926d" (UID: "1983102f-b0ac-4e96-a32d-4f909739926d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 02:07:23.374439 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:07:23.374410    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1983102f-b0ac-4e96-a32d-4f909739926d-kube-api-access-ncr9f" (OuterVolumeSpecName: "kube-api-access-ncr9f") pod "1983102f-b0ac-4e96-a32d-4f909739926d" (UID: "1983102f-b0ac-4e96-a32d-4f909739926d"). InnerVolumeSpecName "kube-api-access-ncr9f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 02:07:23.374475 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:07:23.374423    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1983102f-b0ac-4e96-a32d-4f909739926d-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "1983102f-b0ac-4e96-a32d-4f909739926d" (UID: "1983102f-b0ac-4e96-a32d-4f909739926d"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 02:07:23.473169 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:07:23.473092    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncr9f\" (UniqueName: \"kubernetes.io/projected/1983102f-b0ac-4e96-a32d-4f909739926d-kube-api-access-ncr9f\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:07:23.473169 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:07:23.473159    2547 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1983102f-b0ac-4e96-a32d-4f909739926d-metrics-certs\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:07:23.473169 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:07:23.473169    2547 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1983102f-b0ac-4e96-a32d-4f909739926d-webhook-cert\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:07:24.016822 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:07:24.016777    2547 generic.go:356] "Generic (PLEG): container finished" podID="1983102f-b0ac-4e96-a32d-4f909739926d" containerID="8ea44ac3a721cf61b1068b9900ff13df143871db85e92b87e8761e3fa3ed003a" exitCode=0
Sep 08 02:07:24.016822 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:07:24.016812    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6cb48d57d-6l8qp" event={"ID":"1983102f-b0ac-4e96-a32d-4f909739926d","Type":"ContainerDied","Data":"8ea44ac3a721cf61b1068b9900ff13df143871db85e92b87e8761e3fa3ed003a"}
Sep 08 02:07:24.017291 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:07:24.016828    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6cb48d57d-6l8qp"
Sep 08 02:07:24.017291 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:07:24.016866    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6cb48d57d-6l8qp" event={"ID":"1983102f-b0ac-4e96-a32d-4f909739926d","Type":"ContainerDied","Data":"1f3bf541658e5b2ea62baab6279642f19756d021a849e67d73a49c4544cafd9a"}
Sep 08 02:07:24.017291 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:07:24.016889    2547 scope.go:117] "RemoveContainer" containerID="8ea44ac3a721cf61b1068b9900ff13df143871db85e92b87e8761e3fa3ed003a"
Sep 08 02:07:24.025125 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:07:24.025106    2547 scope.go:117] "RemoveContainer" containerID="8ea44ac3a721cf61b1068b9900ff13df143871db85e92b87e8761e3fa3ed003a"
Sep 08 02:07:24.025459 ip-10-0-142-78 kubenswrapper[2547]: E0908 02:07:24.025440    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ea44ac3a721cf61b1068b9900ff13df143871db85e92b87e8761e3fa3ed003a\": container with ID starting with 8ea44ac3a721cf61b1068b9900ff13df143871db85e92b87e8761e3fa3ed003a not found: ID does not exist" containerID="8ea44ac3a721cf61b1068b9900ff13df143871db85e92b87e8761e3fa3ed003a"
Sep 08 02:07:24.025511 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:07:24.025468    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ea44ac3a721cf61b1068b9900ff13df143871db85e92b87e8761e3fa3ed003a"} err="failed to get container status \"8ea44ac3a721cf61b1068b9900ff13df143871db85e92b87e8761e3fa3ed003a\": rpc error: code = NotFound desc = could not find container \"8ea44ac3a721cf61b1068b9900ff13df143871db85e92b87e8761e3fa3ed003a\": container with ID starting with 8ea44ac3a721cf61b1068b9900ff13df143871db85e92b87e8761e3fa3ed003a not found: ID does not exist"
Sep 08 02:07:24.036973 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:07:24.036943    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6cb48d57d-6l8qp"]
Sep 08 02:07:24.040370 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:07:24.040347    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6cb48d57d-6l8qp"]
Sep 08 02:07:24.993195 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:07:24.993154    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1983102f-b0ac-4e96-a32d-4f909739926d" path="/var/lib/kubelet/pods/1983102f-b0ac-4e96-a32d-4f909739926d/volumes"
Sep 08 02:07:45.913573 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:07:45.913494    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-fmdgk"]
Sep 08 02:07:45.913956 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:07:45.913770    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1983102f-b0ac-4e96-a32d-4f909739926d" containerName="manager"
Sep 08 02:07:45.913956 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:07:45.913781    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="1983102f-b0ac-4e96-a32d-4f909739926d" containerName="manager"
Sep 08 02:07:45.913956 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:07:45.913825    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="1983102f-b0ac-4e96-a32d-4f909739926d" containerName="manager"
Sep 08 02:07:45.917762 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:07:45.917741    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-fmdgk"
Sep 08 02:07:45.919999 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:07:45.919968    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-9n5m5\""
Sep 08 02:07:45.920098 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:07:45.920038    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 08 02:07:45.926496 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:07:45.926472    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-fmdgk"]
Sep 08 02:07:45.955835 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:07:45.955797    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d582b5b-9b6f-4e26-b6be-bafecf8c1c99-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-fmdgk\" (UID: \"6d582b5b-9b6f-4e26-b6be-bafecf8c1c99\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-fmdgk"
Sep 08 02:07:45.955835 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:07:45.955832    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6xs4\" (UniqueName: \"kubernetes.io/projected/6d582b5b-9b6f-4e26-b6be-bafecf8c1c99-kube-api-access-w6xs4\") pod \"payload-pre-processing-59fbc47bf8-fmdgk\" (UID: \"6d582b5b-9b6f-4e26-b6be-bafecf8c1c99\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-fmdgk"
Sep 08 02:07:46.057206 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:07:46.057167    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d582b5b-9b6f-4e26-b6be-bafecf8c1c99-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-fmdgk\" (UID: \"6d582b5b-9b6f-4e26-b6be-bafecf8c1c99\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-fmdgk"
Sep 08 02:07:46.057391 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:07:46.057212    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6xs4\" (UniqueName: \"kubernetes.io/projected/6d582b5b-9b6f-4e26-b6be-bafecf8c1c99-kube-api-access-w6xs4\") pod \"payload-pre-processing-59fbc47bf8-fmdgk\" (UID: \"6d582b5b-9b6f-4e26-b6be-bafecf8c1c99\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-fmdgk"
Sep 08 02:07:46.057984 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:07:46.057962    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d582b5b-9b6f-4e26-b6be-bafecf8c1c99-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-fmdgk\" (UID: \"6d582b5b-9b6f-4e26-b6be-bafecf8c1c99\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-fmdgk"
Sep 08 02:07:46.067743 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:07:46.067713    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6xs4\" (UniqueName: \"kubernetes.io/projected/6d582b5b-9b6f-4e26-b6be-bafecf8c1c99-kube-api-access-w6xs4\") pod \"payload-pre-processing-59fbc47bf8-fmdgk\" (UID: \"6d582b5b-9b6f-4e26-b6be-bafecf8c1c99\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-fmdgk"
Sep 08 02:07:46.227646 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:07:46.227542    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-fmdgk"
Sep 08 02:07:46.493998 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:07:46.493971    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-fmdgk"]
Sep 08 02:07:46.496496 ip-10-0-142-78 kubenswrapper[2547]: W0908 02:07:46.496452    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d582b5b_9b6f_4e26_b6be_bafecf8c1c99.slice/crio-30cf7a23cb82247fb0c5020ff33d8dd06cfd6679603aaaa829fffcdfcbf166b3 WatchSource:0}: Error finding container 30cf7a23cb82247fb0c5020ff33d8dd06cfd6679603aaaa829fffcdfcbf166b3: Status 404 returned error can't find the container with id 30cf7a23cb82247fb0c5020ff33d8dd06cfd6679603aaaa829fffcdfcbf166b3
Sep 08 02:07:47.109239 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:07:47.109198    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-fmdgk" event={"ID":"6d582b5b-9b6f-4e26-b6be-bafecf8c1c99","Type":"ContainerStarted","Data":"30cf7a23cb82247fb0c5020ff33d8dd06cfd6679603aaaa829fffcdfcbf166b3"}
Sep 08 02:07:49.119646 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:07:49.119604    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-fmdgk" event={"ID":"6d582b5b-9b6f-4e26-b6be-bafecf8c1c99","Type":"ContainerStarted","Data":"1db5d5ed90650e865372be74950228a5545bdf55f1dfa263a856043bee78e675"}
Sep 08 02:07:49.120049 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:07:49.119677    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-fmdgk"
Sep 08 02:07:49.134960 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:07:49.134904    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-fmdgk" podStartSLOduration=1.682358673 podStartE2EDuration="4.134885163s" totalImagesPullingTime="2.45252649s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:07:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:07:49.1336412 +0000 UTC m=+540.657416280" watchObservedRunningTime="2026-09-08 02:07:49.134885163 +0000 UTC m=+540.658660243"
Sep 08 02:08:00.125112 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:08:00.125077    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-fmdgk"
Sep 08 02:08:06.115650 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:08:06.115613    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kct29"]
Sep 08 02:08:06.117812 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:08:06.117796    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kct29"
Sep 08 02:08:06.121225 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:08:06.121200    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Sep 08 02:08:06.121225 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:08:06.121210    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 08 02:08:06.121428 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:08:06.121254    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 08 02:08:06.122188 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:08:06.122169    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-x2jnv\""
Sep 08 02:08:06.129071 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:08:06.129046    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kct29"]
Sep 08 02:08:06.187786 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:08:06.187757    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/334aa197-482b-4f94-864b-bd2166f33f06-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kct29\" (UID: \"334aa197-482b-4f94-864b-bd2166f33f06\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kct29"
Sep 08 02:08:06.187786 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:08:06.187789    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95cr2\" (UniqueName: \"kubernetes.io/projected/334aa197-482b-4f94-864b-bd2166f33f06-kube-api-access-95cr2\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kct29\" (UID: \"334aa197-482b-4f94-864b-bd2166f33f06\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kct29"
Sep 08 02:08:06.187999 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:08:06.187808    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/334aa197-482b-4f94-864b-bd2166f33f06-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kct29\" (UID: \"334aa197-482b-4f94-864b-bd2166f33f06\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kct29"
Sep 08 02:08:06.187999 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:08:06.187826    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/334aa197-482b-4f94-864b-bd2166f33f06-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kct29\" (UID: \"334aa197-482b-4f94-864b-bd2166f33f06\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kct29"
Sep 08 02:08:06.187999 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:08:06.187903    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/334aa197-482b-4f94-864b-bd2166f33f06-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kct29\" (UID: \"334aa197-482b-4f94-864b-bd2166f33f06\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kct29"
Sep 08 02:08:06.187999 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:08:06.187934    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/334aa197-482b-4f94-864b-bd2166f33f06-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kct29\" (UID: \"334aa197-482b-4f94-864b-bd2166f33f06\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kct29"
Sep 08 02:08:06.288510 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:08:06.288466    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/334aa197-482b-4f94-864b-bd2166f33f06-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kct29\" (UID: \"334aa197-482b-4f94-864b-bd2166f33f06\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kct29"
Sep 08 02:08:06.288510 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:08:06.288518    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/334aa197-482b-4f94-864b-bd2166f33f06-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kct29\" (UID: \"334aa197-482b-4f94-864b-bd2166f33f06\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kct29"
Sep 08 02:08:06.288769 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:08:06.288558    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/334aa197-482b-4f94-864b-bd2166f33f06-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kct29\" (UID: \"334aa197-482b-4f94-864b-bd2166f33f06\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kct29"
Sep 08 02:08:06.288769 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:08:06.288576    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95cr2\" (UniqueName: \"kubernetes.io/projected/334aa197-482b-4f94-864b-bd2166f33f06-kube-api-access-95cr2\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kct29\" (UID: \"334aa197-482b-4f94-864b-bd2166f33f06\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kct29"
Sep 08 02:08:06.288769 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:08:06.288595    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/334aa197-482b-4f94-864b-bd2166f33f06-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kct29\" (UID: \"334aa197-482b-4f94-864b-bd2166f33f06\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kct29"
Sep 08 02:08:06.288769 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:08:06.288610    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/334aa197-482b-4f94-864b-bd2166f33f06-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kct29\" (UID: \"334aa197-482b-4f94-864b-bd2166f33f06\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kct29"
Sep 08 02:08:06.289000 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:08:06.288981    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/334aa197-482b-4f94-864b-bd2166f33f06-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kct29\" (UID: \"334aa197-482b-4f94-864b-bd2166f33f06\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kct29"
Sep 08 02:08:06.289074 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:08:06.289054    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/334aa197-482b-4f94-864b-bd2166f33f06-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kct29\" (UID: \"334aa197-482b-4f94-864b-bd2166f33f06\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kct29"
Sep 08 02:08:06.289126 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:08:06.289080    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/334aa197-482b-4f94-864b-bd2166f33f06-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kct29\" (UID: \"334aa197-482b-4f94-864b-bd2166f33f06\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kct29"
Sep 08 02:08:06.290730 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:08:06.290701    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/334aa197-482b-4f94-864b-bd2166f33f06-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kct29\" (UID: \"334aa197-482b-4f94-864b-bd2166f33f06\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kct29"
Sep 08 02:08:06.291001 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:08:06.290983    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/334aa197-482b-4f94-864b-bd2166f33f06-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kct29\" (UID: \"334aa197-482b-4f94-864b-bd2166f33f06\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kct29"
Sep 08 02:08:06.295790 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:08:06.295764    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-95cr2\" (UniqueName: \"kubernetes.io/projected/334aa197-482b-4f94-864b-bd2166f33f06-kube-api-access-95cr2\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kct29\" (UID: \"334aa197-482b-4f94-864b-bd2166f33f06\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kct29"
Sep 08 02:08:06.429094 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:08:06.429042    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kct29"
Sep 08 02:08:06.699441 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:08:06.699302    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kct29"]
Sep 08 02:08:06.702004 ip-10-0-142-78 kubenswrapper[2547]: W0908 02:08:06.701978    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod334aa197_482b_4f94_864b_bd2166f33f06.slice/crio-71ecddf3c5bc4eea1e578f1f1dbe3835116d9640cb097bc4cd5cfe1c34ab4f08 WatchSource:0}: Error finding container 71ecddf3c5bc4eea1e578f1f1dbe3835116d9640cb097bc4cd5cfe1c34ab4f08: Status 404 returned error can't find the container with id 71ecddf3c5bc4eea1e578f1f1dbe3835116d9640cb097bc4cd5cfe1c34ab4f08
Sep 08 02:08:07.193850 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:08:07.193812    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kct29" event={"ID":"334aa197-482b-4f94-864b-bd2166f33f06","Type":"ContainerStarted","Data":"71ecddf3c5bc4eea1e578f1f1dbe3835116d9640cb097bc4cd5cfe1c34ab4f08"}
Sep 08 02:08:08.199046 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:08:08.199011    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kct29" event={"ID":"334aa197-482b-4f94-864b-bd2166f33f06","Type":"ContainerStarted","Data":"3bd3b5d2a0a40a44a41fe15042c7d30757de440155d0b27108356afd96a51475"}
Sep 08 02:08:08.215416 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:08:08.215363    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kct29" podStartSLOduration=1.236126654 podStartE2EDuration="2.215346743s" totalImagesPullingTime="979.220089ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:08:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:08:08.214217461 +0000 UTC m=+559.737992541" watchObservedRunningTime="2026-09-08 02:08:08.215346743 +0000 UTC m=+559.739121823"
Sep 08 02:08:16.429730 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:08:16.429671    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kct29"
Sep 08 02:08:16.430203 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:08:16.429759    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kct29"
Sep 08 02:08:16.442248 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:08:16.442220    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kct29"
Sep 08 02:08:17.245182 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:08:17.245152    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kct29"
Sep 08 02:09:05.417980 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:09:05.417944    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-shnmg"]
Sep 08 02:09:05.421473 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:09:05.421456    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-shnmg"
Sep 08 02:09:05.423668 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:09:05.423645    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Sep 08 02:09:05.429060 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:09:05.429031    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-shnmg"]
Sep 08 02:09:05.566214 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:09:05.566170    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r86rn\" (UniqueName: \"kubernetes.io/projected/e0ea35bd-ed0a-4c4b-9d04-6eab00f44427-kube-api-access-r86rn\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-shnmg\" (UID: \"e0ea35bd-ed0a-4c4b-9d04-6eab00f44427\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-shnmg"
Sep 08 02:09:05.566214 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:09:05.566219    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0ea35bd-ed0a-4c4b-9d04-6eab00f44427-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-shnmg\" (UID: \"e0ea35bd-ed0a-4c4b-9d04-6eab00f44427\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-shnmg"
Sep 08 02:09:05.566422 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:09:05.566244    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0ea35bd-ed0a-4c4b-9d04-6eab00f44427-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-shnmg\" (UID: \"e0ea35bd-ed0a-4c4b-9d04-6eab00f44427\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-shnmg"
Sep 08 02:09:05.566422 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:09:05.566270    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0ea35bd-ed0a-4c4b-9d04-6eab00f44427-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-shnmg\" (UID: \"e0ea35bd-ed0a-4c4b-9d04-6eab00f44427\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-shnmg"
Sep 08 02:09:05.566422 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:09:05.566287    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0ea35bd-ed0a-4c4b-9d04-6eab00f44427-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-shnmg\" (UID: \"e0ea35bd-ed0a-4c4b-9d04-6eab00f44427\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-shnmg"
Sep 08 02:09:05.566422 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:09:05.566378    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0ea35bd-ed0a-4c4b-9d04-6eab00f44427-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-shnmg\" (UID: \"e0ea35bd-ed0a-4c4b-9d04-6eab00f44427\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-shnmg"
Sep 08 02:09:05.667639 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:09:05.667605    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r86rn\" (UniqueName: \"kubernetes.io/projected/e0ea35bd-ed0a-4c4b-9d04-6eab00f44427-kube-api-access-r86rn\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-shnmg\" (UID: \"e0ea35bd-ed0a-4c4b-9d04-6eab00f44427\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-shnmg"
Sep 08 02:09:05.667639 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:09:05.667643    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0ea35bd-ed0a-4c4b-9d04-6eab00f44427-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-shnmg\" (UID: \"e0ea35bd-ed0a-4c4b-9d04-6eab00f44427\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-shnmg"
Sep 08 02:09:05.667809 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:09:05.667668    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0ea35bd-ed0a-4c4b-9d04-6eab00f44427-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-shnmg\" (UID: \"e0ea35bd-ed0a-4c4b-9d04-6eab00f44427\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-shnmg"
Sep 08 02:09:05.667809 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:09:05.667703    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0ea35bd-ed0a-4c4b-9d04-6eab00f44427-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-shnmg\" (UID: \"e0ea35bd-ed0a-4c4b-9d04-6eab00f44427\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-shnmg"
Sep 08 02:09:05.667809 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:09:05.667729    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0ea35bd-ed0a-4c4b-9d04-6eab00f44427-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-shnmg\" (UID: \"e0ea35bd-ed0a-4c4b-9d04-6eab00f44427\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-shnmg"
Sep 08 02:09:05.667924 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:09:05.667895    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0ea35bd-ed0a-4c4b-9d04-6eab00f44427-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-shnmg\" (UID: \"e0ea35bd-ed0a-4c4b-9d04-6eab00f44427\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-shnmg"
Sep 08 02:09:05.668181 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:09:05.668113    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0ea35bd-ed0a-4c4b-9d04-6eab00f44427-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-shnmg\" (UID: \"e0ea35bd-ed0a-4c4b-9d04-6eab00f44427\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-shnmg"
Sep 08 02:09:05.668181 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:09:05.668167    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0ea35bd-ed0a-4c4b-9d04-6eab00f44427-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-shnmg\" (UID: \"e0ea35bd-ed0a-4c4b-9d04-6eab00f44427\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-shnmg"
Sep 08 02:09:05.668306 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:09:05.668239    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0ea35bd-ed0a-4c4b-9d04-6eab00f44427-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-shnmg\" (UID: \"e0ea35bd-ed0a-4c4b-9d04-6eab00f44427\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-shnmg"
Sep 08 02:09:05.669860 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:09:05.669839    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0ea35bd-ed0a-4c4b-9d04-6eab00f44427-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-shnmg\" (UID: \"e0ea35bd-ed0a-4c4b-9d04-6eab00f44427\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-shnmg"
Sep 08 02:09:05.670214 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:09:05.670197    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0ea35bd-ed0a-4c4b-9d04-6eab00f44427-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-shnmg\" (UID: \"e0ea35bd-ed0a-4c4b-9d04-6eab00f44427\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-shnmg"
Sep 08 02:09:05.674753 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:09:05.674732    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r86rn\" (UniqueName: \"kubernetes.io/projected/e0ea35bd-ed0a-4c4b-9d04-6eab00f44427-kube-api-access-r86rn\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-shnmg\" (UID: \"e0ea35bd-ed0a-4c4b-9d04-6eab00f44427\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-shnmg"
Sep 08 02:09:05.731478 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:09:05.731437    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-shnmg"
Sep 08 02:09:06.001023 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:09:06.000943    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-shnmg"]
Sep 08 02:09:06.003947 ip-10-0-142-78 kubenswrapper[2547]: W0908 02:09:06.003923    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0ea35bd_ed0a_4c4b_9d04_6eab00f44427.slice/crio-18cc2436167f89dc690ee10ca590bd150a68288425c1c347fd3a91ab604da7bc WatchSource:0}: Error finding container 18cc2436167f89dc690ee10ca590bd150a68288425c1c347fd3a91ab604da7bc: Status 404 returned error can't find the container with id 18cc2436167f89dc690ee10ca590bd150a68288425c1c347fd3a91ab604da7bc
Sep 08 02:09:06.005647 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:09:06.005630    2547 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 02:09:06.422941 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:09:06.422904    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-shnmg" event={"ID":"e0ea35bd-ed0a-4c4b-9d04-6eab00f44427","Type":"ContainerStarted","Data":"42cd521336e70da23b77a5a3a7a0e39390086b15eb049fd83b178d869615769d"}
Sep 08 02:09:06.423361 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:09:06.422965    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-shnmg" event={"ID":"e0ea35bd-ed0a-4c4b-9d04-6eab00f44427","Type":"ContainerStarted","Data":"18cc2436167f89dc690ee10ca590bd150a68288425c1c347fd3a91ab604da7bc"}
Sep 08 02:09:06.439767 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:09:06.439715    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-shnmg" podStartSLOduration=1.259982616 podStartE2EDuration="1.439699366s" totalImagesPullingTime="179.71675ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:09:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:09:06.43822668 +0000 UTC m=+617.962001763" watchObservedRunningTime="2026-09-08 02:09:06.439699366 +0000 UTC m=+617.963474446"
Sep 08 02:09:15.731810 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:09:15.731720    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-shnmg"
Sep 08 02:09:15.731810 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:09:15.731788    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-shnmg"
Sep 08 02:09:15.744499 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:09:15.744468    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-shnmg"
Sep 08 02:09:16.472875 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:09:16.472839    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-shnmg"
Sep 08 02:09:20.113737 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:09:20.113696    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-d8vv5"]
Sep 08 02:09:20.118063 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:09:20.118041    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d8vv5"
Sep 08 02:09:20.120694 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:09:20.120669    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Sep 08 02:09:20.127639 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:09:20.127610    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-d8vv5"]
Sep 08 02:09:20.179375 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:09:20.179339    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/289f142c-f6f5-4b56-990d-e53f69dd5e9e-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d8vv5\" (UID: \"289f142c-f6f5-4b56-990d-e53f69dd5e9e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d8vv5"
Sep 08 02:09:20.179612 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:09:20.179386    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/289f142c-f6f5-4b56-990d-e53f69dd5e9e-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d8vv5\" (UID: \"289f142c-f6f5-4b56-990d-e53f69dd5e9e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d8vv5"
Sep 08 02:09:20.179612 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:09:20.179407    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/289f142c-f6f5-4b56-990d-e53f69dd5e9e-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d8vv5\" (UID: \"289f142c-f6f5-4b56-990d-e53f69dd5e9e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d8vv5"
Sep 08 02:09:20.179612 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:09:20.179466    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zr4x9\" (UniqueName: \"kubernetes.io/projected/289f142c-f6f5-4b56-990d-e53f69dd5e9e-kube-api-access-zr4x9\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d8vv5\" (UID: \"289f142c-f6f5-4b56-990d-e53f69dd5e9e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d8vv5"
Sep 08 02:09:20.179612 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:09:20.179499    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/289f142c-f6f5-4b56-990d-e53f69dd5e9e-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d8vv5\" (UID: \"289f142c-f6f5-4b56-990d-e53f69dd5e9e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d8vv5"
Sep 08 02:09:20.179612 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:09:20.179595    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/289f142c-f6f5-4b56-990d-e53f69dd5e9e-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d8vv5\" (UID: \"289f142c-f6f5-4b56-990d-e53f69dd5e9e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d8vv5"
Sep 08 02:09:20.280110 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:09:20.280069    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/289f142c-f6f5-4b56-990d-e53f69dd5e9e-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d8vv5\" (UID: \"289f142c-f6f5-4b56-990d-e53f69dd5e9e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d8vv5"
Sep 08 02:09:20.280110 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:09:20.280115    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/289f142c-f6f5-4b56-990d-e53f69dd5e9e-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d8vv5\" (UID: \"289f142c-f6f5-4b56-990d-e53f69dd5e9e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d8vv5"
Sep 08 02:09:20.280375 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:09:20.280177    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/289f142c-f6f5-4b56-990d-e53f69dd5e9e-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d8vv5\" (UID: \"289f142c-f6f5-4b56-990d-e53f69dd5e9e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d8vv5"
Sep 08 02:09:20.280375 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:09:20.280196    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/289f142c-f6f5-4b56-990d-e53f69dd5e9e-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d8vv5\" (UID: \"289f142c-f6f5-4b56-990d-e53f69dd5e9e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d8vv5"
Sep 08 02:09:20.280375 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:09:20.280228    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zr4x9\" (UniqueName: \"kubernetes.io/projected/289f142c-f6f5-4b56-990d-e53f69dd5e9e-kube-api-access-zr4x9\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d8vv5\" (UID: \"289f142c-f6f5-4b56-990d-e53f69dd5e9e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d8vv5"
Sep 08 02:09:20.280375 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:09:20.280257    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/289f142c-f6f5-4b56-990d-e53f69dd5e9e-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d8vv5\" (UID: \"289f142c-f6f5-4b56-990d-e53f69dd5e9e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d8vv5"
Sep 08 02:09:20.280581 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:09:20.280520    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/289f142c-f6f5-4b56-990d-e53f69dd5e9e-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d8vv5\" (UID: \"289f142c-f6f5-4b56-990d-e53f69dd5e9e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d8vv5"
Sep 08 02:09:20.280581 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:09:20.280561    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/289f142c-f6f5-4b56-990d-e53f69dd5e9e-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d8vv5\" (UID: \"289f142c-f6f5-4b56-990d-e53f69dd5e9e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d8vv5"
Sep 08 02:09:20.280690 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:09:20.280669    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/289f142c-f6f5-4b56-990d-e53f69dd5e9e-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d8vv5\" (UID: \"289f142c-f6f5-4b56-990d-e53f69dd5e9e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d8vv5"
Sep 08 02:09:20.282461 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:09:20.282442    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/289f142c-f6f5-4b56-990d-e53f69dd5e9e-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d8vv5\" (UID: \"289f142c-f6f5-4b56-990d-e53f69dd5e9e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d8vv5"
Sep 08 02:09:20.282756 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:09:20.282735    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/289f142c-f6f5-4b56-990d-e53f69dd5e9e-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d8vv5\" (UID: \"289f142c-f6f5-4b56-990d-e53f69dd5e9e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d8vv5"
Sep 08 02:09:20.288488 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:09:20.288464    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zr4x9\" (UniqueName: \"kubernetes.io/projected/289f142c-f6f5-4b56-990d-e53f69dd5e9e-kube-api-access-zr4x9\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d8vv5\" (UID: \"289f142c-f6f5-4b56-990d-e53f69dd5e9e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d8vv5"
Sep 08 02:09:20.429204 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:09:20.429159    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d8vv5"
Sep 08 02:09:20.701772 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:09:20.701694    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-d8vv5"]
Sep 08 02:09:20.704788 ip-10-0-142-78 kubenswrapper[2547]: W0908 02:09:20.704752    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod289f142c_f6f5_4b56_990d_e53f69dd5e9e.slice/crio-112f663959f13de48184e4aebcdde073e035d0afd4088b1adbbcff843d2a5ee0 WatchSource:0}: Error finding container 112f663959f13de48184e4aebcdde073e035d0afd4088b1adbbcff843d2a5ee0: Status 404 returned error can't find the container with id 112f663959f13de48184e4aebcdde073e035d0afd4088b1adbbcff843d2a5ee0
Sep 08 02:09:21.484799 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:09:21.484759    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d8vv5" event={"ID":"289f142c-f6f5-4b56-990d-e53f69dd5e9e","Type":"ContainerStarted","Data":"087ab8d529945343fce4f6de94b8eab134fc13ce4f1b16763b00b25d0b62b92b"}
Sep 08 02:09:21.485318 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:09:21.484817    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d8vv5" event={"ID":"289f142c-f6f5-4b56-990d-e53f69dd5e9e","Type":"ContainerStarted","Data":"112f663959f13de48184e4aebcdde073e035d0afd4088b1adbbcff843d2a5ee0"}
Sep 08 02:09:21.501520 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:09:21.501462    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d8vv5" podStartSLOduration=1.331464672 podStartE2EDuration="1.501444633s" totalImagesPullingTime="169.979961ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:09:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:09:21.499592229 +0000 UTC m=+633.023367319" watchObservedRunningTime="2026-09-08 02:09:21.501444633 +0000 UTC m=+633.025219709"
Sep 08 02:09:30.429346 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:09:30.429305    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d8vv5"
Sep 08 02:09:30.429897 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:09:30.429378    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d8vv5"
Sep 08 02:09:30.442327 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:09:30.442298    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d8vv5"
Sep 08 02:09:30.531275 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:09:30.531242    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d8vv5"
Sep 08 02:09:58.015894 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:09:58.015852    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-bwvxn"]
Sep 08 02:09:58.019442 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:09:58.019418    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bwvxn"
Sep 08 02:09:58.021721 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:09:58.021697    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 08 02:09:58.027121 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:09:58.027099    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-bwvxn"]
Sep 08 02:09:58.090893 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:09:58.090850    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e0ddfff-324b-445d-9b2e-92448da092ec-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bwvxn\" (UID: \"0e0ddfff-324b-445d-9b2e-92448da092ec\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bwvxn"
Sep 08 02:09:58.090893 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:09:58.090889    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e0ddfff-324b-445d-9b2e-92448da092ec-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bwvxn\" (UID: \"0e0ddfff-324b-445d-9b2e-92448da092ec\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bwvxn"
Sep 08 02:09:58.091165 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:09:58.090909    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62475\" (UniqueName: \"kubernetes.io/projected/0e0ddfff-324b-445d-9b2e-92448da092ec-kube-api-access-62475\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bwvxn\" (UID: \"0e0ddfff-324b-445d-9b2e-92448da092ec\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bwvxn"
Sep 08 02:09:58.091165 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:09:58.091011    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e0ddfff-324b-445d-9b2e-92448da092ec-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bwvxn\" (UID: \"0e0ddfff-324b-445d-9b2e-92448da092ec\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bwvxn"
Sep 08 02:09:58.091165 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:09:58.091067    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e0ddfff-324b-445d-9b2e-92448da092ec-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bwvxn\" (UID: \"0e0ddfff-324b-445d-9b2e-92448da092ec\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bwvxn"
Sep 08 02:09:58.091165 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:09:58.091126    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e0ddfff-324b-445d-9b2e-92448da092ec-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bwvxn\" (UID: \"0e0ddfff-324b-445d-9b2e-92448da092ec\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bwvxn"
Sep 08 02:09:58.192467 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:09:58.192425    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e0ddfff-324b-445d-9b2e-92448da092ec-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bwvxn\" (UID: \"0e0ddfff-324b-445d-9b2e-92448da092ec\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bwvxn"
Sep 08 02:09:58.192646 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:09:58.192482    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e0ddfff-324b-445d-9b2e-92448da092ec-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bwvxn\" (UID: \"0e0ddfff-324b-445d-9b2e-92448da092ec\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bwvxn"
Sep 08 02:09:58.192646 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:09:58.192530    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e0ddfff-324b-445d-9b2e-92448da092ec-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bwvxn\" (UID: \"0e0ddfff-324b-445d-9b2e-92448da092ec\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bwvxn"
Sep 08 02:09:58.192646 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:09:58.192585    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e0ddfff-324b-445d-9b2e-92448da092ec-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bwvxn\" (UID: \"0e0ddfff-324b-445d-9b2e-92448da092ec\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bwvxn"
Sep 08 02:09:58.192646 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:09:58.192608    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e0ddfff-324b-445d-9b2e-92448da092ec-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bwvxn\" (UID: \"0e0ddfff-324b-445d-9b2e-92448da092ec\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bwvxn"
Sep 08 02:09:58.192646 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:09:58.192635    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62475\" (UniqueName: \"kubernetes.io/projected/0e0ddfff-324b-445d-9b2e-92448da092ec-kube-api-access-62475\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bwvxn\" (UID: \"0e0ddfff-324b-445d-9b2e-92448da092ec\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bwvxn"
Sep 08 02:09:58.192922 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:09:58.192900    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e0ddfff-324b-445d-9b2e-92448da092ec-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bwvxn\" (UID: \"0e0ddfff-324b-445d-9b2e-92448da092ec\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bwvxn"
Sep 08 02:09:58.193006 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:09:58.192976    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e0ddfff-324b-445d-9b2e-92448da092ec-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bwvxn\" (UID: \"0e0ddfff-324b-445d-9b2e-92448da092ec\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bwvxn"
Sep 08 02:09:58.193070 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:09:58.193051    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e0ddfff-324b-445d-9b2e-92448da092ec-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bwvxn\" (UID: \"0e0ddfff-324b-445d-9b2e-92448da092ec\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bwvxn"
Sep 08 02:09:58.194874 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:09:58.194844    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e0ddfff-324b-445d-9b2e-92448da092ec-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bwvxn\" (UID: \"0e0ddfff-324b-445d-9b2e-92448da092ec\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bwvxn"
Sep 08 02:09:58.195058 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:09:58.195036    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e0ddfff-324b-445d-9b2e-92448da092ec-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bwvxn\" (UID: \"0e0ddfff-324b-445d-9b2e-92448da092ec\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bwvxn"
Sep 08 02:09:58.201910 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:09:58.201888    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62475\" (UniqueName: \"kubernetes.io/projected/0e0ddfff-324b-445d-9b2e-92448da092ec-kube-api-access-62475\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bwvxn\" (UID: \"0e0ddfff-324b-445d-9b2e-92448da092ec\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bwvxn"
Sep 08 02:09:58.329260 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:09:58.329170    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bwvxn"
Sep 08 02:09:58.599080 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:09:58.599041    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-bwvxn"]
Sep 08 02:09:58.601702 ip-10-0-142-78 kubenswrapper[2547]: W0908 02:09:58.601678    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0e0ddfff_324b_445d_9b2e_92448da092ec.slice/crio-5b205a99a90e6935288d2e6c805d8df14a6617c2771ad00ab15c638d0a3b9a73 WatchSource:0}: Error finding container 5b205a99a90e6935288d2e6c805d8df14a6617c2771ad00ab15c638d0a3b9a73: Status 404 returned error can't find the container with id 5b205a99a90e6935288d2e6c805d8df14a6617c2771ad00ab15c638d0a3b9a73
Sep 08 02:09:58.638051 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:09:58.638018    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bwvxn" event={"ID":"0e0ddfff-324b-445d-9b2e-92448da092ec","Type":"ContainerStarted","Data":"5b205a99a90e6935288d2e6c805d8df14a6617c2771ad00ab15c638d0a3b9a73"}
Sep 08 02:09:59.643661 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:09:59.643619    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bwvxn" event={"ID":"0e0ddfff-324b-445d-9b2e-92448da092ec","Type":"ContainerStarted","Data":"0eddda3190f9b710587ffefff6f15e462e6ddf380e5c38970e2785adf4832aca"}
Sep 08 02:09:59.659000 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:09:59.658947    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bwvxn" podStartSLOduration=2.484282411 podStartE2EDuration="2.658931373s" totalImagesPullingTime="174.648962ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:09:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:09:59.658692574 +0000 UTC m=+671.182467657" watchObservedRunningTime="2026-09-08 02:09:59.658931373 +0000 UTC m=+671.182706453"
Sep 08 02:10:08.329541 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:10:08.329503    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bwvxn"
Sep 08 02:10:08.329541 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:10:08.329548    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bwvxn"
Sep 08 02:10:08.341732 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:10:08.341699    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bwvxn"
Sep 08 02:10:08.694920 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:10:08.694885    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bwvxn"
Sep 08 02:11:22.630242 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:22.630161    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5d27de-58656496c5-qzvbz"]
Sep 08 02:11:22.633152 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:22.633123    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-5d27de-58656496c5-qzvbz"
Sep 08 02:11:22.635368 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:22.635349    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-5d27de-dockercfg-qk5jh\""
Sep 08 02:11:22.635475 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:22.635440    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-5d27de\""
Sep 08 02:11:22.646984 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:22.646958    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5d27de-58656496c5-qzvbz"]
Sep 08 02:11:22.675603 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:22.675571    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5d27de-78684b5958-vtg65"]
Sep 08 02:11:22.678497 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:22.678482    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5d27de-78684b5958-vtg65"
Sep 08 02:11:22.692225 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:22.692198    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5d27de-78684b5958-vtg65"]
Sep 08 02:11:22.693888 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:22.693868    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzftx\" (UniqueName: \"kubernetes.io/projected/d6d26402-535b-4f5d-8cff-91b3bc126699-kube-api-access-wzftx\") pod \"payload-pre-processing-e2e-inf-a-5d27de-78684b5958-vtg65\" (UID: \"d6d26402-535b-4f5d-8cff-91b3bc126699\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5d27de-78684b5958-vtg65"
Sep 08 02:11:22.693993 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:22.693974    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6d26402-535b-4f5d-8cff-91b3bc126699-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-5d27de-78684b5958-vtg65\" (UID: \"d6d26402-535b-4f5d-8cff-91b3bc126699\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5d27de-78684b5958-vtg65"
Sep 08 02:11:22.795202 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:22.795158    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6d26402-535b-4f5d-8cff-91b3bc126699-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-5d27de-78684b5958-vtg65\" (UID: \"d6d26402-535b-4f5d-8cff-91b3bc126699\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5d27de-78684b5958-vtg65"
Sep 08 02:11:22.795395 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:22.795216    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzftx\" (UniqueName: \"kubernetes.io/projected/d6d26402-535b-4f5d-8cff-91b3bc126699-kube-api-access-wzftx\") pod \"payload-pre-processing-e2e-inf-a-5d27de-78684b5958-vtg65\" (UID: \"d6d26402-535b-4f5d-8cff-91b3bc126699\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5d27de-78684b5958-vtg65"
Sep 08 02:11:22.795395 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:22.795278    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1dabb5b-1a34-4dc3-a7c5-eaa6db21ce32-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-5d27de-58656496c5-qzvbz\" (UID: \"d1dabb5b-1a34-4dc3-a7c5-eaa6db21ce32\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5d27de-58656496c5-qzvbz"
Sep 08 02:11:22.795395 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:22.795328    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hdk2\" (UniqueName: \"kubernetes.io/projected/d1dabb5b-1a34-4dc3-a7c5-eaa6db21ce32-kube-api-access-2hdk2\") pod \"payload-processing-e2e-inf-a-5d27de-58656496c5-qzvbz\" (UID: \"d1dabb5b-1a34-4dc3-a7c5-eaa6db21ce32\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5d27de-58656496c5-qzvbz"
Sep 08 02:11:22.795845 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:22.795823    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6d26402-535b-4f5d-8cff-91b3bc126699-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-5d27de-78684b5958-vtg65\" (UID: \"d6d26402-535b-4f5d-8cff-91b3bc126699\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5d27de-78684b5958-vtg65"
Sep 08 02:11:22.804493 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:22.804464    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzftx\" (UniqueName: \"kubernetes.io/projected/d6d26402-535b-4f5d-8cff-91b3bc126699-kube-api-access-wzftx\") pod \"payload-pre-processing-e2e-inf-a-5d27de-78684b5958-vtg65\" (UID: \"d6d26402-535b-4f5d-8cff-91b3bc126699\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5d27de-78684b5958-vtg65"
Sep 08 02:11:22.895742 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:22.895703    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1dabb5b-1a34-4dc3-a7c5-eaa6db21ce32-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-5d27de-58656496c5-qzvbz\" (UID: \"d1dabb5b-1a34-4dc3-a7c5-eaa6db21ce32\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5d27de-58656496c5-qzvbz"
Sep 08 02:11:22.895902 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:22.895754    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hdk2\" (UniqueName: \"kubernetes.io/projected/d1dabb5b-1a34-4dc3-a7c5-eaa6db21ce32-kube-api-access-2hdk2\") pod \"payload-processing-e2e-inf-a-5d27de-58656496c5-qzvbz\" (UID: \"d1dabb5b-1a34-4dc3-a7c5-eaa6db21ce32\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5d27de-58656496c5-qzvbz"
Sep 08 02:11:22.896548 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:22.896527    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1dabb5b-1a34-4dc3-a7c5-eaa6db21ce32-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-5d27de-58656496c5-qzvbz\" (UID: \"d1dabb5b-1a34-4dc3-a7c5-eaa6db21ce32\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5d27de-58656496c5-qzvbz"
Sep 08 02:11:22.903177 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:22.903155    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hdk2\" (UniqueName: \"kubernetes.io/projected/d1dabb5b-1a34-4dc3-a7c5-eaa6db21ce32-kube-api-access-2hdk2\") pod \"payload-processing-e2e-inf-a-5d27de-58656496c5-qzvbz\" (UID: \"d1dabb5b-1a34-4dc3-a7c5-eaa6db21ce32\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5d27de-58656496c5-qzvbz"
Sep 08 02:11:22.942797 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:22.942762    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-5d27de-58656496c5-qzvbz"
Sep 08 02:11:22.988833 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:22.988757    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5d27de-78684b5958-vtg65"
Sep 08 02:11:23.214748 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:23.214644    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5d27de-58656496c5-qzvbz"]
Sep 08 02:11:23.218737 ip-10-0-142-78 kubenswrapper[2547]: W0908 02:11:23.218702    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1dabb5b_1a34_4dc3_a7c5_eaa6db21ce32.slice/crio-2f6ba7333fb07053fad03526d03555bbb96d6fb181fe6e1b1bd496cf256ddefb WatchSource:0}: Error finding container 2f6ba7333fb07053fad03526d03555bbb96d6fb181fe6e1b1bd496cf256ddefb: Status 404 returned error can't find the container with id 2f6ba7333fb07053fad03526d03555bbb96d6fb181fe6e1b1bd496cf256ddefb
Sep 08 02:11:23.264666 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:23.264640    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5d27de-78684b5958-vtg65"]
Sep 08 02:11:23.268426 ip-10-0-142-78 kubenswrapper[2547]: W0908 02:11:23.268400    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6d26402_535b_4f5d_8cff_91b3bc126699.slice/crio-500b2364064f581d4089d8d9a505fa595afe1846d2072391c3d9f940f0511d29 WatchSource:0}: Error finding container 500b2364064f581d4089d8d9a505fa595afe1846d2072391c3d9f940f0511d29: Status 404 returned error can't find the container with id 500b2364064f581d4089d8d9a505fa595afe1846d2072391c3d9f940f0511d29
Sep 08 02:11:23.993274 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:23.993237    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-5d27de-58656496c5-qzvbz" event={"ID":"d1dabb5b-1a34-4dc3-a7c5-eaa6db21ce32","Type":"ContainerStarted","Data":"e5b937ae156ff34b0b4492beddcb479329dcbee74fac44939a4131d399884ff7"}
Sep 08 02:11:23.993737 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:23.993297    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-5d27de-58656496c5-qzvbz" event={"ID":"d1dabb5b-1a34-4dc3-a7c5-eaa6db21ce32","Type":"ContainerStarted","Data":"2f6ba7333fb07053fad03526d03555bbb96d6fb181fe6e1b1bd496cf256ddefb"}
Sep 08 02:11:23.993737 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:23.993376    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-5d27de-58656496c5-qzvbz"
Sep 08 02:11:23.994953 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:23.994921    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5d27de-78684b5958-vtg65" event={"ID":"d6d26402-535b-4f5d-8cff-91b3bc126699","Type":"ContainerStarted","Data":"43a776e44312ae6e2bfc39107d915a25a6ca73b700b22e89a6b6648a1afa0605"}
Sep 08 02:11:23.995075 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:23.994957    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5d27de-78684b5958-vtg65" event={"ID":"d6d26402-535b-4f5d-8cff-91b3bc126699","Type":"ContainerStarted","Data":"500b2364064f581d4089d8d9a505fa595afe1846d2072391c3d9f940f0511d29"}
Sep 08 02:11:23.995075 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:23.994975    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5d27de-78684b5958-vtg65"
Sep 08 02:11:24.011162 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:24.011091    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-5d27de-58656496c5-qzvbz" podStartSLOduration=2.011078007 podStartE2EDuration="2.011078007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:11:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:11:24.009795102 +0000 UTC m=+755.533570182" watchObservedRunningTime="2026-09-08 02:11:24.011078007 +0000 UTC m=+755.534853087"
Sep 08 02:11:24.024829 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:24.024776    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5d27de-78684b5958-vtg65" podStartSLOduration=2.02476319 podStartE2EDuration="2.02476319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:11:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:11:24.023400734 +0000 UTC m=+755.547175814" watchObservedRunningTime="2026-09-08 02:11:24.02476319 +0000 UTC m=+755.548538270"
Sep 08 02:11:30.075605 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:30.075568    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-213d54df-5f57b864cd-9jfw7"]
Sep 08 02:11:30.079046 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:30.079023    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-213d54df-5f57b864cd-9jfw7"
Sep 08 02:11:30.081826 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:30.081782    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-213d54df\""
Sep 08 02:11:30.081969 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:30.081930    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-213d54df-dockercfg-n9sj7\""
Sep 08 02:11:30.090840 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:30.090813    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-213d54df-5f57b864cd-9jfw7"]
Sep 08 02:11:30.149033 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:30.148993    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8a225a9-0b57-4042-8033-166d2291558f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-213d54df-5f57b864cd-9jfw7\" (UID: \"e8a225a9-0b57-4042-8033-166d2291558f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-213d54df-5f57b864cd-9jfw7"
Sep 08 02:11:30.149033 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:30.149028    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fh62\" (UniqueName: \"kubernetes.io/projected/e8a225a9-0b57-4042-8033-166d2291558f-kube-api-access-7fh62\") pod \"payload-pre-processing-e2e-ait-213d54df-5f57b864cd-9jfw7\" (UID: \"e8a225a9-0b57-4042-8033-166d2291558f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-213d54df-5f57b864cd-9jfw7"
Sep 08 02:11:30.250169 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:30.250109    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8a225a9-0b57-4042-8033-166d2291558f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-213d54df-5f57b864cd-9jfw7\" (UID: \"e8a225a9-0b57-4042-8033-166d2291558f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-213d54df-5f57b864cd-9jfw7"
Sep 08 02:11:30.250169 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:30.250169    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fh62\" (UniqueName: \"kubernetes.io/projected/e8a225a9-0b57-4042-8033-166d2291558f-kube-api-access-7fh62\") pod \"payload-pre-processing-e2e-ait-213d54df-5f57b864cd-9jfw7\" (UID: \"e8a225a9-0b57-4042-8033-166d2291558f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-213d54df-5f57b864cd-9jfw7"
Sep 08 02:11:30.250893 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:30.250869    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8a225a9-0b57-4042-8033-166d2291558f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-213d54df-5f57b864cd-9jfw7\" (UID: \"e8a225a9-0b57-4042-8033-166d2291558f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-213d54df-5f57b864cd-9jfw7"
Sep 08 02:11:30.257334 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:30.257309    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fh62\" (UniqueName: \"kubernetes.io/projected/e8a225a9-0b57-4042-8033-166d2291558f-kube-api-access-7fh62\") pod \"payload-pre-processing-e2e-ait-213d54df-5f57b864cd-9jfw7\" (UID: \"e8a225a9-0b57-4042-8033-166d2291558f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-213d54df-5f57b864cd-9jfw7"
Sep 08 02:11:30.389712 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:30.389676    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-213d54df-5f57b864cd-9jfw7"
Sep 08 02:11:30.662113 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:30.662085    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-213d54df-5f57b864cd-9jfw7"]
Sep 08 02:11:30.664784 ip-10-0-142-78 kubenswrapper[2547]: W0908 02:11:30.664754    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8a225a9_0b57_4042_8033_166d2291558f.slice/crio-b479d2138dc13e1cc76a9f256bc1b773903016f2371d5f119a41b0d435f7e54c WatchSource:0}: Error finding container b479d2138dc13e1cc76a9f256bc1b773903016f2371d5f119a41b0d435f7e54c: Status 404 returned error can't find the container with id b479d2138dc13e1cc76a9f256bc1b773903016f2371d5f119a41b0d435f7e54c
Sep 08 02:11:31.023944 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:31.023851    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-213d54df-5f57b864cd-9jfw7" event={"ID":"e8a225a9-0b57-4042-8033-166d2291558f","Type":"ContainerStarted","Data":"4417bed515fb87575514f93afeeb9cc7591204a3f75b74dad9974629f1e57077"}
Sep 08 02:11:31.023944 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:31.023898    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-213d54df-5f57b864cd-9jfw7" event={"ID":"e8a225a9-0b57-4042-8033-166d2291558f","Type":"ContainerStarted","Data":"b479d2138dc13e1cc76a9f256bc1b773903016f2371d5f119a41b0d435f7e54c"}
Sep 08 02:11:31.024182 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:31.023969    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-213d54df-5f57b864cd-9jfw7"
Sep 08 02:11:31.042158 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:31.042082    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-213d54df-5f57b864cd-9jfw7" podStartSLOduration=1.042067087 podStartE2EDuration="1.042067087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:11:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:11:31.040378653 +0000 UTC m=+762.564153733" watchObservedRunningTime="2026-09-08 02:11:31.042067087 +0000 UTC m=+762.565842167"
Sep 08 02:11:32.801947 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:32.801910    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-308137-6f98dd6cd6-kjbgr"]
Sep 08 02:11:32.809012 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:32.808977    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-308137-6f98dd6cd6-kjbgr"
Sep 08 02:11:32.811517 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:32.811488    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-308137-dockercfg-szfch\""
Sep 08 02:11:32.811647 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:32.811492    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-308137\""
Sep 08 02:11:32.815828 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:32.815801    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-308137-6f98dd6cd6-kjbgr"]
Sep 08 02:11:32.828343 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:32.828316    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-308137-78b55f848f-85lm9"]
Sep 08 02:11:32.831579 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:32.831560    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-308137-78b55f848f-85lm9"
Sep 08 02:11:32.838555 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:32.838521    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-308137-78b55f848f-85lm9"]
Sep 08 02:11:32.871795 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:32.871762    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0db8cd0-b76b-47ef-b523-e784fe3dc4bf-plugins-config-volume\") pod \"payload-processing-e2e-autores-308137-78b55f848f-85lm9\" (UID: \"a0db8cd0-b76b-47ef-b523-e784fe3dc4bf\") " pod="openshift-ingress/payload-processing-e2e-autores-308137-78b55f848f-85lm9"
Sep 08 02:11:32.871795 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:32.871802    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdkn5\" (UniqueName: \"kubernetes.io/projected/4cfe1a5f-d9d4-47b0-8bfb-e7ad471b8c81-kube-api-access-jdkn5\") pod \"payload-pre-processing-e2e-autores-308137-6f98dd6cd6-kjbgr\" (UID: \"4cfe1a5f-d9d4-47b0-8bfb-e7ad471b8c81\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-308137-6f98dd6cd6-kjbgr"
Sep 08 02:11:32.871998 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:32.871842    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m444h\" (UniqueName: \"kubernetes.io/projected/a0db8cd0-b76b-47ef-b523-e784fe3dc4bf-kube-api-access-m444h\") pod \"payload-processing-e2e-autores-308137-78b55f848f-85lm9\" (UID: \"a0db8cd0-b76b-47ef-b523-e784fe3dc4bf\") " pod="openshift-ingress/payload-processing-e2e-autores-308137-78b55f848f-85lm9"
Sep 08 02:11:32.871998 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:32.871907    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cfe1a5f-d9d4-47b0-8bfb-e7ad471b8c81-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-308137-6f98dd6cd6-kjbgr\" (UID: \"4cfe1a5f-d9d4-47b0-8bfb-e7ad471b8c81\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-308137-6f98dd6cd6-kjbgr"
Sep 08 02:11:32.896860 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:32.896824    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-a-d20bc0ce-kserve-6fd76b86d-7tqkx"]
Sep 08 02:11:32.899987 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:32.899969    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-d20bc0ce-kserve-6fd76b86d-7tqkx"
Sep 08 02:11:32.902535 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:32.902511    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-a-d20bc0ce-kserve-self-signed-certs\""
Sep 08 02:11:32.908650 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:32.908626    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-d20bc0ce-kserve-6fd76b86d-7tqkx"]
Sep 08 02:11:32.972761 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:32.972728    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc319422-532a-49e2-9aea-733525d4461f-model-cache\") pod \"e2e-conflict-a-d20bc0ce-kserve-6fd76b86d-7tqkx\" (UID: \"fc319422-532a-49e2-9aea-733525d4461f\") " pod="llm/e2e-conflict-a-d20bc0ce-kserve-6fd76b86d-7tqkx"
Sep 08 02:11:32.972761 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:32.972764    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc319422-532a-49e2-9aea-733525d4461f-dshm\") pod \"e2e-conflict-a-d20bc0ce-kserve-6fd76b86d-7tqkx\" (UID: \"fc319422-532a-49e2-9aea-733525d4461f\") " pod="llm/e2e-conflict-a-d20bc0ce-kserve-6fd76b86d-7tqkx"
Sep 08 02:11:32.973012 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:32.972802    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc319422-532a-49e2-9aea-733525d4461f-home\") pod \"e2e-conflict-a-d20bc0ce-kserve-6fd76b86d-7tqkx\" (UID: \"fc319422-532a-49e2-9aea-733525d4461f\") " pod="llm/e2e-conflict-a-d20bc0ce-kserve-6fd76b86d-7tqkx"
Sep 08 02:11:32.973012 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:32.972821    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc319422-532a-49e2-9aea-733525d4461f-tmp-dir\") pod \"e2e-conflict-a-d20bc0ce-kserve-6fd76b86d-7tqkx\" (UID: \"fc319422-532a-49e2-9aea-733525d4461f\") " pod="llm/e2e-conflict-a-d20bc0ce-kserve-6fd76b86d-7tqkx"
Sep 08 02:11:32.973012 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:32.972884    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0db8cd0-b76b-47ef-b523-e784fe3dc4bf-plugins-config-volume\") pod \"payload-processing-e2e-autores-308137-78b55f848f-85lm9\" (UID: \"a0db8cd0-b76b-47ef-b523-e784fe3dc4bf\") " pod="openshift-ingress/payload-processing-e2e-autores-308137-78b55f848f-85lm9"
Sep 08 02:11:32.973012 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:32.972931    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdkn5\" (UniqueName: \"kubernetes.io/projected/4cfe1a5f-d9d4-47b0-8bfb-e7ad471b8c81-kube-api-access-jdkn5\") pod \"payload-pre-processing-e2e-autores-308137-6f98dd6cd6-kjbgr\" (UID: \"4cfe1a5f-d9d4-47b0-8bfb-e7ad471b8c81\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-308137-6f98dd6cd6-kjbgr"
Sep 08 02:11:32.973012 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:32.973004    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc319422-532a-49e2-9aea-733525d4461f-tls-certs\") pod \"e2e-conflict-a-d20bc0ce-kserve-6fd76b86d-7tqkx\" (UID: \"fc319422-532a-49e2-9aea-733525d4461f\") " pod="llm/e2e-conflict-a-d20bc0ce-kserve-6fd76b86d-7tqkx"
Sep 08 02:11:32.973318 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:32.973036    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m444h\" (UniqueName: \"kubernetes.io/projected/a0db8cd0-b76b-47ef-b523-e784fe3dc4bf-kube-api-access-m444h\") pod \"payload-processing-e2e-autores-308137-78b55f848f-85lm9\" (UID: \"a0db8cd0-b76b-47ef-b523-e784fe3dc4bf\") " pod="openshift-ingress/payload-processing-e2e-autores-308137-78b55f848f-85lm9"
Sep 08 02:11:32.973318 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:32.973078    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cfe1a5f-d9d4-47b0-8bfb-e7ad471b8c81-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-308137-6f98dd6cd6-kjbgr\" (UID: \"4cfe1a5f-d9d4-47b0-8bfb-e7ad471b8c81\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-308137-6f98dd6cd6-kjbgr"
Sep 08 02:11:32.973318 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:32.973105    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45kr4\" (UniqueName: \"kubernetes.io/projected/fc319422-532a-49e2-9aea-733525d4461f-kube-api-access-45kr4\") pod \"e2e-conflict-a-d20bc0ce-kserve-6fd76b86d-7tqkx\" (UID: \"fc319422-532a-49e2-9aea-733525d4461f\") " pod="llm/e2e-conflict-a-d20bc0ce-kserve-6fd76b86d-7tqkx"
Sep 08 02:11:32.973726 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:32.973707    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0db8cd0-b76b-47ef-b523-e784fe3dc4bf-plugins-config-volume\") pod \"payload-processing-e2e-autores-308137-78b55f848f-85lm9\" (UID: \"a0db8cd0-b76b-47ef-b523-e784fe3dc4bf\") " pod="openshift-ingress/payload-processing-e2e-autores-308137-78b55f848f-85lm9"
Sep 08 02:11:32.973826 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:32.973808    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cfe1a5f-d9d4-47b0-8bfb-e7ad471b8c81-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-308137-6f98dd6cd6-kjbgr\" (UID: \"4cfe1a5f-d9d4-47b0-8bfb-e7ad471b8c81\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-308137-6f98dd6cd6-kjbgr"
Sep 08 02:11:32.981792 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:32.981769    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdkn5\" (UniqueName: \"kubernetes.io/projected/4cfe1a5f-d9d4-47b0-8bfb-e7ad471b8c81-kube-api-access-jdkn5\") pod \"payload-pre-processing-e2e-autores-308137-6f98dd6cd6-kjbgr\" (UID: \"4cfe1a5f-d9d4-47b0-8bfb-e7ad471b8c81\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-308137-6f98dd6cd6-kjbgr"
Sep 08 02:11:32.981901 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:32.981836    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m444h\" (UniqueName: \"kubernetes.io/projected/a0db8cd0-b76b-47ef-b523-e784fe3dc4bf-kube-api-access-m444h\") pod \"payload-processing-e2e-autores-308137-78b55f848f-85lm9\" (UID: \"a0db8cd0-b76b-47ef-b523-e784fe3dc4bf\") " pod="openshift-ingress/payload-processing-e2e-autores-308137-78b55f848f-85lm9"
Sep 08 02:11:33.073933 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:33.073838    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc319422-532a-49e2-9aea-733525d4461f-tls-certs\") pod \"e2e-conflict-a-d20bc0ce-kserve-6fd76b86d-7tqkx\" (UID: \"fc319422-532a-49e2-9aea-733525d4461f\") " pod="llm/e2e-conflict-a-d20bc0ce-kserve-6fd76b86d-7tqkx"
Sep 08 02:11:33.073933 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:33.073881    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45kr4\" (UniqueName: \"kubernetes.io/projected/fc319422-532a-49e2-9aea-733525d4461f-kube-api-access-45kr4\") pod \"e2e-conflict-a-d20bc0ce-kserve-6fd76b86d-7tqkx\" (UID: \"fc319422-532a-49e2-9aea-733525d4461f\") " pod="llm/e2e-conflict-a-d20bc0ce-kserve-6fd76b86d-7tqkx"
Sep 08 02:11:33.073933 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:33.073915    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc319422-532a-49e2-9aea-733525d4461f-model-cache\") pod \"e2e-conflict-a-d20bc0ce-kserve-6fd76b86d-7tqkx\" (UID: \"fc319422-532a-49e2-9aea-733525d4461f\") " pod="llm/e2e-conflict-a-d20bc0ce-kserve-6fd76b86d-7tqkx"
Sep 08 02:11:33.074240 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:33.073940    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc319422-532a-49e2-9aea-733525d4461f-dshm\") pod \"e2e-conflict-a-d20bc0ce-kserve-6fd76b86d-7tqkx\" (UID: \"fc319422-532a-49e2-9aea-733525d4461f\") " pod="llm/e2e-conflict-a-d20bc0ce-kserve-6fd76b86d-7tqkx"
Sep 08 02:11:33.074240 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:33.073999    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc319422-532a-49e2-9aea-733525d4461f-home\") pod \"e2e-conflict-a-d20bc0ce-kserve-6fd76b86d-7tqkx\" (UID: \"fc319422-532a-49e2-9aea-733525d4461f\") " pod="llm/e2e-conflict-a-d20bc0ce-kserve-6fd76b86d-7tqkx"
Sep 08 02:11:33.074240 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:33.074023    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc319422-532a-49e2-9aea-733525d4461f-tmp-dir\") pod \"e2e-conflict-a-d20bc0ce-kserve-6fd76b86d-7tqkx\" (UID: \"fc319422-532a-49e2-9aea-733525d4461f\") " pod="llm/e2e-conflict-a-d20bc0ce-kserve-6fd76b86d-7tqkx"
Sep 08 02:11:33.074407 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:33.074383    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc319422-532a-49e2-9aea-733525d4461f-model-cache\") pod \"e2e-conflict-a-d20bc0ce-kserve-6fd76b86d-7tqkx\" (UID: \"fc319422-532a-49e2-9aea-733525d4461f\") " pod="llm/e2e-conflict-a-d20bc0ce-kserve-6fd76b86d-7tqkx"
Sep 08 02:11:33.074492 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:33.074469    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc319422-532a-49e2-9aea-733525d4461f-tmp-dir\") pod \"e2e-conflict-a-d20bc0ce-kserve-6fd76b86d-7tqkx\" (UID: \"fc319422-532a-49e2-9aea-733525d4461f\") " pod="llm/e2e-conflict-a-d20bc0ce-kserve-6fd76b86d-7tqkx"
Sep 08 02:11:33.074589 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:33.074570    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc319422-532a-49e2-9aea-733525d4461f-home\") pod \"e2e-conflict-a-d20bc0ce-kserve-6fd76b86d-7tqkx\" (UID: \"fc319422-532a-49e2-9aea-733525d4461f\") " pod="llm/e2e-conflict-a-d20bc0ce-kserve-6fd76b86d-7tqkx"
Sep 08 02:11:33.076275 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:33.076256    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc319422-532a-49e2-9aea-733525d4461f-dshm\") pod \"e2e-conflict-a-d20bc0ce-kserve-6fd76b86d-7tqkx\" (UID: \"fc319422-532a-49e2-9aea-733525d4461f\") " pod="llm/e2e-conflict-a-d20bc0ce-kserve-6fd76b86d-7tqkx"
Sep 08 02:11:33.076532 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:33.076514    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc319422-532a-49e2-9aea-733525d4461f-tls-certs\") pod \"e2e-conflict-a-d20bc0ce-kserve-6fd76b86d-7tqkx\" (UID: \"fc319422-532a-49e2-9aea-733525d4461f\") " pod="llm/e2e-conflict-a-d20bc0ce-kserve-6fd76b86d-7tqkx"
Sep 08 02:11:33.081209 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:33.081186    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45kr4\" (UniqueName: \"kubernetes.io/projected/fc319422-532a-49e2-9aea-733525d4461f-kube-api-access-45kr4\") pod \"e2e-conflict-a-d20bc0ce-kserve-6fd76b86d-7tqkx\" (UID: \"fc319422-532a-49e2-9aea-733525d4461f\") " pod="llm/e2e-conflict-a-d20bc0ce-kserve-6fd76b86d-7tqkx"
Sep 08 02:11:33.121842 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:33.121796    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-308137-6f98dd6cd6-kjbgr"
Sep 08 02:11:33.143097 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:33.143063    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-308137-78b55f848f-85lm9"
Sep 08 02:11:33.210807 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:33.210679    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-d20bc0ce-kserve-6fd76b86d-7tqkx"
Sep 08 02:11:33.395527 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:33.395438    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-308137-6f98dd6cd6-kjbgr"]
Sep 08 02:11:33.398067 ip-10-0-142-78 kubenswrapper[2547]: W0908 02:11:33.398037    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cfe1a5f_d9d4_47b0_8bfb_e7ad471b8c81.slice/crio-43b1ff44a3ef471cca58297661e29961adff84eed0ba2690930f373c1b968388 WatchSource:0}: Error finding container 43b1ff44a3ef471cca58297661e29961adff84eed0ba2690930f373c1b968388: Status 404 returned error can't find the container with id 43b1ff44a3ef471cca58297661e29961adff84eed0ba2690930f373c1b968388
Sep 08 02:11:33.422440 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:33.422411    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-308137-78b55f848f-85lm9"]
Sep 08 02:11:33.430262 ip-10-0-142-78 kubenswrapper[2547]: W0908 02:11:33.430238    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0db8cd0_b76b_47ef_b523_e784fe3dc4bf.slice/crio-d7881a59a8efe4fc2bdd1e22433af1b4e0163ef8a33a5ea443933678e43e972e WatchSource:0}: Error finding container d7881a59a8efe4fc2bdd1e22433af1b4e0163ef8a33a5ea443933678e43e972e: Status 404 returned error can't find the container with id d7881a59a8efe4fc2bdd1e22433af1b4e0163ef8a33a5ea443933678e43e972e
Sep 08 02:11:33.491386 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:33.491362    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-d20bc0ce-kserve-6fd76b86d-7tqkx"]
Sep 08 02:11:33.493819 ip-10-0-142-78 kubenswrapper[2547]: W0908 02:11:33.493796    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfc319422_532a_49e2_9aea_733525d4461f.slice/crio-d7e773a979d74367c002b229b701302f4dd1b23cf527a9441f0ecf18c70bf386 WatchSource:0}: Error finding container d7e773a979d74367c002b229b701302f4dd1b23cf527a9441f0ecf18c70bf386: Status 404 returned error can't find the container with id d7e773a979d74367c002b229b701302f4dd1b23cf527a9441f0ecf18c70bf386
Sep 08 02:11:34.041821 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:34.041783    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-308137-78b55f848f-85lm9" event={"ID":"a0db8cd0-b76b-47ef-b523-e784fe3dc4bf","Type":"ContainerStarted","Data":"3aa7cd99bc65d631808f8609b27b140ba00e2d49bf0dfdfb44fc3dd672b7cbba"}
Sep 08 02:11:34.042323 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:34.041849    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-308137-78b55f848f-85lm9"
Sep 08 02:11:34.042323 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:34.041873    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-308137-78b55f848f-85lm9" event={"ID":"a0db8cd0-b76b-47ef-b523-e784fe3dc4bf","Type":"ContainerStarted","Data":"d7881a59a8efe4fc2bdd1e22433af1b4e0163ef8a33a5ea443933678e43e972e"}
Sep 08 02:11:34.043316 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:34.043289    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-d20bc0ce-kserve-6fd76b86d-7tqkx" event={"ID":"fc319422-532a-49e2-9aea-733525d4461f","Type":"ContainerStarted","Data":"b97d5876801b3650476c95e4dd1f6b3f7075474a789577c42728bc1b45b24c26"}
Sep 08 02:11:34.043428 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:34.043318    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-d20bc0ce-kserve-6fd76b86d-7tqkx" event={"ID":"fc319422-532a-49e2-9aea-733525d4461f","Type":"ContainerStarted","Data":"d7e773a979d74367c002b229b701302f4dd1b23cf527a9441f0ecf18c70bf386"}
Sep 08 02:11:34.044738 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:34.044716    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-308137-6f98dd6cd6-kjbgr" event={"ID":"4cfe1a5f-d9d4-47b0-8bfb-e7ad471b8c81","Type":"ContainerStarted","Data":"2f8cb1ce9262f2ade44e5d09fd33789520259cef445fc85497c3d731b6091e3c"}
Sep 08 02:11:34.044838 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:34.044765    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-308137-6f98dd6cd6-kjbgr"
Sep 08 02:11:34.044838 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:34.044787    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-308137-6f98dd6cd6-kjbgr" event={"ID":"4cfe1a5f-d9d4-47b0-8bfb-e7ad471b8c81","Type":"ContainerStarted","Data":"43b1ff44a3ef471cca58297661e29961adff84eed0ba2690930f373c1b968388"}
Sep 08 02:11:34.060301 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:34.060254    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-308137-78b55f848f-85lm9" podStartSLOduration=2.060239607 podStartE2EDuration="2.060239607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:11:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:11:34.058260431 +0000 UTC m=+765.582035521" watchObservedRunningTime="2026-09-08 02:11:34.060239607 +0000 UTC m=+765.584014687"
Sep 08 02:11:34.076968 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:34.076909    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-308137-6f98dd6cd6-kjbgr" podStartSLOduration=2.076891616 podStartE2EDuration="2.076891616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:11:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:11:34.075760169 +0000 UTC m=+765.599535252" watchObservedRunningTime="2026-09-08 02:11:34.076891616 +0000 UTC m=+765.600666701"
Sep 08 02:11:34.096526 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:34.096478    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-a-d20bc0ce-kserve-6fd76b86d-7tqkx" podStartSLOduration=2.09646416 podStartE2EDuration="2.09646416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:11:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:11:34.094724455 +0000 UTC m=+765.618499550" watchObservedRunningTime="2026-09-08 02:11:34.09646416 +0000 UTC m=+765.620239240"
Sep 08 02:11:35.000718 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:35.000690    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-5d27de-58656496c5-qzvbz"
Sep 08 02:11:35.000917 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:35.000763    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5d27de-78684b5958-vtg65"
Sep 08 02:11:42.033659 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:42.033622    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-213d54df-5f57b864cd-9jfw7"
Sep 08 02:11:43.211767 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:43.211724    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-a-d20bc0ce-kserve-6fd76b86d-7tqkx"
Sep 08 02:11:43.212236 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:43.211785    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-a-d20bc0ce-kserve-6fd76b86d-7tqkx"
Sep 08 02:11:43.224296 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:43.224263    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-a-d20bc0ce-kserve-6fd76b86d-7tqkx"
Sep 08 02:11:44.101644 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:44.101606    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-a-d20bc0ce-kserve-6fd76b86d-7tqkx"
Sep 08 02:11:45.052158 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:45.052101    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-308137-78b55f848f-85lm9"
Sep 08 02:11:45.052576 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:45.052202    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-308137-6f98dd6cd6-kjbgr"
Sep 08 02:11:50.684552 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:50.684512    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8d800a-7fd6f4554-wz878"]
Sep 08 02:11:50.687506 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:50.687488    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8d800a-7fd6f4554-wz878"
Sep 08 02:11:50.689841 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:50.689812    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-8d800a\""
Sep 08 02:11:50.700182 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:50.700158    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8d800a-7fd6f4554-wz878"]
Sep 08 02:11:50.702697 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:50.702669    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8d800a-6b455db645-cxmzp"]
Sep 08 02:11:50.706386 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:50.706368    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8d800a-6b455db645-cxmzp"
Sep 08 02:11:50.708979 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:50.708955    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-8d800a-dockercfg-5t8v7\""
Sep 08 02:11:50.716608 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:50.716581    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8d800a-6b455db645-cxmzp"]
Sep 08 02:11:50.729126 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:50.729088    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmjlf\" (UniqueName: \"kubernetes.io/projected/5d8fcaa0-2ffb-4931-931e-2b0d3f06d083-kube-api-access-lmjlf\") pod \"payload-pre-processing-e2e-inf-b-8d800a-7fd6f4554-wz878\" (UID: \"5d8fcaa0-2ffb-4931-931e-2b0d3f06d083\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8d800a-7fd6f4554-wz878"
Sep 08 02:11:50.729320 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:50.729219    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d8fcaa0-2ffb-4931-931e-2b0d3f06d083-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-8d800a-7fd6f4554-wz878\" (UID: \"5d8fcaa0-2ffb-4931-931e-2b0d3f06d083\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8d800a-7fd6f4554-wz878"
Sep 08 02:11:50.830730 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:50.830690    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0318e043-46c2-435c-ba36-a07110b8e456-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-8d800a-6b455db645-cxmzp\" (UID: \"0318e043-46c2-435c-ba36-a07110b8e456\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8d800a-6b455db645-cxmzp"
Sep 08 02:11:50.830931 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:50.830746    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmjlf\" (UniqueName: \"kubernetes.io/projected/5d8fcaa0-2ffb-4931-931e-2b0d3f06d083-kube-api-access-lmjlf\") pod \"payload-pre-processing-e2e-inf-b-8d800a-7fd6f4554-wz878\" (UID: \"5d8fcaa0-2ffb-4931-931e-2b0d3f06d083\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8d800a-7fd6f4554-wz878"
Sep 08 02:11:50.830931 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:50.830887    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sk8dt\" (UniqueName: \"kubernetes.io/projected/0318e043-46c2-435c-ba36-a07110b8e456-kube-api-access-sk8dt\") pod \"payload-processing-e2e-inf-b-8d800a-6b455db645-cxmzp\" (UID: \"0318e043-46c2-435c-ba36-a07110b8e456\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8d800a-6b455db645-cxmzp"
Sep 08 02:11:50.831042 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:50.830962    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d8fcaa0-2ffb-4931-931e-2b0d3f06d083-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-8d800a-7fd6f4554-wz878\" (UID: \"5d8fcaa0-2ffb-4931-931e-2b0d3f06d083\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8d800a-7fd6f4554-wz878"
Sep 08 02:11:50.831681 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:50.831656    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d8fcaa0-2ffb-4931-931e-2b0d3f06d083-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-8d800a-7fd6f4554-wz878\" (UID: \"5d8fcaa0-2ffb-4931-931e-2b0d3f06d083\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8d800a-7fd6f4554-wz878"
Sep 08 02:11:50.839598 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:50.839574    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmjlf\" (UniqueName: \"kubernetes.io/projected/5d8fcaa0-2ffb-4931-931e-2b0d3f06d083-kube-api-access-lmjlf\") pod \"payload-pre-processing-e2e-inf-b-8d800a-7fd6f4554-wz878\" (UID: \"5d8fcaa0-2ffb-4931-931e-2b0d3f06d083\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8d800a-7fd6f4554-wz878"
Sep 08 02:11:50.931849 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:50.931799    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk8dt\" (UniqueName: \"kubernetes.io/projected/0318e043-46c2-435c-ba36-a07110b8e456-kube-api-access-sk8dt\") pod \"payload-processing-e2e-inf-b-8d800a-6b455db645-cxmzp\" (UID: \"0318e043-46c2-435c-ba36-a07110b8e456\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8d800a-6b455db645-cxmzp"
Sep 08 02:11:50.932042 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:50.931910    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0318e043-46c2-435c-ba36-a07110b8e456-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-8d800a-6b455db645-cxmzp\" (UID: \"0318e043-46c2-435c-ba36-a07110b8e456\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8d800a-6b455db645-cxmzp"
Sep 08 02:11:50.932626 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:50.932601    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0318e043-46c2-435c-ba36-a07110b8e456-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-8d800a-6b455db645-cxmzp\" (UID: \"0318e043-46c2-435c-ba36-a07110b8e456\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8d800a-6b455db645-cxmzp"
Sep 08 02:11:50.940332 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:50.940276    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk8dt\" (UniqueName: \"kubernetes.io/projected/0318e043-46c2-435c-ba36-a07110b8e456-kube-api-access-sk8dt\") pod \"payload-processing-e2e-inf-b-8d800a-6b455db645-cxmzp\" (UID: \"0318e043-46c2-435c-ba36-a07110b8e456\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8d800a-6b455db645-cxmzp"
Sep 08 02:11:50.998965 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:50.998931    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8d800a-7fd6f4554-wz878"
Sep 08 02:11:51.019244 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:51.019211    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8d800a-6b455db645-cxmzp"
Sep 08 02:11:51.272032 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:51.271959    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8d800a-7fd6f4554-wz878"]
Sep 08 02:11:51.274926 ip-10-0-142-78 kubenswrapper[2547]: W0908 02:11:51.274901    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d8fcaa0_2ffb_4931_931e_2b0d3f06d083.slice/crio-3f9d22a6cdbb6ee914ae48f2e0371d7b873e4ceca00065ca56d5834b00706180 WatchSource:0}: Error finding container 3f9d22a6cdbb6ee914ae48f2e0371d7b873e4ceca00065ca56d5834b00706180: Status 404 returned error can't find the container with id 3f9d22a6cdbb6ee914ae48f2e0371d7b873e4ceca00065ca56d5834b00706180
Sep 08 02:11:51.302332 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:51.302307    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8d800a-6b455db645-cxmzp"]
Sep 08 02:11:51.307184 ip-10-0-142-78 kubenswrapper[2547]: W0908 02:11:51.307151    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0318e043_46c2_435c_ba36_a07110b8e456.slice/crio-c615d5505b4f17084678de3e93dff99caac433de3413987145dd06d5d9ceb789 WatchSource:0}: Error finding container c615d5505b4f17084678de3e93dff99caac433de3413987145dd06d5d9ceb789: Status 404 returned error can't find the container with id c615d5505b4f17084678de3e93dff99caac433de3413987145dd06d5d9ceb789
Sep 08 02:11:52.135193 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:52.135150    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8d800a-7fd6f4554-wz878" event={"ID":"5d8fcaa0-2ffb-4931-931e-2b0d3f06d083","Type":"ContainerStarted","Data":"56c4ead1c154da2f43a1be9e896a70960e52900756c7bfd45850116eec4ef6a9"}
Sep 08 02:11:52.135649 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:52.135249    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8d800a-7fd6f4554-wz878" event={"ID":"5d8fcaa0-2ffb-4931-931e-2b0d3f06d083","Type":"ContainerStarted","Data":"3f9d22a6cdbb6ee914ae48f2e0371d7b873e4ceca00065ca56d5834b00706180"}
Sep 08 02:11:52.135649 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:52.135285    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8d800a-7fd6f4554-wz878"
Sep 08 02:11:52.136656 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:52.136631    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8d800a-6b455db645-cxmzp" event={"ID":"0318e043-46c2-435c-ba36-a07110b8e456","Type":"ContainerStarted","Data":"d6257168b5093ff502ce5537899287ab84824059eab8f18c1f591b6929a4ed6b"}
Sep 08 02:11:52.136723 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:52.136677    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8d800a-6b455db645-cxmzp" event={"ID":"0318e043-46c2-435c-ba36-a07110b8e456","Type":"ContainerStarted","Data":"c615d5505b4f17084678de3e93dff99caac433de3413987145dd06d5d9ceb789"}
Sep 08 02:11:52.136723 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:52.136713    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-8d800a-6b455db645-cxmzp"
Sep 08 02:11:52.150341 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:52.150289    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8d800a-7fd6f4554-wz878" podStartSLOduration=2.150274156 podStartE2EDuration="2.150274156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:11:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:11:52.149067992 +0000 UTC m=+783.672843074" watchObservedRunningTime="2026-09-08 02:11:52.150274156 +0000 UTC m=+783.674049237"
Sep 08 02:11:52.163694 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:11:52.163629    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-8d800a-6b455db645-cxmzp" podStartSLOduration=2.163612793 podStartE2EDuration="2.163612793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:11:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:11:52.162602089 +0000 UTC m=+783.686377192" watchObservedRunningTime="2026-09-08 02:11:52.163612793 +0000 UTC m=+783.687387873"
Sep 08 02:12:03.142840 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:12:03.142805    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-8d800a-6b455db645-cxmzp"
Sep 08 02:12:03.143425 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:12:03.142877    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8d800a-7fd6f4554-wz878"
Sep 08 02:12:10.261926 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:12:10.261887    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-213d54df-5f57b864cd-9jfw7"]
Sep 08 02:12:10.262764 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:12:10.262729    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-213d54df-5f57b864cd-9jfw7" podUID="e8a225a9-0b57-4042-8033-166d2291558f" containerName="payload-pre-processing" containerID="cri-o://4417bed515fb87575514f93afeeb9cc7591204a3f75b74dad9974629f1e57077" gracePeriod=30
Sep 08 02:12:10.515508 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:12:10.515433    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-213d54df-5f57b864cd-9jfw7"
Sep 08 02:12:10.612073 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:12:10.612036    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8a225a9-0b57-4042-8033-166d2291558f-plugins-config-volume\") pod \"e8a225a9-0b57-4042-8033-166d2291558f\" (UID: \"e8a225a9-0b57-4042-8033-166d2291558f\") "
Sep 08 02:12:10.612260 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:12:10.612091    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7fh62\" (UniqueName: \"kubernetes.io/projected/e8a225a9-0b57-4042-8033-166d2291558f-kube-api-access-7fh62\") pod \"e8a225a9-0b57-4042-8033-166d2291558f\" (UID: \"e8a225a9-0b57-4042-8033-166d2291558f\") "
Sep 08 02:12:10.612505 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:12:10.612478    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8a225a9-0b57-4042-8033-166d2291558f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e8a225a9-0b57-4042-8033-166d2291558f" (UID: "e8a225a9-0b57-4042-8033-166d2291558f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 02:12:10.614271 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:12:10.614247    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8a225a9-0b57-4042-8033-166d2291558f-kube-api-access-7fh62" (OuterVolumeSpecName: "kube-api-access-7fh62") pod "e8a225a9-0b57-4042-8033-166d2291558f" (UID: "e8a225a9-0b57-4042-8033-166d2291558f"). InnerVolumeSpecName "kube-api-access-7fh62". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 02:12:10.713522 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:12:10.713485    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fh62\" (UniqueName: \"kubernetes.io/projected/e8a225a9-0b57-4042-8033-166d2291558f-kube-api-access-7fh62\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:12:10.713522 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:12:10.713518    2547 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8a225a9-0b57-4042-8033-166d2291558f-plugins-config-volume\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:12:11.230285 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:12:11.230206    2547 generic.go:356] "Generic (PLEG): container finished" podID="e8a225a9-0b57-4042-8033-166d2291558f" containerID="4417bed515fb87575514f93afeeb9cc7591204a3f75b74dad9974629f1e57077" exitCode=0
Sep 08 02:12:11.230285 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:12:11.230253    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-213d54df-5f57b864cd-9jfw7"
Sep 08 02:12:11.230474 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:12:11.230289    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-213d54df-5f57b864cd-9jfw7" event={"ID":"e8a225a9-0b57-4042-8033-166d2291558f","Type":"ContainerDied","Data":"4417bed515fb87575514f93afeeb9cc7591204a3f75b74dad9974629f1e57077"}
Sep 08 02:12:11.230474 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:12:11.230342    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-213d54df-5f57b864cd-9jfw7" event={"ID":"e8a225a9-0b57-4042-8033-166d2291558f","Type":"ContainerDied","Data":"b479d2138dc13e1cc76a9f256bc1b773903016f2371d5f119a41b0d435f7e54c"}
Sep 08 02:12:11.230474 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:12:11.230367    2547 scope.go:117] "RemoveContainer" containerID="4417bed515fb87575514f93afeeb9cc7591204a3f75b74dad9974629f1e57077"
Sep 08 02:12:11.238780 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:12:11.238755    2547 scope.go:117] "RemoveContainer" containerID="4417bed515fb87575514f93afeeb9cc7591204a3f75b74dad9974629f1e57077"
Sep 08 02:12:11.239086 ip-10-0-142-78 kubenswrapper[2547]: E0908 02:12:11.239067    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4417bed515fb87575514f93afeeb9cc7591204a3f75b74dad9974629f1e57077\": container with ID starting with 4417bed515fb87575514f93afeeb9cc7591204a3f75b74dad9974629f1e57077 not found: ID does not exist" containerID="4417bed515fb87575514f93afeeb9cc7591204a3f75b74dad9974629f1e57077"
Sep 08 02:12:11.239176 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:12:11.239094    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4417bed515fb87575514f93afeeb9cc7591204a3f75b74dad9974629f1e57077"} err="failed to get container status \"4417bed515fb87575514f93afeeb9cc7591204a3f75b74dad9974629f1e57077\": rpc error: code = NotFound desc = could not find container \"4417bed515fb87575514f93afeeb9cc7591204a3f75b74dad9974629f1e57077\": container with ID starting with 4417bed515fb87575514f93afeeb9cc7591204a3f75b74dad9974629f1e57077 not found: ID does not exist"
Sep 08 02:12:11.245641 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:12:11.245601    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-213d54df-5f57b864cd-9jfw7"]
Sep 08 02:12:11.247763 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:12:11.247739    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-213d54df-5f57b864cd-9jfw7"]
Sep 08 02:12:12.994696 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:12:12.994654    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8a225a9-0b57-4042-8033-166d2291558f" path="/var/lib/kubelet/pods/e8a225a9-0b57-4042-8033-166d2291558f/volumes"
Sep 08 02:12:30.311758 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:12:30.311723    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-b-d20bc0ce-kserve-6df5985594-r58x6"]
Sep 08 02:12:30.312225 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:12:30.312101    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8a225a9-0b57-4042-8033-166d2291558f" containerName="payload-pre-processing"
Sep 08 02:12:30.312225 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:12:30.312113    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8a225a9-0b57-4042-8033-166d2291558f" containerName="payload-pre-processing"
Sep 08 02:12:30.312225 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:12:30.312215    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8a225a9-0b57-4042-8033-166d2291558f" containerName="payload-pre-processing"
Sep 08 02:12:30.315460 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:12:30.315437    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-d20bc0ce-kserve-6df5985594-r58x6"
Sep 08 02:12:30.317533 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:12:30.317504    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-b-d20bc0ce-kserve-self-signed-certs\""
Sep 08 02:12:30.324820 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:12:30.324793    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-d20bc0ce-kserve-6df5985594-r58x6"]
Sep 08 02:12:30.384108 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:12:30.384065    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f142f130-eceb-4f5e-b35e-a736c3735584-model-cache\") pod \"e2e-conflict-b-d20bc0ce-kserve-6df5985594-r58x6\" (UID: \"f142f130-eceb-4f5e-b35e-a736c3735584\") " pod="llm/e2e-conflict-b-d20bc0ce-kserve-6df5985594-r58x6"
Sep 08 02:12:30.384108 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:12:30.384109    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qgvg\" (UniqueName: \"kubernetes.io/projected/f142f130-eceb-4f5e-b35e-a736c3735584-kube-api-access-7qgvg\") pod \"e2e-conflict-b-d20bc0ce-kserve-6df5985594-r58x6\" (UID: \"f142f130-eceb-4f5e-b35e-a736c3735584\") " pod="llm/e2e-conflict-b-d20bc0ce-kserve-6df5985594-r58x6"
Sep 08 02:12:30.384397 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:12:30.384257    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f142f130-eceb-4f5e-b35e-a736c3735584-tls-certs\") pod \"e2e-conflict-b-d20bc0ce-kserve-6df5985594-r58x6\" (UID: \"f142f130-eceb-4f5e-b35e-a736c3735584\") " pod="llm/e2e-conflict-b-d20bc0ce-kserve-6df5985594-r58x6"
Sep 08 02:12:30.384397 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:12:30.384291    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f142f130-eceb-4f5e-b35e-a736c3735584-dshm\") pod \"e2e-conflict-b-d20bc0ce-kserve-6df5985594-r58x6\" (UID: \"f142f130-eceb-4f5e-b35e-a736c3735584\") " pod="llm/e2e-conflict-b-d20bc0ce-kserve-6df5985594-r58x6"
Sep 08 02:12:30.384397 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:12:30.384355    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f142f130-eceb-4f5e-b35e-a736c3735584-home\") pod \"e2e-conflict-b-d20bc0ce-kserve-6df5985594-r58x6\" (UID: \"f142f130-eceb-4f5e-b35e-a736c3735584\") " pod="llm/e2e-conflict-b-d20bc0ce-kserve-6df5985594-r58x6"
Sep 08 02:12:30.384536 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:12:30.384434    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f142f130-eceb-4f5e-b35e-a736c3735584-tmp-dir\") pod \"e2e-conflict-b-d20bc0ce-kserve-6df5985594-r58x6\" (UID: \"f142f130-eceb-4f5e-b35e-a736c3735584\") " pod="llm/e2e-conflict-b-d20bc0ce-kserve-6df5985594-r58x6"
Sep 08 02:12:30.485331 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:12:30.485289    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f142f130-eceb-4f5e-b35e-a736c3735584-tmp-dir\") pod \"e2e-conflict-b-d20bc0ce-kserve-6df5985594-r58x6\" (UID: \"f142f130-eceb-4f5e-b35e-a736c3735584\") " pod="llm/e2e-conflict-b-d20bc0ce-kserve-6df5985594-r58x6"
Sep 08 02:12:30.485331 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:12:30.485335    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f142f130-eceb-4f5e-b35e-a736c3735584-model-cache\") pod \"e2e-conflict-b-d20bc0ce-kserve-6df5985594-r58x6\" (UID: \"f142f130-eceb-4f5e-b35e-a736c3735584\") " pod="llm/e2e-conflict-b-d20bc0ce-kserve-6df5985594-r58x6"
Sep 08 02:12:30.485593 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:12:30.485364    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qgvg\" (UniqueName: \"kubernetes.io/projected/f142f130-eceb-4f5e-b35e-a736c3735584-kube-api-access-7qgvg\") pod \"e2e-conflict-b-d20bc0ce-kserve-6df5985594-r58x6\" (UID: \"f142f130-eceb-4f5e-b35e-a736c3735584\") " pod="llm/e2e-conflict-b-d20bc0ce-kserve-6df5985594-r58x6"
Sep 08 02:12:30.485593 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:12:30.485433    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f142f130-eceb-4f5e-b35e-a736c3735584-tls-certs\") pod \"e2e-conflict-b-d20bc0ce-kserve-6df5985594-r58x6\" (UID: \"f142f130-eceb-4f5e-b35e-a736c3735584\") " pod="llm/e2e-conflict-b-d20bc0ce-kserve-6df5985594-r58x6"
Sep 08 02:12:30.485593 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:12:30.485458    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f142f130-eceb-4f5e-b35e-a736c3735584-dshm\") pod \"e2e-conflict-b-d20bc0ce-kserve-6df5985594-r58x6\" (UID: \"f142f130-eceb-4f5e-b35e-a736c3735584\") " pod="llm/e2e-conflict-b-d20bc0ce-kserve-6df5985594-r58x6"
Sep 08 02:12:30.485593 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:12:30.485531    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f142f130-eceb-4f5e-b35e-a736c3735584-home\") pod \"e2e-conflict-b-d20bc0ce-kserve-6df5985594-r58x6\" (UID: \"f142f130-eceb-4f5e-b35e-a736c3735584\") " pod="llm/e2e-conflict-b-d20bc0ce-kserve-6df5985594-r58x6"
Sep 08 02:12:30.485802 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:12:30.485728    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f142f130-eceb-4f5e-b35e-a736c3735584-tmp-dir\") pod \"e2e-conflict-b-d20bc0ce-kserve-6df5985594-r58x6\" (UID: \"f142f130-eceb-4f5e-b35e-a736c3735584\") " pod="llm/e2e-conflict-b-d20bc0ce-kserve-6df5985594-r58x6"
Sep 08 02:12:30.485802 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:12:30.485757    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f142f130-eceb-4f5e-b35e-a736c3735584-model-cache\") pod \"e2e-conflict-b-d20bc0ce-kserve-6df5985594-r58x6\" (UID: \"f142f130-eceb-4f5e-b35e-a736c3735584\") " pod="llm/e2e-conflict-b-d20bc0ce-kserve-6df5985594-r58x6"
Sep 08 02:12:30.485904 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:12:30.485841    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f142f130-eceb-4f5e-b35e-a736c3735584-home\") pod \"e2e-conflict-b-d20bc0ce-kserve-6df5985594-r58x6\" (UID: \"f142f130-eceb-4f5e-b35e-a736c3735584\") " pod="llm/e2e-conflict-b-d20bc0ce-kserve-6df5985594-r58x6"
Sep 08 02:12:30.487809 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:12:30.487782    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f142f130-eceb-4f5e-b35e-a736c3735584-dshm\") pod \"e2e-conflict-b-d20bc0ce-kserve-6df5985594-r58x6\" (UID: \"f142f130-eceb-4f5e-b35e-a736c3735584\") " pod="llm/e2e-conflict-b-d20bc0ce-kserve-6df5985594-r58x6"
Sep 08 02:12:30.488057 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:12:30.488040    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f142f130-eceb-4f5e-b35e-a736c3735584-tls-certs\") pod \"e2e-conflict-b-d20bc0ce-kserve-6df5985594-r58x6\" (UID: \"f142f130-eceb-4f5e-b35e-a736c3735584\") " pod="llm/e2e-conflict-b-d20bc0ce-kserve-6df5985594-r58x6"
Sep 08 02:12:30.499031 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:12:30.499002    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qgvg\" (UniqueName: \"kubernetes.io/projected/f142f130-eceb-4f5e-b35e-a736c3735584-kube-api-access-7qgvg\") pod \"e2e-conflict-b-d20bc0ce-kserve-6df5985594-r58x6\" (UID: \"f142f130-eceb-4f5e-b35e-a736c3735584\") " pod="llm/e2e-conflict-b-d20bc0ce-kserve-6df5985594-r58x6"
Sep 08 02:12:30.625974 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:12:30.625876    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-d20bc0ce-kserve-6df5985594-r58x6"
Sep 08 02:12:30.898202 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:12:30.898173    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-d20bc0ce-kserve-6df5985594-r58x6"]
Sep 08 02:12:30.900605 ip-10-0-142-78 kubenswrapper[2547]: W0908 02:12:30.900577    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf142f130_eceb_4f5e_b35e_a736c3735584.slice/crio-1ba7d8c1e474db4f1683538a9770a75b5ddc8bca470a8edabd213cd26306f6b8 WatchSource:0}: Error finding container 1ba7d8c1e474db4f1683538a9770a75b5ddc8bca470a8edabd213cd26306f6b8: Status 404 returned error can't find the container with id 1ba7d8c1e474db4f1683538a9770a75b5ddc8bca470a8edabd213cd26306f6b8
Sep 08 02:12:31.320006 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:12:31.319922    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-d20bc0ce-kserve-6df5985594-r58x6" event={"ID":"f142f130-eceb-4f5e-b35e-a736c3735584","Type":"ContainerStarted","Data":"ce7961ffdb26c3bce858e12af6d13dd6206462ba8aed4627da753bb936db3cf5"}
Sep 08 02:12:31.320006 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:12:31.319963    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-d20bc0ce-kserve-6df5985594-r58x6" event={"ID":"f142f130-eceb-4f5e-b35e-a736c3735584","Type":"ContainerStarted","Data":"1ba7d8c1e474db4f1683538a9770a75b5ddc8bca470a8edabd213cd26306f6b8"}
Sep 08 02:12:31.338202 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:12:31.338119    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-b-d20bc0ce-kserve-6df5985594-r58x6" podStartSLOduration=1.3381016620000001 podStartE2EDuration="1.338101662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:12:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:12:31.336320807 +0000 UTC m=+822.860095901" watchObservedRunningTime="2026-09-08 02:12:31.338101662 +0000 UTC m=+822.861876742"
Sep 08 02:12:40.626729 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:12:40.626690    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-b-d20bc0ce-kserve-6df5985594-r58x6"
Sep 08 02:12:40.627364 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:12:40.626764    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-b-d20bc0ce-kserve-6df5985594-r58x6"
Sep 08 02:12:40.639146 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:12:40.639106    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-b-d20bc0ce-kserve-6df5985594-r58x6"
Sep 08 02:12:41.376646 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:12:41.376617    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-b-d20bc0ce-kserve-6df5985594-r58x6"
Sep 08 02:13:00.701939 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:00.701897    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-308137/autores-model-308137-kserve-5d658dbd79-xhfzl"]
Sep 08 02:13:00.706910 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:00.706881    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-308137/autores-model-308137-kserve-5d658dbd79-xhfzl"
Sep 08 02:13:00.709051 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:00.709023    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-308137\"/\"default-dockercfg-m685c\""
Sep 08 02:13:00.709196 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:00.709119    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-308137\"/\"openshift-service-ca.crt\""
Sep 08 02:13:00.710117 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:00.710095    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-308137\"/\"autores-model-308137-kserve-self-signed-certs\""
Sep 08 02:13:00.710371 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:00.710351    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-308137\"/\"kube-root-ca.crt\""
Sep 08 02:13:00.712522 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:00.712497    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-308137/autores-model-308137-kserve-5d658dbd79-xhfzl"]
Sep 08 02:13:00.859828 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:00.859785    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d198391f-3e32-4881-a9ca-9a4d40d438b3-dshm\") pod \"autores-model-308137-kserve-5d658dbd79-xhfzl\" (UID: \"d198391f-3e32-4881-a9ca-9a4d40d438b3\") " pod="ai-tenant-e2e-autores-308137/autores-model-308137-kserve-5d658dbd79-xhfzl"
Sep 08 02:13:00.859828 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:00.859835    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pz5j\" (UniqueName: \"kubernetes.io/projected/d198391f-3e32-4881-a9ca-9a4d40d438b3-kube-api-access-7pz5j\") pod \"autores-model-308137-kserve-5d658dbd79-xhfzl\" (UID: \"d198391f-3e32-4881-a9ca-9a4d40d438b3\") " pod="ai-tenant-e2e-autores-308137/autores-model-308137-kserve-5d658dbd79-xhfzl"
Sep 08 02:13:00.860100 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:00.859970    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d198391f-3e32-4881-a9ca-9a4d40d438b3-home\") pod \"autores-model-308137-kserve-5d658dbd79-xhfzl\" (UID: \"d198391f-3e32-4881-a9ca-9a4d40d438b3\") " pod="ai-tenant-e2e-autores-308137/autores-model-308137-kserve-5d658dbd79-xhfzl"
Sep 08 02:13:00.860100 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:00.860007    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d198391f-3e32-4881-a9ca-9a4d40d438b3-tmp-dir\") pod \"autores-model-308137-kserve-5d658dbd79-xhfzl\" (UID: \"d198391f-3e32-4881-a9ca-9a4d40d438b3\") " pod="ai-tenant-e2e-autores-308137/autores-model-308137-kserve-5d658dbd79-xhfzl"
Sep 08 02:13:00.860100 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:00.860045    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d198391f-3e32-4881-a9ca-9a4d40d438b3-tls-certs\") pod \"autores-model-308137-kserve-5d658dbd79-xhfzl\" (UID: \"d198391f-3e32-4881-a9ca-9a4d40d438b3\") " pod="ai-tenant-e2e-autores-308137/autores-model-308137-kserve-5d658dbd79-xhfzl"
Sep 08 02:13:00.860100 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:00.860078    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d198391f-3e32-4881-a9ca-9a4d40d438b3-model-cache\") pod \"autores-model-308137-kserve-5d658dbd79-xhfzl\" (UID: \"d198391f-3e32-4881-a9ca-9a4d40d438b3\") " pod="ai-tenant-e2e-autores-308137/autores-model-308137-kserve-5d658dbd79-xhfzl"
Sep 08 02:13:00.960572 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:00.960485    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d198391f-3e32-4881-a9ca-9a4d40d438b3-home\") pod \"autores-model-308137-kserve-5d658dbd79-xhfzl\" (UID: \"d198391f-3e32-4881-a9ca-9a4d40d438b3\") " pod="ai-tenant-e2e-autores-308137/autores-model-308137-kserve-5d658dbd79-xhfzl"
Sep 08 02:13:00.960572 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:00.960523    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d198391f-3e32-4881-a9ca-9a4d40d438b3-tmp-dir\") pod \"autores-model-308137-kserve-5d658dbd79-xhfzl\" (UID: \"d198391f-3e32-4881-a9ca-9a4d40d438b3\") " pod="ai-tenant-e2e-autores-308137/autores-model-308137-kserve-5d658dbd79-xhfzl"
Sep 08 02:13:00.960572 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:00.960545    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d198391f-3e32-4881-a9ca-9a4d40d438b3-tls-certs\") pod \"autores-model-308137-kserve-5d658dbd79-xhfzl\" (UID: \"d198391f-3e32-4881-a9ca-9a4d40d438b3\") " pod="ai-tenant-e2e-autores-308137/autores-model-308137-kserve-5d658dbd79-xhfzl"
Sep 08 02:13:00.960572 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:00.960563    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d198391f-3e32-4881-a9ca-9a4d40d438b3-model-cache\") pod \"autores-model-308137-kserve-5d658dbd79-xhfzl\" (UID: \"d198391f-3e32-4881-a9ca-9a4d40d438b3\") " pod="ai-tenant-e2e-autores-308137/autores-model-308137-kserve-5d658dbd79-xhfzl"
Sep 08 02:13:00.960855 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:00.960613    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d198391f-3e32-4881-a9ca-9a4d40d438b3-dshm\") pod \"autores-model-308137-kserve-5d658dbd79-xhfzl\" (UID: \"d198391f-3e32-4881-a9ca-9a4d40d438b3\") " pod="ai-tenant-e2e-autores-308137/autores-model-308137-kserve-5d658dbd79-xhfzl"
Sep 08 02:13:00.960855 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:00.960646    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pz5j\" (UniqueName: \"kubernetes.io/projected/d198391f-3e32-4881-a9ca-9a4d40d438b3-kube-api-access-7pz5j\") pod \"autores-model-308137-kserve-5d658dbd79-xhfzl\" (UID: \"d198391f-3e32-4881-a9ca-9a4d40d438b3\") " pod="ai-tenant-e2e-autores-308137/autores-model-308137-kserve-5d658dbd79-xhfzl"
Sep 08 02:13:00.960998 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:00.960952    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d198391f-3e32-4881-a9ca-9a4d40d438b3-home\") pod \"autores-model-308137-kserve-5d658dbd79-xhfzl\" (UID: \"d198391f-3e32-4881-a9ca-9a4d40d438b3\") " pod="ai-tenant-e2e-autores-308137/autores-model-308137-kserve-5d658dbd79-xhfzl"
Sep 08 02:13:00.961062 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:00.960990    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d198391f-3e32-4881-a9ca-9a4d40d438b3-tmp-dir\") pod \"autores-model-308137-kserve-5d658dbd79-xhfzl\" (UID: \"d198391f-3e32-4881-a9ca-9a4d40d438b3\") " pod="ai-tenant-e2e-autores-308137/autores-model-308137-kserve-5d658dbd79-xhfzl"
Sep 08 02:13:00.961116 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:00.961064    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d198391f-3e32-4881-a9ca-9a4d40d438b3-model-cache\") pod \"autores-model-308137-kserve-5d658dbd79-xhfzl\" (UID: \"d198391f-3e32-4881-a9ca-9a4d40d438b3\") " pod="ai-tenant-e2e-autores-308137/autores-model-308137-kserve-5d658dbd79-xhfzl"
Sep 08 02:13:00.962970 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:00.962937    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d198391f-3e32-4881-a9ca-9a4d40d438b3-dshm\") pod \"autores-model-308137-kserve-5d658dbd79-xhfzl\" (UID: \"d198391f-3e32-4881-a9ca-9a4d40d438b3\") " pod="ai-tenant-e2e-autores-308137/autores-model-308137-kserve-5d658dbd79-xhfzl"
Sep 08 02:13:00.963321 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:00.963302    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d198391f-3e32-4881-a9ca-9a4d40d438b3-tls-certs\") pod \"autores-model-308137-kserve-5d658dbd79-xhfzl\" (UID: \"d198391f-3e32-4881-a9ca-9a4d40d438b3\") " pod="ai-tenant-e2e-autores-308137/autores-model-308137-kserve-5d658dbd79-xhfzl"
Sep 08 02:13:00.970042 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:00.970012    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pz5j\" (UniqueName: \"kubernetes.io/projected/d198391f-3e32-4881-a9ca-9a4d40d438b3-kube-api-access-7pz5j\") pod \"autores-model-308137-kserve-5d658dbd79-xhfzl\" (UID: \"d198391f-3e32-4881-a9ca-9a4d40d438b3\") " pod="ai-tenant-e2e-autores-308137/autores-model-308137-kserve-5d658dbd79-xhfzl"
Sep 08 02:13:01.018530 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:01.018490    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-308137/autores-model-308137-kserve-5d658dbd79-xhfzl"
Sep 08 02:13:01.292332 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:01.292216    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-308137/autores-model-308137-kserve-5d658dbd79-xhfzl"]
Sep 08 02:13:01.294757 ip-10-0-142-78 kubenswrapper[2547]: W0908 02:13:01.294732    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd198391f_3e32_4881_a9ca_9a4d40d438b3.slice/crio-06e686e004b64cadebfb105fbbca0f60f5689fbdf3ae47a18bbd7a1f03699843 WatchSource:0}: Error finding container 06e686e004b64cadebfb105fbbca0f60f5689fbdf3ae47a18bbd7a1f03699843: Status 404 returned error can't find the container with id 06e686e004b64cadebfb105fbbca0f60f5689fbdf3ae47a18bbd7a1f03699843
Sep 08 02:13:01.460401 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:01.460365    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-308137/autores-model-308137-kserve-5d658dbd79-xhfzl" event={"ID":"d198391f-3e32-4881-a9ca-9a4d40d438b3","Type":"ContainerStarted","Data":"90c5a6bacb07ffc148bcca6b8b3ddfe2957a8d528b2596b3c6367ad1aaf22b08"}
Sep 08 02:13:01.460603 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:01.460412    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-308137/autores-model-308137-kserve-5d658dbd79-xhfzl" event={"ID":"d198391f-3e32-4881-a9ca-9a4d40d438b3","Type":"ContainerStarted","Data":"06e686e004b64cadebfb105fbbca0f60f5689fbdf3ae47a18bbd7a1f03699843"}
Sep 08 02:13:01.480017 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:01.479951    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-308137/autores-model-308137-kserve-5d658dbd79-xhfzl" podStartSLOduration=1.47993481 podStartE2EDuration="1.47993481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:13:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:13:01.477820879 +0000 UTC m=+853.001595952" watchObservedRunningTime="2026-09-08 02:13:01.47993481 +0000 UTC m=+853.003709889"
Sep 08 02:13:11.019278 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:11.019207    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-308137/autores-model-308137-kserve-5d658dbd79-xhfzl"
Sep 08 02:13:11.019278 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:11.019289    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-308137/autores-model-308137-kserve-5d658dbd79-xhfzl"
Sep 08 02:13:11.031927 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:11.031892    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-308137/autores-model-308137-kserve-5d658dbd79-xhfzl"
Sep 08 02:13:11.515983 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:11.515944    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-308137/autores-model-308137-kserve-5d658dbd79-xhfzl"
Sep 08 02:13:19.204331 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:19.204295    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-d20bc0ce-kserve-6df5985594-r58x6"]
Sep 08 02:13:19.905634 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:19.905599    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-308137/autores-model-308137-kserve-5d658dbd79-xhfzl"]
Sep 08 02:13:21.122671 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:21.122628    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-d20bc0ce-kserve-6fd76b86d-7tqkx"]
Sep 08 02:13:30.714248 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:30.714207    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-308137-78b55f848f-85lm9"]
Sep 08 02:13:30.714719 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:30.714476    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-308137-78b55f848f-85lm9" podUID="a0db8cd0-b76b-47ef-b523-e784fe3dc4bf" containerName="payload-processing" containerID="cri-o://3aa7cd99bc65d631808f8609b27b140ba00e2d49bf0dfdfb44fc3dd672b7cbba" gracePeriod=30
Sep 08 02:13:30.723592 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:30.723565    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-308137-6f98dd6cd6-kjbgr"]
Sep 08 02:13:30.723985 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:30.723934    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-308137-6f98dd6cd6-kjbgr" podUID="4cfe1a5f-d9d4-47b0-8bfb-e7ad471b8c81" containerName="payload-pre-processing" containerID="cri-o://2f8cb1ce9262f2ade44e5d09fd33789520259cef445fc85497c3d731b6091e3c" gracePeriod=30
Sep 08 02:13:30.991754 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:30.991725    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-308137-78b55f848f-85lm9"
Sep 08 02:13:30.994493 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:30.994464    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-308137-6f98dd6cd6-kjbgr"
Sep 08 02:13:31.136633 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:31.136595    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m444h\" (UniqueName: \"kubernetes.io/projected/a0db8cd0-b76b-47ef-b523-e784fe3dc4bf-kube-api-access-m444h\") pod \"a0db8cd0-b76b-47ef-b523-e784fe3dc4bf\" (UID: \"a0db8cd0-b76b-47ef-b523-e784fe3dc4bf\") "
Sep 08 02:13:31.136827 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:31.136649    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0db8cd0-b76b-47ef-b523-e784fe3dc4bf-plugins-config-volume\") pod \"a0db8cd0-b76b-47ef-b523-e784fe3dc4bf\" (UID: \"a0db8cd0-b76b-47ef-b523-e784fe3dc4bf\") "
Sep 08 02:13:31.136827 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:31.136673    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jdkn5\" (UniqueName: \"kubernetes.io/projected/4cfe1a5f-d9d4-47b0-8bfb-e7ad471b8c81-kube-api-access-jdkn5\") pod \"4cfe1a5f-d9d4-47b0-8bfb-e7ad471b8c81\" (UID: \"4cfe1a5f-d9d4-47b0-8bfb-e7ad471b8c81\") "
Sep 08 02:13:31.136827 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:31.136747    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cfe1a5f-d9d4-47b0-8bfb-e7ad471b8c81-plugins-config-volume\") pod \"4cfe1a5f-d9d4-47b0-8bfb-e7ad471b8c81\" (UID: \"4cfe1a5f-d9d4-47b0-8bfb-e7ad471b8c81\") "
Sep 08 02:13:31.137075 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:31.137047    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0db8cd0-b76b-47ef-b523-e784fe3dc4bf-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a0db8cd0-b76b-47ef-b523-e784fe3dc4bf" (UID: "a0db8cd0-b76b-47ef-b523-e784fe3dc4bf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 02:13:31.137199 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:31.137173    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cfe1a5f-d9d4-47b0-8bfb-e7ad471b8c81-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4cfe1a5f-d9d4-47b0-8bfb-e7ad471b8c81" (UID: "4cfe1a5f-d9d4-47b0-8bfb-e7ad471b8c81"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 02:13:31.138746 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:31.138725    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cfe1a5f-d9d4-47b0-8bfb-e7ad471b8c81-kube-api-access-jdkn5" (OuterVolumeSpecName: "kube-api-access-jdkn5") pod "4cfe1a5f-d9d4-47b0-8bfb-e7ad471b8c81" (UID: "4cfe1a5f-d9d4-47b0-8bfb-e7ad471b8c81"). InnerVolumeSpecName "kube-api-access-jdkn5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 02:13:31.138796 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:31.138739    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0db8cd0-b76b-47ef-b523-e784fe3dc4bf-kube-api-access-m444h" (OuterVolumeSpecName: "kube-api-access-m444h") pod "a0db8cd0-b76b-47ef-b523-e784fe3dc4bf" (UID: "a0db8cd0-b76b-47ef-b523-e784fe3dc4bf"). InnerVolumeSpecName "kube-api-access-m444h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 02:13:31.238493 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:31.238388    2547 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cfe1a5f-d9d4-47b0-8bfb-e7ad471b8c81-plugins-config-volume\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:13:31.238493 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:31.238435    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m444h\" (UniqueName: \"kubernetes.io/projected/a0db8cd0-b76b-47ef-b523-e784fe3dc4bf-kube-api-access-m444h\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:13:31.238493 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:31.238444    2547 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0db8cd0-b76b-47ef-b523-e784fe3dc4bf-plugins-config-volume\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:13:31.238493 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:31.238453    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdkn5\" (UniqueName: \"kubernetes.io/projected/4cfe1a5f-d9d4-47b0-8bfb-e7ad471b8c81-kube-api-access-jdkn5\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:13:31.602328 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:31.602246    2547 generic.go:356] "Generic (PLEG): container finished" podID="4cfe1a5f-d9d4-47b0-8bfb-e7ad471b8c81" containerID="2f8cb1ce9262f2ade44e5d09fd33789520259cef445fc85497c3d731b6091e3c" exitCode=0
Sep 08 02:13:31.602328 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:31.602291    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-308137-6f98dd6cd6-kjbgr"
Sep 08 02:13:31.602534 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:31.602337    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-308137-6f98dd6cd6-kjbgr" event={"ID":"4cfe1a5f-d9d4-47b0-8bfb-e7ad471b8c81","Type":"ContainerDied","Data":"2f8cb1ce9262f2ade44e5d09fd33789520259cef445fc85497c3d731b6091e3c"}
Sep 08 02:13:31.602534 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:31.602393    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-308137-6f98dd6cd6-kjbgr" event={"ID":"4cfe1a5f-d9d4-47b0-8bfb-e7ad471b8c81","Type":"ContainerDied","Data":"43b1ff44a3ef471cca58297661e29961adff84eed0ba2690930f373c1b968388"}
Sep 08 02:13:31.602534 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:31.602412    2547 scope.go:117] "RemoveContainer" containerID="2f8cb1ce9262f2ade44e5d09fd33789520259cef445fc85497c3d731b6091e3c"
Sep 08 02:13:31.603741 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:31.603719    2547 generic.go:356] "Generic (PLEG): container finished" podID="a0db8cd0-b76b-47ef-b523-e784fe3dc4bf" containerID="3aa7cd99bc65d631808f8609b27b140ba00e2d49bf0dfdfb44fc3dd672b7cbba" exitCode=0
Sep 08 02:13:31.603855 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:31.603757    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-308137-78b55f848f-85lm9" event={"ID":"a0db8cd0-b76b-47ef-b523-e784fe3dc4bf","Type":"ContainerDied","Data":"3aa7cd99bc65d631808f8609b27b140ba00e2d49bf0dfdfb44fc3dd672b7cbba"}
Sep 08 02:13:31.603855 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:31.603779    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-308137-78b55f848f-85lm9"
Sep 08 02:13:31.603855 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:31.603808    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-308137-78b55f848f-85lm9" event={"ID":"a0db8cd0-b76b-47ef-b523-e784fe3dc4bf","Type":"ContainerDied","Data":"d7881a59a8efe4fc2bdd1e22433af1b4e0163ef8a33a5ea443933678e43e972e"}
Sep 08 02:13:31.613869 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:31.613844    2547 scope.go:117] "RemoveContainer" containerID="2f8cb1ce9262f2ade44e5d09fd33789520259cef445fc85497c3d731b6091e3c"
Sep 08 02:13:31.614485 ip-10-0-142-78 kubenswrapper[2547]: E0908 02:13:31.614444    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f8cb1ce9262f2ade44e5d09fd33789520259cef445fc85497c3d731b6091e3c\": container with ID starting with 2f8cb1ce9262f2ade44e5d09fd33789520259cef445fc85497c3d731b6091e3c not found: ID does not exist" containerID="2f8cb1ce9262f2ade44e5d09fd33789520259cef445fc85497c3d731b6091e3c"
Sep 08 02:13:31.614587 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:31.614484    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f8cb1ce9262f2ade44e5d09fd33789520259cef445fc85497c3d731b6091e3c"} err="failed to get container status \"2f8cb1ce9262f2ade44e5d09fd33789520259cef445fc85497c3d731b6091e3c\": rpc error: code = NotFound desc = could not find container \"2f8cb1ce9262f2ade44e5d09fd33789520259cef445fc85497c3d731b6091e3c\": container with ID starting with 2f8cb1ce9262f2ade44e5d09fd33789520259cef445fc85497c3d731b6091e3c not found: ID does not exist"
Sep 08 02:13:31.614587 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:31.614509    2547 scope.go:117] "RemoveContainer" containerID="3aa7cd99bc65d631808f8609b27b140ba00e2d49bf0dfdfb44fc3dd672b7cbba"
Sep 08 02:13:31.623961 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:31.623936    2547 scope.go:117] "RemoveContainer" containerID="3aa7cd99bc65d631808f8609b27b140ba00e2d49bf0dfdfb44fc3dd672b7cbba"
Sep 08 02:13:31.624285 ip-10-0-142-78 kubenswrapper[2547]: E0908 02:13:31.624251    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3aa7cd99bc65d631808f8609b27b140ba00e2d49bf0dfdfb44fc3dd672b7cbba\": container with ID starting with 3aa7cd99bc65d631808f8609b27b140ba00e2d49bf0dfdfb44fc3dd672b7cbba not found: ID does not exist" containerID="3aa7cd99bc65d631808f8609b27b140ba00e2d49bf0dfdfb44fc3dd672b7cbba"
Sep 08 02:13:31.624396 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:31.624283    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3aa7cd99bc65d631808f8609b27b140ba00e2d49bf0dfdfb44fc3dd672b7cbba"} err="failed to get container status \"3aa7cd99bc65d631808f8609b27b140ba00e2d49bf0dfdfb44fc3dd672b7cbba\": rpc error: code = NotFound desc = could not find container \"3aa7cd99bc65d631808f8609b27b140ba00e2d49bf0dfdfb44fc3dd672b7cbba\": container with ID starting with 3aa7cd99bc65d631808f8609b27b140ba00e2d49bf0dfdfb44fc3dd672b7cbba not found: ID does not exist"
Sep 08 02:13:31.627479 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:31.627457    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-308137-6f98dd6cd6-kjbgr"]
Sep 08 02:13:31.631784 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:31.631760    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-308137-6f98dd6cd6-kjbgr"]
Sep 08 02:13:31.641877 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:31.641841    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-308137-78b55f848f-85lm9"]
Sep 08 02:13:31.643416 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:31.643395    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-308137-78b55f848f-85lm9"]
Sep 08 02:13:32.994451 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:32.994414    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4cfe1a5f-d9d4-47b0-8bfb-e7ad471b8c81" path="/var/lib/kubelet/pods/4cfe1a5f-d9d4-47b0-8bfb-e7ad471b8c81/volumes"
Sep 08 02:13:32.994952 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:32.994937    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0db8cd0-b76b-47ef-b523-e784fe3dc4bf" path="/var/lib/kubelet/pods/a0db8cd0-b76b-47ef-b523-e784fe3dc4bf/volumes"
Sep 08 02:13:34.225846 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:34.225798    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-b-d20bc0ce-kserve-6df5985594-r58x6" podUID="f142f130-eceb-4f5e-b35e-a736c3735584" containerName="main" containerID="cri-o://ce7961ffdb26c3bce858e12af6d13dd6206462ba8aed4627da753bb936db3cf5" gracePeriod=45
Sep 08 02:13:34.471563 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:34.471536    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-d20bc0ce-kserve-6df5985594-r58x6"
Sep 08 02:13:34.567887 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:34.567793    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f142f130-eceb-4f5e-b35e-a736c3735584-dshm\") pod \"f142f130-eceb-4f5e-b35e-a736c3735584\" (UID: \"f142f130-eceb-4f5e-b35e-a736c3735584\") "
Sep 08 02:13:34.567887 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:34.567833    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f142f130-eceb-4f5e-b35e-a736c3735584-home\") pod \"f142f130-eceb-4f5e-b35e-a736c3735584\" (UID: \"f142f130-eceb-4f5e-b35e-a736c3735584\") "
Sep 08 02:13:34.568120 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:34.567930    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f142f130-eceb-4f5e-b35e-a736c3735584-tls-certs\") pod \"f142f130-eceb-4f5e-b35e-a736c3735584\" (UID: \"f142f130-eceb-4f5e-b35e-a736c3735584\") "
Sep 08 02:13:34.568120 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:34.567977    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f142f130-eceb-4f5e-b35e-a736c3735584-tmp-dir\") pod \"f142f130-eceb-4f5e-b35e-a736c3735584\" (UID: \"f142f130-eceb-4f5e-b35e-a736c3735584\") "
Sep 08 02:13:34.568120 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:34.568005    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f142f130-eceb-4f5e-b35e-a736c3735584-model-cache\") pod \"f142f130-eceb-4f5e-b35e-a736c3735584\" (UID: \"f142f130-eceb-4f5e-b35e-a736c3735584\") "
Sep 08 02:13:34.568120 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:34.568043    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7qgvg\" (UniqueName: \"kubernetes.io/projected/f142f130-eceb-4f5e-b35e-a736c3735584-kube-api-access-7qgvg\") pod \"f142f130-eceb-4f5e-b35e-a736c3735584\" (UID: \"f142f130-eceb-4f5e-b35e-a736c3735584\") "
Sep 08 02:13:34.568348 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:34.568232    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f142f130-eceb-4f5e-b35e-a736c3735584-home" (OuterVolumeSpecName: "home") pod "f142f130-eceb-4f5e-b35e-a736c3735584" (UID: "f142f130-eceb-4f5e-b35e-a736c3735584"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 02:13:34.568348 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:34.568273    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f142f130-eceb-4f5e-b35e-a736c3735584-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f142f130-eceb-4f5e-b35e-a736c3735584" (UID: "f142f130-eceb-4f5e-b35e-a736c3735584"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 02:13:34.568348 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:34.568292    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f142f130-eceb-4f5e-b35e-a736c3735584-model-cache" (OuterVolumeSpecName: "model-cache") pod "f142f130-eceb-4f5e-b35e-a736c3735584" (UID: "f142f130-eceb-4f5e-b35e-a736c3735584"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 02:13:34.568348 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:34.568340    2547 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f142f130-eceb-4f5e-b35e-a736c3735584-home\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:13:34.570192 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:34.570163    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f142f130-eceb-4f5e-b35e-a736c3735584-kube-api-access-7qgvg" (OuterVolumeSpecName: "kube-api-access-7qgvg") pod "f142f130-eceb-4f5e-b35e-a736c3735584" (UID: "f142f130-eceb-4f5e-b35e-a736c3735584"). InnerVolumeSpecName "kube-api-access-7qgvg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 02:13:34.570308 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:34.570196    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f142f130-eceb-4f5e-b35e-a736c3735584-dshm" (OuterVolumeSpecName: "dshm") pod "f142f130-eceb-4f5e-b35e-a736c3735584" (UID: "f142f130-eceb-4f5e-b35e-a736c3735584"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 02:13:34.570308 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:34.570205    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f142f130-eceb-4f5e-b35e-a736c3735584-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f142f130-eceb-4f5e-b35e-a736c3735584" (UID: "f142f130-eceb-4f5e-b35e-a736c3735584"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 02:13:34.620336 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:34.620300    2547 generic.go:356] "Generic (PLEG): container finished" podID="f142f130-eceb-4f5e-b35e-a736c3735584" containerID="ce7961ffdb26c3bce858e12af6d13dd6206462ba8aed4627da753bb936db3cf5" exitCode=0
Sep 08 02:13:34.620510 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:34.620347    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-d20bc0ce-kserve-6df5985594-r58x6"
Sep 08 02:13:34.620510 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:34.620392    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-d20bc0ce-kserve-6df5985594-r58x6" event={"ID":"f142f130-eceb-4f5e-b35e-a736c3735584","Type":"ContainerDied","Data":"ce7961ffdb26c3bce858e12af6d13dd6206462ba8aed4627da753bb936db3cf5"}
Sep 08 02:13:34.620510 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:34.620430    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-d20bc0ce-kserve-6df5985594-r58x6" event={"ID":"f142f130-eceb-4f5e-b35e-a736c3735584","Type":"ContainerDied","Data":"1ba7d8c1e474db4f1683538a9770a75b5ddc8bca470a8edabd213cd26306f6b8"}
Sep 08 02:13:34.620510 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:34.620448    2547 scope.go:117] "RemoveContainer" containerID="ce7961ffdb26c3bce858e12af6d13dd6206462ba8aed4627da753bb936db3cf5"
Sep 08 02:13:34.628880 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:34.628863    2547 scope.go:117] "RemoveContainer" containerID="ce7961ffdb26c3bce858e12af6d13dd6206462ba8aed4627da753bb936db3cf5"
Sep 08 02:13:34.629199 ip-10-0-142-78 kubenswrapper[2547]: E0908 02:13:34.629178    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce7961ffdb26c3bce858e12af6d13dd6206462ba8aed4627da753bb936db3cf5\": container with ID starting with ce7961ffdb26c3bce858e12af6d13dd6206462ba8aed4627da753bb936db3cf5 not found: ID does not exist" containerID="ce7961ffdb26c3bce858e12af6d13dd6206462ba8aed4627da753bb936db3cf5"
Sep 08 02:13:34.629267 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:34.629209    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce7961ffdb26c3bce858e12af6d13dd6206462ba8aed4627da753bb936db3cf5"} err="failed to get container status \"ce7961ffdb26c3bce858e12af6d13dd6206462ba8aed4627da753bb936db3cf5\": rpc error: code = NotFound desc = could not find container \"ce7961ffdb26c3bce858e12af6d13dd6206462ba8aed4627da753bb936db3cf5\": container with ID starting with ce7961ffdb26c3bce858e12af6d13dd6206462ba8aed4627da753bb936db3cf5 not found: ID does not exist"
Sep 08 02:13:34.640029 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:34.639993    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-d20bc0ce-kserve-6df5985594-r58x6"]
Sep 08 02:13:34.641831 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:34.641811    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-b-d20bc0ce-kserve-6df5985594-r58x6"]
Sep 08 02:13:34.669575 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:34.669542    2547 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f142f130-eceb-4f5e-b35e-a736c3735584-tls-certs\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:13:34.669575 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:34.669572    2547 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f142f130-eceb-4f5e-b35e-a736c3735584-tmp-dir\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:13:34.669575 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:34.669581    2547 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f142f130-eceb-4f5e-b35e-a736c3735584-model-cache\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:13:34.669753 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:34.669589    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qgvg\" (UniqueName: \"kubernetes.io/projected/f142f130-eceb-4f5e-b35e-a736c3735584-kube-api-access-7qgvg\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:13:34.669753 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:34.669599    2547 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f142f130-eceb-4f5e-b35e-a736c3735584-dshm\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:13:34.925882 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:34.925845    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-308137/autores-model-308137-kserve-5d658dbd79-xhfzl" podUID="d198391f-3e32-4881-a9ca-9a4d40d438b3" containerName="main" containerID="cri-o://90c5a6bacb07ffc148bcca6b8b3ddfe2957a8d528b2596b3c6367ad1aaf22b08" gracePeriod=45
Sep 08 02:13:34.994888 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:34.994847    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f142f130-eceb-4f5e-b35e-a736c3735584" path="/var/lib/kubelet/pods/f142f130-eceb-4f5e-b35e-a736c3735584/volumes"
Sep 08 02:13:35.158281 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:35.158256    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-308137/autores-model-308137-kserve-5d658dbd79-xhfzl"
Sep 08 02:13:35.274901 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:35.274859    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d198391f-3e32-4881-a9ca-9a4d40d438b3-model-cache\") pod \"d198391f-3e32-4881-a9ca-9a4d40d438b3\" (UID: \"d198391f-3e32-4881-a9ca-9a4d40d438b3\") "
Sep 08 02:13:35.275351 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:35.274923    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7pz5j\" (UniqueName: \"kubernetes.io/projected/d198391f-3e32-4881-a9ca-9a4d40d438b3-kube-api-access-7pz5j\") pod \"d198391f-3e32-4881-a9ca-9a4d40d438b3\" (UID: \"d198391f-3e32-4881-a9ca-9a4d40d438b3\") "
Sep 08 02:13:35.275351 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:35.274962    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d198391f-3e32-4881-a9ca-9a4d40d438b3-dshm\") pod \"d198391f-3e32-4881-a9ca-9a4d40d438b3\" (UID: \"d198391f-3e32-4881-a9ca-9a4d40d438b3\") "
Sep 08 02:13:35.275351 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:35.274978    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d198391f-3e32-4881-a9ca-9a4d40d438b3-tmp-dir\") pod \"d198391f-3e32-4881-a9ca-9a4d40d438b3\" (UID: \"d198391f-3e32-4881-a9ca-9a4d40d438b3\") "
Sep 08 02:13:35.275351 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:35.274996    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d198391f-3e32-4881-a9ca-9a4d40d438b3-tls-certs\") pod \"d198391f-3e32-4881-a9ca-9a4d40d438b3\" (UID: \"d198391f-3e32-4881-a9ca-9a4d40d438b3\") "
Sep 08 02:13:35.275351 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:35.275050    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d198391f-3e32-4881-a9ca-9a4d40d438b3-home\") pod \"d198391f-3e32-4881-a9ca-9a4d40d438b3\" (UID: \"d198391f-3e32-4881-a9ca-9a4d40d438b3\") "
Sep 08 02:13:35.275351 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:35.275218    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d198391f-3e32-4881-a9ca-9a4d40d438b3-model-cache" (OuterVolumeSpecName: "model-cache") pod "d198391f-3e32-4881-a9ca-9a4d40d438b3" (UID: "d198391f-3e32-4881-a9ca-9a4d40d438b3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 02:13:35.275351 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:35.275291    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d198391f-3e32-4881-a9ca-9a4d40d438b3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d198391f-3e32-4881-a9ca-9a4d40d438b3" (UID: "d198391f-3e32-4881-a9ca-9a4d40d438b3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 02:13:35.275351 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:35.275331    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d198391f-3e32-4881-a9ca-9a4d40d438b3-home" (OuterVolumeSpecName: "home") pod "d198391f-3e32-4881-a9ca-9a4d40d438b3" (UID: "d198391f-3e32-4881-a9ca-9a4d40d438b3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 02:13:35.275688 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:35.275388    2547 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d198391f-3e32-4881-a9ca-9a4d40d438b3-model-cache\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:13:35.275688 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:35.275400    2547 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d198391f-3e32-4881-a9ca-9a4d40d438b3-tmp-dir\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:13:35.275688 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:35.275408    2547 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d198391f-3e32-4881-a9ca-9a4d40d438b3-home\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:13:35.277362 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:35.277338    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d198391f-3e32-4881-a9ca-9a4d40d438b3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d198391f-3e32-4881-a9ca-9a4d40d438b3" (UID: "d198391f-3e32-4881-a9ca-9a4d40d438b3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 02:13:35.277430 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:35.277356    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d198391f-3e32-4881-a9ca-9a4d40d438b3-dshm" (OuterVolumeSpecName: "dshm") pod "d198391f-3e32-4881-a9ca-9a4d40d438b3" (UID: "d198391f-3e32-4881-a9ca-9a4d40d438b3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 02:13:35.277430 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:35.277421    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d198391f-3e32-4881-a9ca-9a4d40d438b3-kube-api-access-7pz5j" (OuterVolumeSpecName: "kube-api-access-7pz5j") pod "d198391f-3e32-4881-a9ca-9a4d40d438b3" (UID: "d198391f-3e32-4881-a9ca-9a4d40d438b3"). InnerVolumeSpecName "kube-api-access-7pz5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 02:13:35.376843 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:35.376792    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pz5j\" (UniqueName: \"kubernetes.io/projected/d198391f-3e32-4881-a9ca-9a4d40d438b3-kube-api-access-7pz5j\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:13:35.376843 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:35.376836    2547 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d198391f-3e32-4881-a9ca-9a4d40d438b3-dshm\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:13:35.376843 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:35.376846    2547 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d198391f-3e32-4881-a9ca-9a4d40d438b3-tls-certs\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:13:35.627177 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:35.627062    2547 generic.go:356] "Generic (PLEG): container finished" podID="d198391f-3e32-4881-a9ca-9a4d40d438b3" containerID="90c5a6bacb07ffc148bcca6b8b3ddfe2957a8d528b2596b3c6367ad1aaf22b08" exitCode=0
Sep 08 02:13:35.627177 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:35.627109    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-308137/autores-model-308137-kserve-5d658dbd79-xhfzl"
Sep 08 02:13:35.627177 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:35.627111    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-308137/autores-model-308137-kserve-5d658dbd79-xhfzl" event={"ID":"d198391f-3e32-4881-a9ca-9a4d40d438b3","Type":"ContainerDied","Data":"90c5a6bacb07ffc148bcca6b8b3ddfe2957a8d528b2596b3c6367ad1aaf22b08"}
Sep 08 02:13:35.627177 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:35.627176    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-308137/autores-model-308137-kserve-5d658dbd79-xhfzl" event={"ID":"d198391f-3e32-4881-a9ca-9a4d40d438b3","Type":"ContainerDied","Data":"06e686e004b64cadebfb105fbbca0f60f5689fbdf3ae47a18bbd7a1f03699843"}
Sep 08 02:13:35.627482 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:35.627200    2547 scope.go:117] "RemoveContainer" containerID="90c5a6bacb07ffc148bcca6b8b3ddfe2957a8d528b2596b3c6367ad1aaf22b08"
Sep 08 02:13:35.635765 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:35.635745    2547 scope.go:117] "RemoveContainer" containerID="90c5a6bacb07ffc148bcca6b8b3ddfe2957a8d528b2596b3c6367ad1aaf22b08"
Sep 08 02:13:35.636027 ip-10-0-142-78 kubenswrapper[2547]: E0908 02:13:35.636008    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90c5a6bacb07ffc148bcca6b8b3ddfe2957a8d528b2596b3c6367ad1aaf22b08\": container with ID starting with 90c5a6bacb07ffc148bcca6b8b3ddfe2957a8d528b2596b3c6367ad1aaf22b08 not found: ID does not exist" containerID="90c5a6bacb07ffc148bcca6b8b3ddfe2957a8d528b2596b3c6367ad1aaf22b08"
Sep 08 02:13:35.636074 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:35.636035    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90c5a6bacb07ffc148bcca6b8b3ddfe2957a8d528b2596b3c6367ad1aaf22b08"} err="failed to get container status \"90c5a6bacb07ffc148bcca6b8b3ddfe2957a8d528b2596b3c6367ad1aaf22b08\": rpc error: code = NotFound desc = could not find container \"90c5a6bacb07ffc148bcca6b8b3ddfe2957a8d528b2596b3c6367ad1aaf22b08\": container with ID starting with 90c5a6bacb07ffc148bcca6b8b3ddfe2957a8d528b2596b3c6367ad1aaf22b08 not found: ID does not exist"
Sep 08 02:13:35.646447 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:35.646423    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-308137/autores-model-308137-kserve-5d658dbd79-xhfzl"]
Sep 08 02:13:35.649927 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:35.649905    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-308137/autores-model-308137-kserve-5d658dbd79-xhfzl"]
Sep 08 02:13:36.142581 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:36.142536    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-a-d20bc0ce-kserve-6fd76b86d-7tqkx" podUID="fc319422-532a-49e2-9aea-733525d4461f" containerName="main" containerID="cri-o://b97d5876801b3650476c95e4dd1f6b3f7075474a789577c42728bc1b45b24c26" gracePeriod=45
Sep 08 02:13:36.394895 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:36.394827    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-d20bc0ce-kserve-6fd76b86d-7tqkx"
Sep 08 02:13:36.487318 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:36.487277    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc319422-532a-49e2-9aea-733525d4461f-tls-certs\") pod \"fc319422-532a-49e2-9aea-733525d4461f\" (UID: \"fc319422-532a-49e2-9aea-733525d4461f\") "
Sep 08 02:13:36.487458 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:36.487332    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc319422-532a-49e2-9aea-733525d4461f-dshm\") pod \"fc319422-532a-49e2-9aea-733525d4461f\" (UID: \"fc319422-532a-49e2-9aea-733525d4461f\") "
Sep 08 02:13:36.487458 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:36.487381    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc319422-532a-49e2-9aea-733525d4461f-model-cache\") pod \"fc319422-532a-49e2-9aea-733525d4461f\" (UID: \"fc319422-532a-49e2-9aea-733525d4461f\") "
Sep 08 02:13:36.487458 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:36.487444    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc319422-532a-49e2-9aea-733525d4461f-tmp-dir\") pod \"fc319422-532a-49e2-9aea-733525d4461f\" (UID: \"fc319422-532a-49e2-9aea-733525d4461f\") "
Sep 08 02:13:36.487608 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:36.487462    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-45kr4\" (UniqueName: \"kubernetes.io/projected/fc319422-532a-49e2-9aea-733525d4461f-kube-api-access-45kr4\") pod \"fc319422-532a-49e2-9aea-733525d4461f\" (UID: \"fc319422-532a-49e2-9aea-733525d4461f\") "
Sep 08 02:13:36.487608 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:36.487494    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc319422-532a-49e2-9aea-733525d4461f-home\") pod \"fc319422-532a-49e2-9aea-733525d4461f\" (UID: \"fc319422-532a-49e2-9aea-733525d4461f\") "
Sep 08 02:13:36.487765 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:36.487739    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc319422-532a-49e2-9aea-733525d4461f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fc319422-532a-49e2-9aea-733525d4461f" (UID: "fc319422-532a-49e2-9aea-733525d4461f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 02:13:36.487879 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:36.487755    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc319422-532a-49e2-9aea-733525d4461f-model-cache" (OuterVolumeSpecName: "model-cache") pod "fc319422-532a-49e2-9aea-733525d4461f" (UID: "fc319422-532a-49e2-9aea-733525d4461f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 02:13:36.487879 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:36.487829    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc319422-532a-49e2-9aea-733525d4461f-home" (OuterVolumeSpecName: "home") pod "fc319422-532a-49e2-9aea-733525d4461f" (UID: "fc319422-532a-49e2-9aea-733525d4461f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 02:13:36.489483 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:36.489458    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc319422-532a-49e2-9aea-733525d4461f-dshm" (OuterVolumeSpecName: "dshm") pod "fc319422-532a-49e2-9aea-733525d4461f" (UID: "fc319422-532a-49e2-9aea-733525d4461f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 02:13:36.489594 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:36.489525    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc319422-532a-49e2-9aea-733525d4461f-kube-api-access-45kr4" (OuterVolumeSpecName: "kube-api-access-45kr4") pod "fc319422-532a-49e2-9aea-733525d4461f" (UID: "fc319422-532a-49e2-9aea-733525d4461f"). InnerVolumeSpecName "kube-api-access-45kr4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 02:13:36.489594 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:36.489549    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc319422-532a-49e2-9aea-733525d4461f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fc319422-532a-49e2-9aea-733525d4461f" (UID: "fc319422-532a-49e2-9aea-733525d4461f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 02:13:36.588227 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:36.588182    2547 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc319422-532a-49e2-9aea-733525d4461f-model-cache\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:13:36.588227 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:36.588218    2547 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc319422-532a-49e2-9aea-733525d4461f-tmp-dir\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:13:36.588227 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:36.588228    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45kr4\" (UniqueName: \"kubernetes.io/projected/fc319422-532a-49e2-9aea-733525d4461f-kube-api-access-45kr4\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:13:36.588227 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:36.588240    2547 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc319422-532a-49e2-9aea-733525d4461f-home\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:13:36.588480 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:36.588249    2547 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc319422-532a-49e2-9aea-733525d4461f-tls-certs\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:13:36.588480 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:36.588257    2547 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc319422-532a-49e2-9aea-733525d4461f-dshm\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:13:36.633866 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:36.633828    2547 generic.go:356] "Generic (PLEG): container finished" podID="fc319422-532a-49e2-9aea-733525d4461f" containerID="b97d5876801b3650476c95e4dd1f6b3f7075474a789577c42728bc1b45b24c26" exitCode=0
Sep 08 02:13:36.633866 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:36.633872    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-d20bc0ce-kserve-6fd76b86d-7tqkx" event={"ID":"fc319422-532a-49e2-9aea-733525d4461f","Type":"ContainerDied","Data":"b97d5876801b3650476c95e4dd1f6b3f7075474a789577c42728bc1b45b24c26"}
Sep 08 02:13:36.634078 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:36.633903    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-d20bc0ce-kserve-6fd76b86d-7tqkx" event={"ID":"fc319422-532a-49e2-9aea-733525d4461f","Type":"ContainerDied","Data":"d7e773a979d74367c002b229b701302f4dd1b23cf527a9441f0ecf18c70bf386"}
Sep 08 02:13:36.634078 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:36.633920    2547 scope.go:117] "RemoveContainer" containerID="b97d5876801b3650476c95e4dd1f6b3f7075474a789577c42728bc1b45b24c26"
Sep 08 02:13:36.634078 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:36.633877    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-d20bc0ce-kserve-6fd76b86d-7tqkx"
Sep 08 02:13:36.642735 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:36.642713    2547 scope.go:117] "RemoveContainer" containerID="b97d5876801b3650476c95e4dd1f6b3f7075474a789577c42728bc1b45b24c26"
Sep 08 02:13:36.643004 ip-10-0-142-78 kubenswrapper[2547]: E0908 02:13:36.642986    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b97d5876801b3650476c95e4dd1f6b3f7075474a789577c42728bc1b45b24c26\": container with ID starting with b97d5876801b3650476c95e4dd1f6b3f7075474a789577c42728bc1b45b24c26 not found: ID does not exist" containerID="b97d5876801b3650476c95e4dd1f6b3f7075474a789577c42728bc1b45b24c26"
Sep 08 02:13:36.643059 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:36.643010    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b97d5876801b3650476c95e4dd1f6b3f7075474a789577c42728bc1b45b24c26"} err="failed to get container status \"b97d5876801b3650476c95e4dd1f6b3f7075474a789577c42728bc1b45b24c26\": rpc error: code = NotFound desc = could not find container \"b97d5876801b3650476c95e4dd1f6b3f7075474a789577c42728bc1b45b24c26\": container with ID starting with b97d5876801b3650476c95e4dd1f6b3f7075474a789577c42728bc1b45b24c26 not found: ID does not exist"
Sep 08 02:13:36.657373 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:36.657303    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-d20bc0ce-kserve-6fd76b86d-7tqkx"]
Sep 08 02:13:36.663083 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:36.663058    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-a-d20bc0ce-kserve-6fd76b86d-7tqkx"]
Sep 08 02:13:36.994849 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:36.994768    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d198391f-3e32-4881-a9ca-9a4d40d438b3" path="/var/lib/kubelet/pods/d198391f-3e32-4881-a9ca-9a4d40d438b3/volumes"
Sep 08 02:13:36.995155 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:36.995128    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc319422-532a-49e2-9aea-733525d4461f" path="/var/lib/kubelet/pods/fc319422-532a-49e2-9aea-733525d4461f/volumes"
Sep 08 02:13:42.568449 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:42.568228    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-173970-54574d8fd-swc7f"]
Sep 08 02:13:42.568820 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:42.568602    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f142f130-eceb-4f5e-b35e-a736c3735584" containerName="main"
Sep 08 02:13:42.568820 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:42.568616    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="f142f130-eceb-4f5e-b35e-a736c3735584" containerName="main"
Sep 08 02:13:42.568820 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:42.568633    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cfe1a5f-d9d4-47b0-8bfb-e7ad471b8c81" containerName="payload-pre-processing"
Sep 08 02:13:42.568820 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:42.568639    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cfe1a5f-d9d4-47b0-8bfb-e7ad471b8c81" containerName="payload-pre-processing"
Sep 08 02:13:42.568820 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:42.568648    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d198391f-3e32-4881-a9ca-9a4d40d438b3" containerName="main"
Sep 08 02:13:42.568820 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:42.568653    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="d198391f-3e32-4881-a9ca-9a4d40d438b3" containerName="main"
Sep 08 02:13:42.568820 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:42.568667    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0db8cd0-b76b-47ef-b523-e784fe3dc4bf" containerName="payload-processing"
Sep 08 02:13:42.568820 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:42.568675    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0db8cd0-b76b-47ef-b523-e784fe3dc4bf" containerName="payload-processing"
Sep 08 02:13:42.568820 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:42.568688    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc319422-532a-49e2-9aea-733525d4461f" containerName="main"
Sep 08 02:13:42.568820 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:42.568694    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc319422-532a-49e2-9aea-733525d4461f" containerName="main"
Sep 08 02:13:42.568820 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:42.568740    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="f142f130-eceb-4f5e-b35e-a736c3735584" containerName="main"
Sep 08 02:13:42.568820 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:42.568747    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cfe1a5f-d9d4-47b0-8bfb-e7ad471b8c81" containerName="payload-pre-processing"
Sep 08 02:13:42.568820 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:42.568755    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0db8cd0-b76b-47ef-b523-e784fe3dc4bf" containerName="payload-processing"
Sep 08 02:13:42.568820 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:42.568760    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc319422-532a-49e2-9aea-733525d4461f" containerName="main"
Sep 08 02:13:42.568820 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:42.568765    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="d198391f-3e32-4881-a9ca-9a4d40d438b3" containerName="main"
Sep 08 02:13:42.572883 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:42.572866    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-173970-54574d8fd-swc7f"
Sep 08 02:13:42.575261 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:42.575238    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-173970\""
Sep 08 02:13:42.575370 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:42.575304    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-173970-dockercfg-5vqt2\""
Sep 08 02:13:42.581363 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:42.581340    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-173970-54574d8fd-swc7f"]
Sep 08 02:13:42.617758 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:42.617561    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-173970-7dc59f76f7-8tbmv"]
Sep 08 02:13:42.621489 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:42.621459    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-173970-7dc59f76f7-8tbmv"
Sep 08 02:13:42.622977 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:42.622952    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-173970-7dc59f76f7-8tbmv"]
Sep 08 02:13:42.743537 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:42.743500    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2v7nb\" (UniqueName: \"kubernetes.io/projected/874fb7ce-f6f6-46df-8516-86d81c64e69c-kube-api-access-2v7nb\") pod \"payload-pre-processing-e2e-explicit-173970-54574d8fd-swc7f\" (UID: \"874fb7ce-f6f6-46df-8516-86d81c64e69c\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-173970-54574d8fd-swc7f"
Sep 08 02:13:42.743537 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:42.743541    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-529v8\" (UniqueName: \"kubernetes.io/projected/fec1575a-b738-45df-824f-16c83c21d025-kube-api-access-529v8\") pod \"payload-processing-e2e-explicit-173970-7dc59f76f7-8tbmv\" (UID: \"fec1575a-b738-45df-824f-16c83c21d025\") " pod="openshift-ingress/payload-processing-e2e-explicit-173970-7dc59f76f7-8tbmv"
Sep 08 02:13:42.743793 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:42.743569    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/874fb7ce-f6f6-46df-8516-86d81c64e69c-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-173970-54574d8fd-swc7f\" (UID: \"874fb7ce-f6f6-46df-8516-86d81c64e69c\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-173970-54574d8fd-swc7f"
Sep 08 02:13:42.743793 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:42.743690    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fec1575a-b738-45df-824f-16c83c21d025-plugins-config-volume\") pod \"payload-processing-e2e-explicit-173970-7dc59f76f7-8tbmv\" (UID: \"fec1575a-b738-45df-824f-16c83c21d025\") " pod="openshift-ingress/payload-processing-e2e-explicit-173970-7dc59f76f7-8tbmv"
Sep 08 02:13:42.845023 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:42.844922    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2v7nb\" (UniqueName: \"kubernetes.io/projected/874fb7ce-f6f6-46df-8516-86d81c64e69c-kube-api-access-2v7nb\") pod \"payload-pre-processing-e2e-explicit-173970-54574d8fd-swc7f\" (UID: \"874fb7ce-f6f6-46df-8516-86d81c64e69c\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-173970-54574d8fd-swc7f"
Sep 08 02:13:42.845023 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:42.844978    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-529v8\" (UniqueName: \"kubernetes.io/projected/fec1575a-b738-45df-824f-16c83c21d025-kube-api-access-529v8\") pod \"payload-processing-e2e-explicit-173970-7dc59f76f7-8tbmv\" (UID: \"fec1575a-b738-45df-824f-16c83c21d025\") " pod="openshift-ingress/payload-processing-e2e-explicit-173970-7dc59f76f7-8tbmv"
Sep 08 02:13:42.845023 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:42.845003    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/874fb7ce-f6f6-46df-8516-86d81c64e69c-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-173970-54574d8fd-swc7f\" (UID: \"874fb7ce-f6f6-46df-8516-86d81c64e69c\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-173970-54574d8fd-swc7f"
Sep 08 02:13:42.845278 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:42.845039    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fec1575a-b738-45df-824f-16c83c21d025-plugins-config-volume\") pod \"payload-processing-e2e-explicit-173970-7dc59f76f7-8tbmv\" (UID: \"fec1575a-b738-45df-824f-16c83c21d025\") " pod="openshift-ingress/payload-processing-e2e-explicit-173970-7dc59f76f7-8tbmv"
Sep 08 02:13:42.845957 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:42.845932    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fec1575a-b738-45df-824f-16c83c21d025-plugins-config-volume\") pod \"payload-processing-e2e-explicit-173970-7dc59f76f7-8tbmv\" (UID: \"fec1575a-b738-45df-824f-16c83c21d025\") " pod="openshift-ingress/payload-processing-e2e-explicit-173970-7dc59f76f7-8tbmv"
Sep 08 02:13:42.845992 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:42.845932    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/874fb7ce-f6f6-46df-8516-86d81c64e69c-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-173970-54574d8fd-swc7f\" (UID: \"874fb7ce-f6f6-46df-8516-86d81c64e69c\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-173970-54574d8fd-swc7f"
Sep 08 02:13:42.852960 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:42.852933    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-529v8\" (UniqueName: \"kubernetes.io/projected/fec1575a-b738-45df-824f-16c83c21d025-kube-api-access-529v8\") pod \"payload-processing-e2e-explicit-173970-7dc59f76f7-8tbmv\" (UID: \"fec1575a-b738-45df-824f-16c83c21d025\") " pod="openshift-ingress/payload-processing-e2e-explicit-173970-7dc59f76f7-8tbmv"
Sep 08 02:13:42.853063 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:42.852934    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2v7nb\" (UniqueName: \"kubernetes.io/projected/874fb7ce-f6f6-46df-8516-86d81c64e69c-kube-api-access-2v7nb\") pod \"payload-pre-processing-e2e-explicit-173970-54574d8fd-swc7f\" (UID: \"874fb7ce-f6f6-46df-8516-86d81c64e69c\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-173970-54574d8fd-swc7f"
Sep 08 02:13:42.883928 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:42.883890    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-173970-54574d8fd-swc7f"
Sep 08 02:13:42.932095 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:42.932063    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-173970-7dc59f76f7-8tbmv"
Sep 08 02:13:43.156018 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:43.155994    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-173970-54574d8fd-swc7f"]
Sep 08 02:13:43.158106 ip-10-0-142-78 kubenswrapper[2547]: W0908 02:13:43.158082    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod874fb7ce_f6f6_46df_8516_86d81c64e69c.slice/crio-d0203d0d025d69fbe6380eafb01caa0af5f9b9e7c24783a73b6747257fe51e48 WatchSource:0}: Error finding container d0203d0d025d69fbe6380eafb01caa0af5f9b9e7c24783a73b6747257fe51e48: Status 404 returned error can't find the container with id d0203d0d025d69fbe6380eafb01caa0af5f9b9e7c24783a73b6747257fe51e48
Sep 08 02:13:43.213180 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:43.213052    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-173970-7dc59f76f7-8tbmv"]
Sep 08 02:13:43.215280 ip-10-0-142-78 kubenswrapper[2547]: W0908 02:13:43.215254    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfec1575a_b738_45df_824f_16c83c21d025.slice/crio-c9e255dccc4230ed02f65ce81f9af005e17ac3d6277c6e94be1f6d19b6f11023 WatchSource:0}: Error finding container c9e255dccc4230ed02f65ce81f9af005e17ac3d6277c6e94be1f6d19b6f11023: Status 404 returned error can't find the container with id c9e255dccc4230ed02f65ce81f9af005e17ac3d6277c6e94be1f6d19b6f11023
Sep 08 02:13:43.671064 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:43.671028    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-173970-54574d8fd-swc7f" event={"ID":"874fb7ce-f6f6-46df-8516-86d81c64e69c","Type":"ContainerStarted","Data":"283add6043eb4e842a0c5618f240415432207c8eeb1ae68c671963476fb06f38"}
Sep 08 02:13:43.671547 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:43.671089    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-173970-54574d8fd-swc7f" event={"ID":"874fb7ce-f6f6-46df-8516-86d81c64e69c","Type":"ContainerStarted","Data":"d0203d0d025d69fbe6380eafb01caa0af5f9b9e7c24783a73b6747257fe51e48"}
Sep 08 02:13:43.671547 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:43.671170    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-173970-54574d8fd-swc7f"
Sep 08 02:13:43.672614 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:43.672589    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-173970-7dc59f76f7-8tbmv" event={"ID":"fec1575a-b738-45df-824f-16c83c21d025","Type":"ContainerStarted","Data":"ad7a9f79738584bae6afac26f76bed719b9fbbeee08fa170270140415baf9f1d"}
Sep 08 02:13:43.672712 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:43.672638    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-173970-7dc59f76f7-8tbmv" event={"ID":"fec1575a-b738-45df-824f-16c83c21d025","Type":"ContainerStarted","Data":"c9e255dccc4230ed02f65ce81f9af005e17ac3d6277c6e94be1f6d19b6f11023"}
Sep 08 02:13:43.672712 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:43.672669    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-173970-7dc59f76f7-8tbmv"
Sep 08 02:13:43.686884 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:43.686833    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-173970-54574d8fd-swc7f" podStartSLOduration=1.686818368 podStartE2EDuration="1.686818368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:13:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:13:43.686383797 +0000 UTC m=+895.210158903" watchObservedRunningTime="2026-09-08 02:13:43.686818368 +0000 UTC m=+895.210593447"
Sep 08 02:13:43.702792 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:43.702742    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-173970-7dc59f76f7-8tbmv" podStartSLOduration=1.702726904 podStartE2EDuration="1.702726904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:13:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:13:43.701239694 +0000 UTC m=+895.225014775" watchObservedRunningTime="2026-09-08 02:13:43.702726904 +0000 UTC m=+895.226501983"
Sep 08 02:13:54.679179 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:54.679124    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-173970-7dc59f76f7-8tbmv"
Sep 08 02:13:54.679635 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:13:54.679221    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-173970-54574d8fd-swc7f"
Sep 08 02:14:18.116357 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:14:18.116318    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-8d800a/test-model-8d800a-kserve-7f8cd79b9f-kv4m7"]
Sep 08 02:14:18.119656 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:14:18.119639    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-8d800a/test-model-8d800a-kserve-7f8cd79b9f-kv4m7"
Sep 08 02:14:18.121915 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:14:18.121894    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-8d800a\"/\"test-model-8d800a-kserve-self-signed-certs\""
Sep 08 02:14:18.122032 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:14:18.121943    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-8d800a\"/\"openshift-service-ca.crt\""
Sep 08 02:14:18.122934 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:14:18.122915    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-8d800a\"/\"default-dockercfg-rv2ll\""
Sep 08 02:14:18.122934 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:14:18.122929    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-8d800a\"/\"kube-root-ca.crt\""
Sep 08 02:14:18.127378 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:14:18.127355    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-8d800a/test-model-8d800a-kserve-7f8cd79b9f-kv4m7"]
Sep 08 02:14:18.236088 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:14:18.236053    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be861d49-5caf-4a99-b5bf-2b53f85dd905-dshm\") pod \"test-model-8d800a-kserve-7f8cd79b9f-kv4m7\" (UID: \"be861d49-5caf-4a99-b5bf-2b53f85dd905\") " pod="ai-tenant-e2e-inf-b-8d800a/test-model-8d800a-kserve-7f8cd79b9f-kv4m7"
Sep 08 02:14:18.236088 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:14:18.236090    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be861d49-5caf-4a99-b5bf-2b53f85dd905-tls-certs\") pod \"test-model-8d800a-kserve-7f8cd79b9f-kv4m7\" (UID: \"be861d49-5caf-4a99-b5bf-2b53f85dd905\") " pod="ai-tenant-e2e-inf-b-8d800a/test-model-8d800a-kserve-7f8cd79b9f-kv4m7"
Sep 08 02:14:18.236311 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:14:18.236111    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x57p4\" (UniqueName: \"kubernetes.io/projected/be861d49-5caf-4a99-b5bf-2b53f85dd905-kube-api-access-x57p4\") pod \"test-model-8d800a-kserve-7f8cd79b9f-kv4m7\" (UID: \"be861d49-5caf-4a99-b5bf-2b53f85dd905\") " pod="ai-tenant-e2e-inf-b-8d800a/test-model-8d800a-kserve-7f8cd79b9f-kv4m7"
Sep 08 02:14:18.236311 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:14:18.236232    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be861d49-5caf-4a99-b5bf-2b53f85dd905-model-cache\") pod \"test-model-8d800a-kserve-7f8cd79b9f-kv4m7\" (UID: \"be861d49-5caf-4a99-b5bf-2b53f85dd905\") " pod="ai-tenant-e2e-inf-b-8d800a/test-model-8d800a-kserve-7f8cd79b9f-kv4m7"
Sep 08 02:14:18.236311 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:14:18.236294    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be861d49-5caf-4a99-b5bf-2b53f85dd905-tmp-dir\") pod \"test-model-8d800a-kserve-7f8cd79b9f-kv4m7\" (UID: \"be861d49-5caf-4a99-b5bf-2b53f85dd905\") " pod="ai-tenant-e2e-inf-b-8d800a/test-model-8d800a-kserve-7f8cd79b9f-kv4m7"
Sep 08 02:14:18.236411 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:14:18.236328    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be861d49-5caf-4a99-b5bf-2b53f85dd905-home\") pod \"test-model-8d800a-kserve-7f8cd79b9f-kv4m7\" (UID: \"be861d49-5caf-4a99-b5bf-2b53f85dd905\") " pod="ai-tenant-e2e-inf-b-8d800a/test-model-8d800a-kserve-7f8cd79b9f-kv4m7"
Sep 08 02:14:18.337440 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:14:18.337405    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be861d49-5caf-4a99-b5bf-2b53f85dd905-tmp-dir\") pod \"test-model-8d800a-kserve-7f8cd79b9f-kv4m7\" (UID: \"be861d49-5caf-4a99-b5bf-2b53f85dd905\") " pod="ai-tenant-e2e-inf-b-8d800a/test-model-8d800a-kserve-7f8cd79b9f-kv4m7"
Sep 08 02:14:18.337440 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:14:18.337442    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be861d49-5caf-4a99-b5bf-2b53f85dd905-home\") pod \"test-model-8d800a-kserve-7f8cd79b9f-kv4m7\" (UID: \"be861d49-5caf-4a99-b5bf-2b53f85dd905\") " pod="ai-tenant-e2e-inf-b-8d800a/test-model-8d800a-kserve-7f8cd79b9f-kv4m7"
Sep 08 02:14:18.337643 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:14:18.337575    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be861d49-5caf-4a99-b5bf-2b53f85dd905-dshm\") pod \"test-model-8d800a-kserve-7f8cd79b9f-kv4m7\" (UID: \"be861d49-5caf-4a99-b5bf-2b53f85dd905\") " pod="ai-tenant-e2e-inf-b-8d800a/test-model-8d800a-kserve-7f8cd79b9f-kv4m7"
Sep 08 02:14:18.337643 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:14:18.337615    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be861d49-5caf-4a99-b5bf-2b53f85dd905-tls-certs\") pod \"test-model-8d800a-kserve-7f8cd79b9f-kv4m7\" (UID: \"be861d49-5caf-4a99-b5bf-2b53f85dd905\") " pod="ai-tenant-e2e-inf-b-8d800a/test-model-8d800a-kserve-7f8cd79b9f-kv4m7"
Sep 08 02:14:18.337643 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:14:18.337640    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x57p4\" (UniqueName: \"kubernetes.io/projected/be861d49-5caf-4a99-b5bf-2b53f85dd905-kube-api-access-x57p4\") pod \"test-model-8d800a-kserve-7f8cd79b9f-kv4m7\" (UID: \"be861d49-5caf-4a99-b5bf-2b53f85dd905\") " pod="ai-tenant-e2e-inf-b-8d800a/test-model-8d800a-kserve-7f8cd79b9f-kv4m7"
Sep 08 02:14:18.337796 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:14:18.337685    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be861d49-5caf-4a99-b5bf-2b53f85dd905-model-cache\") pod \"test-model-8d800a-kserve-7f8cd79b9f-kv4m7\" (UID: \"be861d49-5caf-4a99-b5bf-2b53f85dd905\") " pod="ai-tenant-e2e-inf-b-8d800a/test-model-8d800a-kserve-7f8cd79b9f-kv4m7"
Sep 08 02:14:18.337854 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:14:18.337824    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be861d49-5caf-4a99-b5bf-2b53f85dd905-home\") pod \"test-model-8d800a-kserve-7f8cd79b9f-kv4m7\" (UID: \"be861d49-5caf-4a99-b5bf-2b53f85dd905\") " pod="ai-tenant-e2e-inf-b-8d800a/test-model-8d800a-kserve-7f8cd79b9f-kv4m7"
Sep 08 02:14:18.337912 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:14:18.337880    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be861d49-5caf-4a99-b5bf-2b53f85dd905-tmp-dir\") pod \"test-model-8d800a-kserve-7f8cd79b9f-kv4m7\" (UID: \"be861d49-5caf-4a99-b5bf-2b53f85dd905\") " pod="ai-tenant-e2e-inf-b-8d800a/test-model-8d800a-kserve-7f8cd79b9f-kv4m7"
Sep 08 02:14:18.338003 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:14:18.337986    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be861d49-5caf-4a99-b5bf-2b53f85dd905-model-cache\") pod \"test-model-8d800a-kserve-7f8cd79b9f-kv4m7\" (UID: \"be861d49-5caf-4a99-b5bf-2b53f85dd905\") " pod="ai-tenant-e2e-inf-b-8d800a/test-model-8d800a-kserve-7f8cd79b9f-kv4m7"
Sep 08 02:14:18.339710 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:14:18.339691    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be861d49-5caf-4a99-b5bf-2b53f85dd905-dshm\") pod \"test-model-8d800a-kserve-7f8cd79b9f-kv4m7\" (UID: \"be861d49-5caf-4a99-b5bf-2b53f85dd905\") " pod="ai-tenant-e2e-inf-b-8d800a/test-model-8d800a-kserve-7f8cd79b9f-kv4m7"
Sep 08 02:14:18.340131 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:14:18.340109    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be861d49-5caf-4a99-b5bf-2b53f85dd905-tls-certs\") pod \"test-model-8d800a-kserve-7f8cd79b9f-kv4m7\" (UID: \"be861d49-5caf-4a99-b5bf-2b53f85dd905\") " pod="ai-tenant-e2e-inf-b-8d800a/test-model-8d800a-kserve-7f8cd79b9f-kv4m7"
Sep 08 02:14:18.345353 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:14:18.345330    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x57p4\" (UniqueName: \"kubernetes.io/projected/be861d49-5caf-4a99-b5bf-2b53f85dd905-kube-api-access-x57p4\") pod \"test-model-8d800a-kserve-7f8cd79b9f-kv4m7\" (UID: \"be861d49-5caf-4a99-b5bf-2b53f85dd905\") " pod="ai-tenant-e2e-inf-b-8d800a/test-model-8d800a-kserve-7f8cd79b9f-kv4m7"
Sep 08 02:14:18.429090 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:14:18.429053    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-8d800a/test-model-8d800a-kserve-7f8cd79b9f-kv4m7"
Sep 08 02:14:18.698679 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:14:18.698652    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-8d800a/test-model-8d800a-kserve-7f8cd79b9f-kv4m7"]
Sep 08 02:14:18.700563 ip-10-0-142-78 kubenswrapper[2547]: W0908 02:14:18.700536    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbe861d49_5caf_4a99_b5bf_2b53f85dd905.slice/crio-6752e064ecbec7bb291b257a6bc52dc2bc42377b6faff2d7efa751c2bfa5bb01 WatchSource:0}: Error finding container 6752e064ecbec7bb291b257a6bc52dc2bc42377b6faff2d7efa751c2bfa5bb01: Status 404 returned error can't find the container with id 6752e064ecbec7bb291b257a6bc52dc2bc42377b6faff2d7efa751c2bfa5bb01
Sep 08 02:14:18.832365 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:14:18.832325    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-8d800a/test-model-8d800a-kserve-7f8cd79b9f-kv4m7" event={"ID":"be861d49-5caf-4a99-b5bf-2b53f85dd905","Type":"ContainerStarted","Data":"b30cffb48d7756ed8640d91a4d725405021c35f6a058ec267d52257e18aa251e"}
Sep 08 02:14:18.832512 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:14:18.832375    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-8d800a/test-model-8d800a-kserve-7f8cd79b9f-kv4m7" event={"ID":"be861d49-5caf-4a99-b5bf-2b53f85dd905","Type":"ContainerStarted","Data":"6752e064ecbec7bb291b257a6bc52dc2bc42377b6faff2d7efa751c2bfa5bb01"}
Sep 08 02:14:18.846953 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:14:18.846903    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-8d800a/test-model-8d800a-kserve-7f8cd79b9f-kv4m7" podStartSLOduration=0.846888596 podStartE2EDuration="846.888596ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:14:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:14:18.846026933 +0000 UTC m=+930.369802012" watchObservedRunningTime="2026-09-08 02:14:18.846888596 +0000 UTC m=+930.370663676"
Sep 08 02:14:28.429994 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:14:28.429956    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-8d800a/test-model-8d800a-kserve-7f8cd79b9f-kv4m7"
Sep 08 02:14:28.429994 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:14:28.430007    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-8d800a/test-model-8d800a-kserve-7f8cd79b9f-kv4m7"
Sep 08 02:14:28.442547 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:14:28.442516    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-8d800a/test-model-8d800a-kserve-7f8cd79b9f-kv4m7"
Sep 08 02:14:28.889507 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:14:28.889476    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-8d800a/test-model-8d800a-kserve-7f8cd79b9f-kv4m7"
Sep 08 02:14:32.202076 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:14:32.202036    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-173970/explicit-model-173970-kserve-5cdc85d579-tbgcz"]
Sep 08 02:14:32.206503 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:14:32.206475    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-173970/explicit-model-173970-kserve-5cdc85d579-tbgcz"
Sep 08 02:14:32.208779 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:14:32.208758    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-173970\"/\"openshift-service-ca.crt\""
Sep 08 02:14:32.209803 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:14:32.209767    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-173970\"/\"explicit-model-173970-kserve-self-signed-certs\""
Sep 08 02:14:32.210098 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:14:32.210080    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-173970\"/\"kube-root-ca.crt\""
Sep 08 02:14:32.210492 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:14:32.210472    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-173970\"/\"default-dockercfg-8xqpg\""
Sep 08 02:14:32.212771 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:14:32.212748    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-173970/explicit-model-173970-kserve-5cdc85d579-tbgcz"]
Sep 08 02:14:32.262220 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:14:32.262167    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd31adf2-d114-497b-989e-9dd23146725e-dshm\") pod \"explicit-model-173970-kserve-5cdc85d579-tbgcz\" (UID: \"fd31adf2-d114-497b-989e-9dd23146725e\") " pod="ai-tenant-e2e-explicit-173970/explicit-model-173970-kserve-5cdc85d579-tbgcz"
Sep 08 02:14:32.262220 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:14:32.262216    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd31adf2-d114-497b-989e-9dd23146725e-tls-certs\") pod \"explicit-model-173970-kserve-5cdc85d579-tbgcz\" (UID: \"fd31adf2-d114-497b-989e-9dd23146725e\") " pod="ai-tenant-e2e-explicit-173970/explicit-model-173970-kserve-5cdc85d579-tbgcz"
Sep 08 02:14:32.262429 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:14:32.262319    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2fv6\" (UniqueName: \"kubernetes.io/projected/fd31adf2-d114-497b-989e-9dd23146725e-kube-api-access-r2fv6\") pod \"explicit-model-173970-kserve-5cdc85d579-tbgcz\" (UID: \"fd31adf2-d114-497b-989e-9dd23146725e\") " pod="ai-tenant-e2e-explicit-173970/explicit-model-173970-kserve-5cdc85d579-tbgcz"
Sep 08 02:14:32.262429 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:14:32.262370    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd31adf2-d114-497b-989e-9dd23146725e-tmp-dir\") pod \"explicit-model-173970-kserve-5cdc85d579-tbgcz\" (UID: \"fd31adf2-d114-497b-989e-9dd23146725e\") " pod="ai-tenant-e2e-explicit-173970/explicit-model-173970-kserve-5cdc85d579-tbgcz"
Sep 08 02:14:32.262494 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:14:32.262439    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd31adf2-d114-497b-989e-9dd23146725e-model-cache\") pod \"explicit-model-173970-kserve-5cdc85d579-tbgcz\" (UID: \"fd31adf2-d114-497b-989e-9dd23146725e\") " pod="ai-tenant-e2e-explicit-173970/explicit-model-173970-kserve-5cdc85d579-tbgcz"
Sep 08 02:14:32.262494 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:14:32.262478    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd31adf2-d114-497b-989e-9dd23146725e-home\") pod \"explicit-model-173970-kserve-5cdc85d579-tbgcz\" (UID: \"fd31adf2-d114-497b-989e-9dd23146725e\") " pod="ai-tenant-e2e-explicit-173970/explicit-model-173970-kserve-5cdc85d579-tbgcz"
Sep 08 02:14:32.363773 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:14:32.363738    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2fv6\" (UniqueName: \"kubernetes.io/projected/fd31adf2-d114-497b-989e-9dd23146725e-kube-api-access-r2fv6\") pod \"explicit-model-173970-kserve-5cdc85d579-tbgcz\" (UID: \"fd31adf2-d114-497b-989e-9dd23146725e\") " pod="ai-tenant-e2e-explicit-173970/explicit-model-173970-kserve-5cdc85d579-tbgcz"
Sep 08 02:14:32.363916 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:14:32.363786    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd31adf2-d114-497b-989e-9dd23146725e-tmp-dir\") pod \"explicit-model-173970-kserve-5cdc85d579-tbgcz\" (UID: \"fd31adf2-d114-497b-989e-9dd23146725e\") " pod="ai-tenant-e2e-explicit-173970/explicit-model-173970-kserve-5cdc85d579-tbgcz"
Sep 08 02:14:32.363916 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:14:32.363818    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd31adf2-d114-497b-989e-9dd23146725e-model-cache\") pod \"explicit-model-173970-kserve-5cdc85d579-tbgcz\" (UID: \"fd31adf2-d114-497b-989e-9dd23146725e\") " pod="ai-tenant-e2e-explicit-173970/explicit-model-173970-kserve-5cdc85d579-tbgcz"
Sep 08 02:14:32.363916 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:14:32.363849    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd31adf2-d114-497b-989e-9dd23146725e-home\") pod \"explicit-model-173970-kserve-5cdc85d579-tbgcz\" (UID: \"fd31adf2-d114-497b-989e-9dd23146725e\") " pod="ai-tenant-e2e-explicit-173970/explicit-model-173970-kserve-5cdc85d579-tbgcz"
Sep 08 02:14:32.363916 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:14:32.363870    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd31adf2-d114-497b-989e-9dd23146725e-dshm\") pod \"explicit-model-173970-kserve-5cdc85d579-tbgcz\" (UID: \"fd31adf2-d114-497b-989e-9dd23146725e\") " pod="ai-tenant-e2e-explicit-173970/explicit-model-173970-kserve-5cdc85d579-tbgcz"
Sep 08 02:14:32.363916 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:14:32.363894    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd31adf2-d114-497b-989e-9dd23146725e-tls-certs\") pod \"explicit-model-173970-kserve-5cdc85d579-tbgcz\" (UID: \"fd31adf2-d114-497b-989e-9dd23146725e\") " pod="ai-tenant-e2e-explicit-173970/explicit-model-173970-kserve-5cdc85d579-tbgcz"
Sep 08 02:14:32.364294 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:14:32.364274    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd31adf2-d114-497b-989e-9dd23146725e-tmp-dir\") pod \"explicit-model-173970-kserve-5cdc85d579-tbgcz\" (UID: \"fd31adf2-d114-497b-989e-9dd23146725e\") " pod="ai-tenant-e2e-explicit-173970/explicit-model-173970-kserve-5cdc85d579-tbgcz"
Sep 08 02:14:32.364356 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:14:32.364328    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd31adf2-d114-497b-989e-9dd23146725e-home\") pod \"explicit-model-173970-kserve-5cdc85d579-tbgcz\" (UID: \"fd31adf2-d114-497b-989e-9dd23146725e\") " pod="ai-tenant-e2e-explicit-173970/explicit-model-173970-kserve-5cdc85d579-tbgcz"
Sep 08 02:14:32.364413 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:14:32.364351    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd31adf2-d114-497b-989e-9dd23146725e-model-cache\") pod \"explicit-model-173970-kserve-5cdc85d579-tbgcz\" (UID: \"fd31adf2-d114-497b-989e-9dd23146725e\") " pod="ai-tenant-e2e-explicit-173970/explicit-model-173970-kserve-5cdc85d579-tbgcz"
Sep 08 02:14:32.366151 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:14:32.366118    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd31adf2-d114-497b-989e-9dd23146725e-dshm\") pod \"explicit-model-173970-kserve-5cdc85d579-tbgcz\" (UID: \"fd31adf2-d114-497b-989e-9dd23146725e\") " pod="ai-tenant-e2e-explicit-173970/explicit-model-173970-kserve-5cdc85d579-tbgcz"
Sep 08 02:14:32.366416 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:14:32.366399    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd31adf2-d114-497b-989e-9dd23146725e-tls-certs\") pod \"explicit-model-173970-kserve-5cdc85d579-tbgcz\" (UID: \"fd31adf2-d114-497b-989e-9dd23146725e\") " pod="ai-tenant-e2e-explicit-173970/explicit-model-173970-kserve-5cdc85d579-tbgcz"
Sep 08 02:14:32.370698 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:14:32.370676    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2fv6\" (UniqueName: \"kubernetes.io/projected/fd31adf2-d114-497b-989e-9dd23146725e-kube-api-access-r2fv6\") pod \"explicit-model-173970-kserve-5cdc85d579-tbgcz\" (UID: \"fd31adf2-d114-497b-989e-9dd23146725e\") " pod="ai-tenant-e2e-explicit-173970/explicit-model-173970-kserve-5cdc85d579-tbgcz"
Sep 08 02:14:32.518381 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:14:32.518286    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-173970/explicit-model-173970-kserve-5cdc85d579-tbgcz"
Sep 08 02:14:32.787523 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:14:32.787433    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-173970/explicit-model-173970-kserve-5cdc85d579-tbgcz"]
Sep 08 02:14:32.790531 ip-10-0-142-78 kubenswrapper[2547]: W0908 02:14:32.790499    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfd31adf2_d114_497b_989e_9dd23146725e.slice/crio-80c0b967c97fe418c216b275e231abbc99712bd0918e09289682b33207f3f23b WatchSource:0}: Error finding container 80c0b967c97fe418c216b275e231abbc99712bd0918e09289682b33207f3f23b: Status 404 returned error can't find the container with id 80c0b967c97fe418c216b275e231abbc99712bd0918e09289682b33207f3f23b
Sep 08 02:14:32.897705 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:14:32.897670    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-173970/explicit-model-173970-kserve-5cdc85d579-tbgcz" event={"ID":"fd31adf2-d114-497b-989e-9dd23146725e","Type":"ContainerStarted","Data":"29a4d17ee3c6abfa6108bd81592ca70a7ac649f6e8e068ae81bf7bc436f9d5d0"}
Sep 08 02:14:32.897868 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:14:32.897717    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-173970/explicit-model-173970-kserve-5cdc85d579-tbgcz" event={"ID":"fd31adf2-d114-497b-989e-9dd23146725e","Type":"ContainerStarted","Data":"80c0b967c97fe418c216b275e231abbc99712bd0918e09289682b33207f3f23b"}
Sep 08 02:14:32.914644 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:14:32.914589    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-173970/explicit-model-173970-kserve-5cdc85d579-tbgcz" podStartSLOduration=0.914573724 podStartE2EDuration="914.573724ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:14:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:14:32.913594393 +0000 UTC m=+944.437369572" watchObservedRunningTime="2026-09-08 02:14:32.914573724 +0000 UTC m=+944.438348804"
Sep 08 02:14:42.518431 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:14:42.518384    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-173970/explicit-model-173970-kserve-5cdc85d579-tbgcz"
Sep 08 02:14:42.519028 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:14:42.518527    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-173970/explicit-model-173970-kserve-5cdc85d579-tbgcz"
Sep 08 02:14:42.530848 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:14:42.530823    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-173970/explicit-model-173970-kserve-5cdc85d579-tbgcz"
Sep 08 02:14:42.964347 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:14:42.964309    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-173970/explicit-model-173970-kserve-5cdc85d579-tbgcz"
Sep 08 02:15:00.153785 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:00.153745    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5d27de-29813895-hj94w"]
Sep 08 02:15:00.157241 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:00.157221    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29813895-dl7f8"]
Sep 08 02:15:00.157355 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:00.157332    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5d27de-29813895-hj94w"
Sep 08 02:15:00.160316 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:00.160290    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29813895-dl7f8"
Sep 08 02:15:00.160565 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:00.160547    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8mbwc\""
Sep 08 02:15:00.164968 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:00.164943    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29813895-dl7f8"]
Sep 08 02:15:00.168492 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:00.168467    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5d27de-29813895-hj94w"]
Sep 08 02:15:00.173037 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:00.173012    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5d27de-29813895-hj94w"
Sep 08 02:15:00.178292 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:00.178272    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29813895-dl7f8"
Sep 08 02:15:00.455131 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:00.455106    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5d27de-29813895-hj94w"]
Sep 08 02:15:00.457073 ip-10-0-142-78 kubenswrapper[2547]: W0908 02:15:00.457044    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9948a4f7_6893_45ef_85df_9d339b51cdb2.slice/crio-0a98dec07bda20ddcdc191f0e830493f7dacea4ffa7584401b6206ee8a030350 WatchSource:0}: Error finding container 0a98dec07bda20ddcdc191f0e830493f7dacea4ffa7584401b6206ee8a030350: Status 404 returned error can't find the container with id 0a98dec07bda20ddcdc191f0e830493f7dacea4ffa7584401b6206ee8a030350
Sep 08 02:15:00.459013 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:00.458994    2547 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 02:15:00.462358 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:00.462336    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29813895-dl7f8"]
Sep 08 02:15:00.463955 ip-10-0-142-78 kubenswrapper[2547]: W0908 02:15:00.463930    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf435bc6_e06f_45f6_a2bc_76001b84f52a.slice/crio-e44f19485ddfdef9aa85537a05f52a2d438fead92108f107a27c6ae85078f000 WatchSource:0}: Error finding container e44f19485ddfdef9aa85537a05f52a2d438fead92108f107a27c6ae85078f000: Status 404 returned error can't find the container with id e44f19485ddfdef9aa85537a05f52a2d438fead92108f107a27c6ae85078f000
Sep 08 02:15:01.032172 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:01.032120    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29813895-dl7f8" event={"ID":"bf435bc6-e06f-45f6-a2bc-76001b84f52a","Type":"ContainerStarted","Data":"e44f19485ddfdef9aa85537a05f52a2d438fead92108f107a27c6ae85078f000"}
Sep 08 02:15:01.033409 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:01.033385    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5d27de-29813895-hj94w" event={"ID":"9948a4f7-6893-45ef-85df-9d339b51cdb2","Type":"ContainerStarted","Data":"0a98dec07bda20ddcdc191f0e830493f7dacea4ffa7584401b6206ee8a030350"}
Sep 08 02:15:01.615367 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:01.615327    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-173970/explicit-model-173970-kserve-5cdc85d579-tbgcz"]
Sep 08 02:15:04.050201 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:04.050169    2547 generic.go:356] "Generic (PLEG): container finished" podID="9948a4f7-6893-45ef-85df-9d339b51cdb2" containerID="0a1610cde153aef0a52065673aef5fd18dd76ba5bb7847b46ff4d8e3d809bd01" exitCode=0
Sep 08 02:15:04.050635 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:04.050261    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5d27de-29813895-hj94w" event={"ID":"9948a4f7-6893-45ef-85df-9d339b51cdb2","Type":"ContainerDied","Data":"0a1610cde153aef0a52065673aef5fd18dd76ba5bb7847b46ff4d8e3d809bd01"}
Sep 08 02:15:04.051784 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:04.051764    2547 generic.go:356] "Generic (PLEG): container finished" podID="bf435bc6-e06f-45f6-a2bc-76001b84f52a" containerID="101610b3a09cf794d363729c807a855944b90c313bb08b816f81bf5f85488d52" exitCode=0
Sep 08 02:15:04.051876 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:04.051837    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29813895-dl7f8" event={"ID":"bf435bc6-e06f-45f6-a2bc-76001b84f52a","Type":"ContainerDied","Data":"101610b3a09cf794d363729c807a855944b90c313bb08b816f81bf5f85488d52"}
Sep 08 02:15:05.194853 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:05.194823    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5d27de-29813895-hj94w"
Sep 08 02:15:05.209506 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:05.209457    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29813895-dl7f8"
Sep 08 02:15:06.062201 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:06.062170    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5d27de-29813895-hj94w"
Sep 08 02:15:06.062361 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:06.062169    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5d27de-29813895-hj94w" event={"ID":"9948a4f7-6893-45ef-85df-9d339b51cdb2","Type":"ContainerDied","Data":"0a98dec07bda20ddcdc191f0e830493f7dacea4ffa7584401b6206ee8a030350"}
Sep 08 02:15:06.062361 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:06.062304    2547 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a98dec07bda20ddcdc191f0e830493f7dacea4ffa7584401b6206ee8a030350"
Sep 08 02:15:06.063447 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:06.063431    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29813895-dl7f8"
Sep 08 02:15:06.063526 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:06.063444    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29813895-dl7f8" event={"ID":"bf435bc6-e06f-45f6-a2bc-76001b84f52a","Type":"ContainerDied","Data":"e44f19485ddfdef9aa85537a05f52a2d438fead92108f107a27c6ae85078f000"}
Sep 08 02:15:06.063526 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:06.063485    2547 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e44f19485ddfdef9aa85537a05f52a2d438fead92108f107a27c6ae85078f000"
Sep 08 02:15:12.209031 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:12.208924    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-173970-7dc59f76f7-8tbmv"]
Sep 08 02:15:12.209479 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:12.209209    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-173970-7dc59f76f7-8tbmv" podUID="fec1575a-b738-45df-824f-16c83c21d025" containerName="payload-processing" containerID="cri-o://ad7a9f79738584bae6afac26f76bed719b9fbbeee08fa170270140415baf9f1d" gracePeriod=30
Sep 08 02:15:12.216342 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:12.215794    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-173970-54574d8fd-swc7f"]
Sep 08 02:15:12.216342 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:12.216043    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-173970-54574d8fd-swc7f" podUID="874fb7ce-f6f6-46df-8516-86d81c64e69c" containerName="payload-pre-processing" containerID="cri-o://283add6043eb4e842a0c5618f240415432207c8eeb1ae68c671963476fb06f38" gracePeriod=30
Sep 08 02:15:12.482780 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:12.482755    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-173970-7dc59f76f7-8tbmv"
Sep 08 02:15:12.500562 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:12.500534    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-173970-54574d8fd-swc7f"
Sep 08 02:15:12.617969 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:12.617931    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-529v8\" (UniqueName: \"kubernetes.io/projected/fec1575a-b738-45df-824f-16c83c21d025-kube-api-access-529v8\") pod \"fec1575a-b738-45df-824f-16c83c21d025\" (UID: \"fec1575a-b738-45df-824f-16c83c21d025\") "
Sep 08 02:15:12.618195 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:12.618009    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2v7nb\" (UniqueName: \"kubernetes.io/projected/874fb7ce-f6f6-46df-8516-86d81c64e69c-kube-api-access-2v7nb\") pod \"874fb7ce-f6f6-46df-8516-86d81c64e69c\" (UID: \"874fb7ce-f6f6-46df-8516-86d81c64e69c\") "
Sep 08 02:15:12.618195 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:12.618034    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/874fb7ce-f6f6-46df-8516-86d81c64e69c-plugins-config-volume\") pod \"874fb7ce-f6f6-46df-8516-86d81c64e69c\" (UID: \"874fb7ce-f6f6-46df-8516-86d81c64e69c\") "
Sep 08 02:15:12.618195 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:12.618062    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fec1575a-b738-45df-824f-16c83c21d025-plugins-config-volume\") pod \"fec1575a-b738-45df-824f-16c83c21d025\" (UID: \"fec1575a-b738-45df-824f-16c83c21d025\") "
Sep 08 02:15:12.618503 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:12.618476    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fec1575a-b738-45df-824f-16c83c21d025-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fec1575a-b738-45df-824f-16c83c21d025" (UID: "fec1575a-b738-45df-824f-16c83c21d025"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 02:15:12.618569 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:12.618484    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/874fb7ce-f6f6-46df-8516-86d81c64e69c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "874fb7ce-f6f6-46df-8516-86d81c64e69c" (UID: "874fb7ce-f6f6-46df-8516-86d81c64e69c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 02:15:12.620201 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:12.620166    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/874fb7ce-f6f6-46df-8516-86d81c64e69c-kube-api-access-2v7nb" (OuterVolumeSpecName: "kube-api-access-2v7nb") pod "874fb7ce-f6f6-46df-8516-86d81c64e69c" (UID: "874fb7ce-f6f6-46df-8516-86d81c64e69c"). InnerVolumeSpecName "kube-api-access-2v7nb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 02:15:12.620201 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:12.620185    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fec1575a-b738-45df-824f-16c83c21d025-kube-api-access-529v8" (OuterVolumeSpecName: "kube-api-access-529v8") pod "fec1575a-b738-45df-824f-16c83c21d025" (UID: "fec1575a-b738-45df-824f-16c83c21d025"). InnerVolumeSpecName "kube-api-access-529v8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 02:15:12.718888 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:12.718772    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2v7nb\" (UniqueName: \"kubernetes.io/projected/874fb7ce-f6f6-46df-8516-86d81c64e69c-kube-api-access-2v7nb\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:15:12.718888 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:12.718822    2547 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/874fb7ce-f6f6-46df-8516-86d81c64e69c-plugins-config-volume\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:15:12.718888 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:12.718840    2547 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fec1575a-b738-45df-824f-16c83c21d025-plugins-config-volume\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:15:12.718888 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:12.718851    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-529v8\" (UniqueName: \"kubernetes.io/projected/fec1575a-b738-45df-824f-16c83c21d025-kube-api-access-529v8\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:15:13.103269 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:13.103232    2547 generic.go:356] "Generic (PLEG): container finished" podID="874fb7ce-f6f6-46df-8516-86d81c64e69c" containerID="283add6043eb4e842a0c5618f240415432207c8eeb1ae68c671963476fb06f38" exitCode=0
Sep 08 02:15:13.103607 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:13.103292    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-173970-54574d8fd-swc7f"
Sep 08 02:15:13.103607 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:13.103301    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-173970-54574d8fd-swc7f" event={"ID":"874fb7ce-f6f6-46df-8516-86d81c64e69c","Type":"ContainerDied","Data":"283add6043eb4e842a0c5618f240415432207c8eeb1ae68c671963476fb06f38"}
Sep 08 02:15:13.103607 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:13.103380    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-173970-54574d8fd-swc7f" event={"ID":"874fb7ce-f6f6-46df-8516-86d81c64e69c","Type":"ContainerDied","Data":"d0203d0d025d69fbe6380eafb01caa0af5f9b9e7c24783a73b6747257fe51e48"}
Sep 08 02:15:13.103607 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:13.103415    2547 scope.go:117] "RemoveContainer" containerID="283add6043eb4e842a0c5618f240415432207c8eeb1ae68c671963476fb06f38"
Sep 08 02:15:13.105053 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:13.104942    2547 generic.go:356] "Generic (PLEG): container finished" podID="fec1575a-b738-45df-824f-16c83c21d025" containerID="ad7a9f79738584bae6afac26f76bed719b9fbbeee08fa170270140415baf9f1d" exitCode=0
Sep 08 02:15:13.105053 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:13.104985    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-173970-7dc59f76f7-8tbmv"
Sep 08 02:15:13.105053 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:13.105038    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-173970-7dc59f76f7-8tbmv" event={"ID":"fec1575a-b738-45df-824f-16c83c21d025","Type":"ContainerDied","Data":"ad7a9f79738584bae6afac26f76bed719b9fbbeee08fa170270140415baf9f1d"}
Sep 08 02:15:13.105233 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:13.105124    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-173970-7dc59f76f7-8tbmv" event={"ID":"fec1575a-b738-45df-824f-16c83c21d025","Type":"ContainerDied","Data":"c9e255dccc4230ed02f65ce81f9af005e17ac3d6277c6e94be1f6d19b6f11023"}
Sep 08 02:15:13.112691 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:13.112672    2547 scope.go:117] "RemoveContainer" containerID="283add6043eb4e842a0c5618f240415432207c8eeb1ae68c671963476fb06f38"
Sep 08 02:15:13.113026 ip-10-0-142-78 kubenswrapper[2547]: E0908 02:15:13.112999    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"283add6043eb4e842a0c5618f240415432207c8eeb1ae68c671963476fb06f38\": container with ID starting with 283add6043eb4e842a0c5618f240415432207c8eeb1ae68c671963476fb06f38 not found: ID does not exist" containerID="283add6043eb4e842a0c5618f240415432207c8eeb1ae68c671963476fb06f38"
Sep 08 02:15:13.113085 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:13.113043    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"283add6043eb4e842a0c5618f240415432207c8eeb1ae68c671963476fb06f38"} err="failed to get container status \"283add6043eb4e842a0c5618f240415432207c8eeb1ae68c671963476fb06f38\": rpc error: code = NotFound desc = could not find container \"283add6043eb4e842a0c5618f240415432207c8eeb1ae68c671963476fb06f38\": container with ID starting with 283add6043eb4e842a0c5618f240415432207c8eeb1ae68c671963476fb06f38 not found: ID does not exist"
Sep 08 02:15:13.113085 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:13.113064    2547 scope.go:117] "RemoveContainer" containerID="ad7a9f79738584bae6afac26f76bed719b9fbbeee08fa170270140415baf9f1d"
Sep 08 02:15:13.120852 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:13.120832    2547 scope.go:117] "RemoveContainer" containerID="ad7a9f79738584bae6afac26f76bed719b9fbbeee08fa170270140415baf9f1d"
Sep 08 02:15:13.121184 ip-10-0-142-78 kubenswrapper[2547]: E0908 02:15:13.121163    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad7a9f79738584bae6afac26f76bed719b9fbbeee08fa170270140415baf9f1d\": container with ID starting with ad7a9f79738584bae6afac26f76bed719b9fbbeee08fa170270140415baf9f1d not found: ID does not exist" containerID="ad7a9f79738584bae6afac26f76bed719b9fbbeee08fa170270140415baf9f1d"
Sep 08 02:15:13.121278 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:13.121189    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad7a9f79738584bae6afac26f76bed719b9fbbeee08fa170270140415baf9f1d"} err="failed to get container status \"ad7a9f79738584bae6afac26f76bed719b9fbbeee08fa170270140415baf9f1d\": rpc error: code = NotFound desc = could not find container \"ad7a9f79738584bae6afac26f76bed719b9fbbeee08fa170270140415baf9f1d\": container with ID starting with ad7a9f79738584bae6afac26f76bed719b9fbbeee08fa170270140415baf9f1d not found: ID does not exist"
Sep 08 02:15:13.122799 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:13.122774    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-173970-7dc59f76f7-8tbmv"]
Sep 08 02:15:13.126062 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:13.126041    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-173970-7dc59f76f7-8tbmv"]
Sep 08 02:15:13.137654 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:13.137628    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-173970-54574d8fd-swc7f"]
Sep 08 02:15:13.141342 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:13.141312    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-173970-54574d8fd-swc7f"]
Sep 08 02:15:14.993580 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:14.993543    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="874fb7ce-f6f6-46df-8516-86d81c64e69c" path="/var/lib/kubelet/pods/874fb7ce-f6f6-46df-8516-86d81c64e69c/volumes"
Sep 08 02:15:14.994064 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:14.993880    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fec1575a-b738-45df-824f-16c83c21d025" path="/var/lib/kubelet/pods/fec1575a-b738-45df-824f-16c83c21d025/volumes"
Sep 08 02:15:16.640115 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:16.640054    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-173970/explicit-model-173970-kserve-5cdc85d579-tbgcz" podUID="fd31adf2-d114-497b-989e-9dd23146725e" containerName="main" containerID="cri-o://29a4d17ee3c6abfa6108bd81592ca70a7ac649f6e8e068ae81bf7bc436f9d5d0" gracePeriod=45
Sep 08 02:15:16.895085 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:16.895021    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-173970/explicit-model-173970-kserve-5cdc85d579-tbgcz"
Sep 08 02:15:16.957320 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:16.957284    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r2fv6\" (UniqueName: \"kubernetes.io/projected/fd31adf2-d114-497b-989e-9dd23146725e-kube-api-access-r2fv6\") pod \"fd31adf2-d114-497b-989e-9dd23146725e\" (UID: \"fd31adf2-d114-497b-989e-9dd23146725e\") "
Sep 08 02:15:16.957495 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:16.957334    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd31adf2-d114-497b-989e-9dd23146725e-dshm\") pod \"fd31adf2-d114-497b-989e-9dd23146725e\" (UID: \"fd31adf2-d114-497b-989e-9dd23146725e\") "
Sep 08 02:15:16.957495 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:16.957364    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd31adf2-d114-497b-989e-9dd23146725e-model-cache\") pod \"fd31adf2-d114-497b-989e-9dd23146725e\" (UID: \"fd31adf2-d114-497b-989e-9dd23146725e\") "
Sep 08 02:15:16.957495 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:16.957423    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd31adf2-d114-497b-989e-9dd23146725e-home\") pod \"fd31adf2-d114-497b-989e-9dd23146725e\" (UID: \"fd31adf2-d114-497b-989e-9dd23146725e\") "
Sep 08 02:15:16.957495 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:16.957470    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd31adf2-d114-497b-989e-9dd23146725e-tmp-dir\") pod \"fd31adf2-d114-497b-989e-9dd23146725e\" (UID: \"fd31adf2-d114-497b-989e-9dd23146725e\") "
Sep 08 02:15:16.957697 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:16.957496    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd31adf2-d114-497b-989e-9dd23146725e-tls-certs\") pod \"fd31adf2-d114-497b-989e-9dd23146725e\" (UID: \"fd31adf2-d114-497b-989e-9dd23146725e\") "
Sep 08 02:15:16.957697 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:16.957676    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd31adf2-d114-497b-989e-9dd23146725e-model-cache" (OuterVolumeSpecName: "model-cache") pod "fd31adf2-d114-497b-989e-9dd23146725e" (UID: "fd31adf2-d114-497b-989e-9dd23146725e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 02:15:16.957810 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:16.957739    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd31adf2-d114-497b-989e-9dd23146725e-home" (OuterVolumeSpecName: "home") pod "fd31adf2-d114-497b-989e-9dd23146725e" (UID: "fd31adf2-d114-497b-989e-9dd23146725e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 02:15:16.957810 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:16.957789    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd31adf2-d114-497b-989e-9dd23146725e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fd31adf2-d114-497b-989e-9dd23146725e" (UID: "fd31adf2-d114-497b-989e-9dd23146725e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 02:15:16.957890 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:16.957801    2547 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd31adf2-d114-497b-989e-9dd23146725e-model-cache\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:15:16.959630 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:16.959604    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd31adf2-d114-497b-989e-9dd23146725e-dshm" (OuterVolumeSpecName: "dshm") pod "fd31adf2-d114-497b-989e-9dd23146725e" (UID: "fd31adf2-d114-497b-989e-9dd23146725e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 02:15:16.959726 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:16.959683    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd31adf2-d114-497b-989e-9dd23146725e-kube-api-access-r2fv6" (OuterVolumeSpecName: "kube-api-access-r2fv6") pod "fd31adf2-d114-497b-989e-9dd23146725e" (UID: "fd31adf2-d114-497b-989e-9dd23146725e"). InnerVolumeSpecName "kube-api-access-r2fv6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 02:15:16.959726 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:16.959715    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd31adf2-d114-497b-989e-9dd23146725e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fd31adf2-d114-497b-989e-9dd23146725e" (UID: "fd31adf2-d114-497b-989e-9dd23146725e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 02:15:17.059227 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:17.059185    2547 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd31adf2-d114-497b-989e-9dd23146725e-dshm\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:15:17.059227 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:17.059217    2547 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd31adf2-d114-497b-989e-9dd23146725e-home\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:15:17.059227 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:17.059226    2547 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd31adf2-d114-497b-989e-9dd23146725e-tmp-dir\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:15:17.059227 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:17.059235    2547 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd31adf2-d114-497b-989e-9dd23146725e-tls-certs\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:15:17.059490 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:17.059245    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2fv6\" (UniqueName: \"kubernetes.io/projected/fd31adf2-d114-497b-989e-9dd23146725e-kube-api-access-r2fv6\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:15:17.126904 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:17.126869    2547 generic.go:356] "Generic (PLEG): container finished" podID="fd31adf2-d114-497b-989e-9dd23146725e" containerID="29a4d17ee3c6abfa6108bd81592ca70a7ac649f6e8e068ae81bf7bc436f9d5d0" exitCode=0
Sep 08 02:15:17.127080 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:17.126916    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-173970/explicit-model-173970-kserve-5cdc85d579-tbgcz"
Sep 08 02:15:17.127080 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:17.126935    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-173970/explicit-model-173970-kserve-5cdc85d579-tbgcz" event={"ID":"fd31adf2-d114-497b-989e-9dd23146725e","Type":"ContainerDied","Data":"29a4d17ee3c6abfa6108bd81592ca70a7ac649f6e8e068ae81bf7bc436f9d5d0"}
Sep 08 02:15:17.127080 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:17.126966    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-173970/explicit-model-173970-kserve-5cdc85d579-tbgcz" event={"ID":"fd31adf2-d114-497b-989e-9dd23146725e","Type":"ContainerDied","Data":"80c0b967c97fe418c216b275e231abbc99712bd0918e09289682b33207f3f23b"}
Sep 08 02:15:17.127080 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:17.126982    2547 scope.go:117] "RemoveContainer" containerID="29a4d17ee3c6abfa6108bd81592ca70a7ac649f6e8e068ae81bf7bc436f9d5d0"
Sep 08 02:15:17.135129 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:17.135112    2547 scope.go:117] "RemoveContainer" containerID="29a4d17ee3c6abfa6108bd81592ca70a7ac649f6e8e068ae81bf7bc436f9d5d0"
Sep 08 02:15:17.135405 ip-10-0-142-78 kubenswrapper[2547]: E0908 02:15:17.135385    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29a4d17ee3c6abfa6108bd81592ca70a7ac649f6e8e068ae81bf7bc436f9d5d0\": container with ID starting with 29a4d17ee3c6abfa6108bd81592ca70a7ac649f6e8e068ae81bf7bc436f9d5d0 not found: ID does not exist" containerID="29a4d17ee3c6abfa6108bd81592ca70a7ac649f6e8e068ae81bf7bc436f9d5d0"
Sep 08 02:15:17.135454 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:17.135413    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29a4d17ee3c6abfa6108bd81592ca70a7ac649f6e8e068ae81bf7bc436f9d5d0"} err="failed to get container status \"29a4d17ee3c6abfa6108bd81592ca70a7ac649f6e8e068ae81bf7bc436f9d5d0\": rpc error: code = NotFound desc = could not find container \"29a4d17ee3c6abfa6108bd81592ca70a7ac649f6e8e068ae81bf7bc436f9d5d0\": container with ID starting with 29a4d17ee3c6abfa6108bd81592ca70a7ac649f6e8e068ae81bf7bc436f9d5d0 not found: ID does not exist"
Sep 08 02:15:17.142580 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:17.142527    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-173970/explicit-model-173970-kserve-5cdc85d579-tbgcz"]
Sep 08 02:15:17.144390 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:17.144370    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-173970/explicit-model-173970-kserve-5cdc85d579-tbgcz"]
Sep 08 02:15:18.995394 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:18.995361    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd31adf2-d114-497b-989e-9dd23146725e" path="/var/lib/kubelet/pods/fd31adf2-d114-497b-989e-9dd23146725e/volumes"
Sep 08 02:15:37.103903 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:37.103865    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8fa74948-6b998c9754-xw5cs"]
Sep 08 02:15:37.104420 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:37.104236    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="874fb7ce-f6f6-46df-8516-86d81c64e69c" containerName="payload-pre-processing"
Sep 08 02:15:37.104420 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:37.104250    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="874fb7ce-f6f6-46df-8516-86d81c64e69c" containerName="payload-pre-processing"
Sep 08 02:15:37.104420 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:37.104261    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf435bc6-e06f-45f6-a2bc-76001b84f52a" containerName="cleanup"
Sep 08 02:15:37.104420 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:37.104267    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf435bc6-e06f-45f6-a2bc-76001b84f52a" containerName="cleanup"
Sep 08 02:15:37.104420 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:37.104278    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd31adf2-d114-497b-989e-9dd23146725e" containerName="main"
Sep 08 02:15:37.104420 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:37.104287    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd31adf2-d114-497b-989e-9dd23146725e" containerName="main"
Sep 08 02:15:37.104420 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:37.104299    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9948a4f7-6893-45ef-85df-9d339b51cdb2" containerName="cleanup"
Sep 08 02:15:37.104420 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:37.104304    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="9948a4f7-6893-45ef-85df-9d339b51cdb2" containerName="cleanup"
Sep 08 02:15:37.104420 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:37.104313    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fec1575a-b738-45df-824f-16c83c21d025" containerName="payload-processing"
Sep 08 02:15:37.104420 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:37.104318    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="fec1575a-b738-45df-824f-16c83c21d025" containerName="payload-processing"
Sep 08 02:15:37.104420 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:37.104376    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd31adf2-d114-497b-989e-9dd23146725e" containerName="main"
Sep 08 02:15:37.104420 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:37.104386    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="9948a4f7-6893-45ef-85df-9d339b51cdb2" containerName="cleanup"
Sep 08 02:15:37.104420 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:37.104393    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="874fb7ce-f6f6-46df-8516-86d81c64e69c" containerName="payload-pre-processing"
Sep 08 02:15:37.104420 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:37.104400    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf435bc6-e06f-45f6-a2bc-76001b84f52a" containerName="cleanup"
Sep 08 02:15:37.104420 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:37.104404    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="fec1575a-b738-45df-824f-16c83c21d025" containerName="payload-processing"
Sep 08 02:15:37.111823 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:37.111803    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8fa74948-6b998c9754-xw5cs"
Sep 08 02:15:37.114216 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:37.114188    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-8fa74948-dockercfg-m2pxb\""
Sep 08 02:15:37.114344 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:37.114243    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-8fa74948\""
Sep 08 02:15:37.117510 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:37.117483    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8fa74948-6b998c9754-xw5cs"]
Sep 08 02:15:37.133519 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:37.133487    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrv5b\" (UniqueName: \"kubernetes.io/projected/293a49f1-6902-4192-80cd-105f9594bf89-kube-api-access-wrv5b\") pod \"payload-pre-processing-e2e-mt-8fa74948-6b998c9754-xw5cs\" (UID: \"293a49f1-6902-4192-80cd-105f9594bf89\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8fa74948-6b998c9754-xw5cs"
Sep 08 02:15:37.133682 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:37.133529    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/293a49f1-6902-4192-80cd-105f9594bf89-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-8fa74948-6b998c9754-xw5cs\" (UID: \"293a49f1-6902-4192-80cd-105f9594bf89\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8fa74948-6b998c9754-xw5cs"
Sep 08 02:15:37.137510 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:37.137478    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8fa74948-7f464b7cd9-75cxz"]
Sep 08 02:15:37.141264 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:37.141242    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8fa74948-7f464b7cd9-75cxz"
Sep 08 02:15:37.153442 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:37.153414    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8fa74948-7f464b7cd9-75cxz"]
Sep 08 02:15:37.234810 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:37.234768    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/293a49f1-6902-4192-80cd-105f9594bf89-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-8fa74948-6b998c9754-xw5cs\" (UID: \"293a49f1-6902-4192-80cd-105f9594bf89\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8fa74948-6b998c9754-xw5cs"
Sep 08 02:15:37.235011 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:37.234873    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zlm4\" (UniqueName: \"kubernetes.io/projected/bf419798-9b6d-465a-8bee-5301c8a52031-kube-api-access-7zlm4\") pod \"payload-processing-e2e-mt-8fa74948-7f464b7cd9-75cxz\" (UID: \"bf419798-9b6d-465a-8bee-5301c8a52031\") " pod="openshift-ingress/payload-processing-e2e-mt-8fa74948-7f464b7cd9-75cxz"
Sep 08 02:15:37.235011 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:37.234948    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf419798-9b6d-465a-8bee-5301c8a52031-plugins-config-volume\") pod \"payload-processing-e2e-mt-8fa74948-7f464b7cd9-75cxz\" (UID: \"bf419798-9b6d-465a-8bee-5301c8a52031\") " pod="openshift-ingress/payload-processing-e2e-mt-8fa74948-7f464b7cd9-75cxz"
Sep 08 02:15:37.235011 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:37.235001    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrv5b\" (UniqueName: \"kubernetes.io/projected/293a49f1-6902-4192-80cd-105f9594bf89-kube-api-access-wrv5b\") pod \"payload-pre-processing-e2e-mt-8fa74948-6b998c9754-xw5cs\" (UID: \"293a49f1-6902-4192-80cd-105f9594bf89\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8fa74948-6b998c9754-xw5cs"
Sep 08 02:15:37.235790 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:37.235769    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/293a49f1-6902-4192-80cd-105f9594bf89-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-8fa74948-6b998c9754-xw5cs\" (UID: \"293a49f1-6902-4192-80cd-105f9594bf89\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8fa74948-6b998c9754-xw5cs"
Sep 08 02:15:37.243872 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:37.243846    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrv5b\" (UniqueName: \"kubernetes.io/projected/293a49f1-6902-4192-80cd-105f9594bf89-kube-api-access-wrv5b\") pod \"payload-pre-processing-e2e-mt-8fa74948-6b998c9754-xw5cs\" (UID: \"293a49f1-6902-4192-80cd-105f9594bf89\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8fa74948-6b998c9754-xw5cs"
Sep 08 02:15:37.335599 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:37.335558    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zlm4\" (UniqueName: \"kubernetes.io/projected/bf419798-9b6d-465a-8bee-5301c8a52031-kube-api-access-7zlm4\") pod \"payload-processing-e2e-mt-8fa74948-7f464b7cd9-75cxz\" (UID: \"bf419798-9b6d-465a-8bee-5301c8a52031\") " pod="openshift-ingress/payload-processing-e2e-mt-8fa74948-7f464b7cd9-75cxz"
Sep 08 02:15:37.335781 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:37.335613    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf419798-9b6d-465a-8bee-5301c8a52031-plugins-config-volume\") pod \"payload-processing-e2e-mt-8fa74948-7f464b7cd9-75cxz\" (UID: \"bf419798-9b6d-465a-8bee-5301c8a52031\") " pod="openshift-ingress/payload-processing-e2e-mt-8fa74948-7f464b7cd9-75cxz"
Sep 08 02:15:37.336306 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:37.336288    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf419798-9b6d-465a-8bee-5301c8a52031-plugins-config-volume\") pod \"payload-processing-e2e-mt-8fa74948-7f464b7cd9-75cxz\" (UID: \"bf419798-9b6d-465a-8bee-5301c8a52031\") " pod="openshift-ingress/payload-processing-e2e-mt-8fa74948-7f464b7cd9-75cxz"
Sep 08 02:15:37.343076 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:37.343050    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zlm4\" (UniqueName: \"kubernetes.io/projected/bf419798-9b6d-465a-8bee-5301c8a52031-kube-api-access-7zlm4\") pod \"payload-processing-e2e-mt-8fa74948-7f464b7cd9-75cxz\" (UID: \"bf419798-9b6d-465a-8bee-5301c8a52031\") " pod="openshift-ingress/payload-processing-e2e-mt-8fa74948-7f464b7cd9-75cxz"
Sep 08 02:15:37.422994 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:37.422907    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8fa74948-6b998c9754-xw5cs"
Sep 08 02:15:37.451669 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:37.451635    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8fa74948-7f464b7cd9-75cxz"
Sep 08 02:15:37.693820 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:37.693791    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8fa74948-6b998c9754-xw5cs"]
Sep 08 02:15:37.696212 ip-10-0-142-78 kubenswrapper[2547]: W0908 02:15:37.696185    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod293a49f1_6902_4192_80cd_105f9594bf89.slice/crio-5adb8c0a8d4f6f58fadf3bea1b3814478d9a823e3ae3cc9d5644f304ebb971ef WatchSource:0}: Error finding container 5adb8c0a8d4f6f58fadf3bea1b3814478d9a823e3ae3cc9d5644f304ebb971ef: Status 404 returned error can't find the container with id 5adb8c0a8d4f6f58fadf3bea1b3814478d9a823e3ae3cc9d5644f304ebb971ef
Sep 08 02:15:37.731599 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:37.731575    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8fa74948-7f464b7cd9-75cxz"]
Sep 08 02:15:37.733997 ip-10-0-142-78 kubenswrapper[2547]: W0908 02:15:37.733974    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf419798_9b6d_465a_8bee_5301c8a52031.slice/crio-503e3aa8fd80948e34215a00bbf3268ccffefdba487ff67e98f95314a872a8ed WatchSource:0}: Error finding container 503e3aa8fd80948e34215a00bbf3268ccffefdba487ff67e98f95314a872a8ed: Status 404 returned error can't find the container with id 503e3aa8fd80948e34215a00bbf3268ccffefdba487ff67e98f95314a872a8ed
Sep 08 02:15:38.224369 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:38.224333    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8fa74948-7f464b7cd9-75cxz" event={"ID":"bf419798-9b6d-465a-8bee-5301c8a52031","Type":"ContainerStarted","Data":"11f04e1f3c3c651e89993e33471edc7f149c3530ec7195961dae636466fa88af"}
Sep 08 02:15:38.224829 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:38.224389    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8fa74948-7f464b7cd9-75cxz" event={"ID":"bf419798-9b6d-465a-8bee-5301c8a52031","Type":"ContainerStarted","Data":"503e3aa8fd80948e34215a00bbf3268ccffefdba487ff67e98f95314a872a8ed"}
Sep 08 02:15:38.224829 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:38.224418    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-8fa74948-7f464b7cd9-75cxz"
Sep 08 02:15:38.225867 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:38.225841    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8fa74948-6b998c9754-xw5cs" event={"ID":"293a49f1-6902-4192-80cd-105f9594bf89","Type":"ContainerStarted","Data":"74a8cf9633e1570c02f9d0cc265e962e93b6470f002ef9eb8ab2fb22459fe1e6"}
Sep 08 02:15:38.225960 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:38.225888    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8fa74948-6b998c9754-xw5cs" event={"ID":"293a49f1-6902-4192-80cd-105f9594bf89","Type":"ContainerStarted","Data":"5adb8c0a8d4f6f58fadf3bea1b3814478d9a823e3ae3cc9d5644f304ebb971ef"}
Sep 08 02:15:38.225960 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:38.225926    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-8fa74948-6b998c9754-xw5cs"
Sep 08 02:15:38.239826 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:38.239756    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-8fa74948-7f464b7cd9-75cxz" podStartSLOduration=1.239736145 podStartE2EDuration="1.239736145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:15:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:15:38.238422999 +0000 UTC m=+1009.762198107" watchObservedRunningTime="2026-09-08 02:15:38.239736145 +0000 UTC m=+1009.763511226"
Sep 08 02:15:38.254194 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:38.254120    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-8fa74948-6b998c9754-xw5cs" podStartSLOduration=1.2541042660000001 podStartE2EDuration="1.254104266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:15:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:15:38.252158424 +0000 UTC m=+1009.775933507" watchObservedRunningTime="2026-09-08 02:15:38.254104266 +0000 UTC m=+1009.777879347"
Sep 08 02:15:44.198351 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:44.198312    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-cf06cb/noten-model-cf06cb-kserve-5d65f6d79f-xwz2r"]
Sep 08 02:15:44.201929 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:44.201910    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-cf06cb/noten-model-cf06cb-kserve-5d65f6d79f-xwz2r"
Sep 08 02:15:44.212927 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:44.212890    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-cf06cb\"/\"noten-model-cf06cb-kserve-self-signed-certs\""
Sep 08 02:15:44.213096 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:44.212938    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-cf06cb\"/\"openshift-service-ca.crt\""
Sep 08 02:15:44.213096 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:44.213059    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-cf06cb\"/\"default-dockercfg-mdv9r\""
Sep 08 02:15:44.213776 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:44.213753    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-cf06cb\"/\"kube-root-ca.crt\""
Sep 08 02:15:44.216347 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:44.216324    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-cf06cb/noten-model-cf06cb-kserve-5d65f6d79f-xwz2r"]
Sep 08 02:15:44.285443 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:44.285407    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a07a202b-95da-4d7f-aa33-c1d939e68ba5-tmp-dir\") pod \"noten-model-cf06cb-kserve-5d65f6d79f-xwz2r\" (UID: \"a07a202b-95da-4d7f-aa33-c1d939e68ba5\") " pod="ai-tenant-e2e-noten-cf06cb/noten-model-cf06cb-kserve-5d65f6d79f-xwz2r"
Sep 08 02:15:44.285626 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:44.285449    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a07a202b-95da-4d7f-aa33-c1d939e68ba5-tls-certs\") pod \"noten-model-cf06cb-kserve-5d65f6d79f-xwz2r\" (UID: \"a07a202b-95da-4d7f-aa33-c1d939e68ba5\") " pod="ai-tenant-e2e-noten-cf06cb/noten-model-cf06cb-kserve-5d65f6d79f-xwz2r"
Sep 08 02:15:44.285626 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:44.285470    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8pdf\" (UniqueName: \"kubernetes.io/projected/a07a202b-95da-4d7f-aa33-c1d939e68ba5-kube-api-access-x8pdf\") pod \"noten-model-cf06cb-kserve-5d65f6d79f-xwz2r\" (UID: \"a07a202b-95da-4d7f-aa33-c1d939e68ba5\") " pod="ai-tenant-e2e-noten-cf06cb/noten-model-cf06cb-kserve-5d65f6d79f-xwz2r"
Sep 08 02:15:44.285626 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:44.285514    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a07a202b-95da-4d7f-aa33-c1d939e68ba5-home\") pod \"noten-model-cf06cb-kserve-5d65f6d79f-xwz2r\" (UID: \"a07a202b-95da-4d7f-aa33-c1d939e68ba5\") " pod="ai-tenant-e2e-noten-cf06cb/noten-model-cf06cb-kserve-5d65f6d79f-xwz2r"
Sep 08 02:15:44.285626 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:44.285547    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a07a202b-95da-4d7f-aa33-c1d939e68ba5-dshm\") pod \"noten-model-cf06cb-kserve-5d65f6d79f-xwz2r\" (UID: \"a07a202b-95da-4d7f-aa33-c1d939e68ba5\") " pod="ai-tenant-e2e-noten-cf06cb/noten-model-cf06cb-kserve-5d65f6d79f-xwz2r"
Sep 08 02:15:44.285626 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:44.285573    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a07a202b-95da-4d7f-aa33-c1d939e68ba5-model-cache\") pod \"noten-model-cf06cb-kserve-5d65f6d79f-xwz2r\" (UID: \"a07a202b-95da-4d7f-aa33-c1d939e68ba5\") " pod="ai-tenant-e2e-noten-cf06cb/noten-model-cf06cb-kserve-5d65f6d79f-xwz2r"
Sep 08 02:15:44.386224 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:44.386178    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a07a202b-95da-4d7f-aa33-c1d939e68ba5-home\") pod \"noten-model-cf06cb-kserve-5d65f6d79f-xwz2r\" (UID: \"a07a202b-95da-4d7f-aa33-c1d939e68ba5\") " pod="ai-tenant-e2e-noten-cf06cb/noten-model-cf06cb-kserve-5d65f6d79f-xwz2r"
Sep 08 02:15:44.386378 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:44.386262    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a07a202b-95da-4d7f-aa33-c1d939e68ba5-dshm\") pod \"noten-model-cf06cb-kserve-5d65f6d79f-xwz2r\" (UID: \"a07a202b-95da-4d7f-aa33-c1d939e68ba5\") " pod="ai-tenant-e2e-noten-cf06cb/noten-model-cf06cb-kserve-5d65f6d79f-xwz2r"
Sep 08 02:15:44.386378 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:44.386311    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a07a202b-95da-4d7f-aa33-c1d939e68ba5-model-cache\") pod \"noten-model-cf06cb-kserve-5d65f6d79f-xwz2r\" (UID: \"a07a202b-95da-4d7f-aa33-c1d939e68ba5\") " pod="ai-tenant-e2e-noten-cf06cb/noten-model-cf06cb-kserve-5d65f6d79f-xwz2r"
Sep 08 02:15:44.386378 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:44.386364    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a07a202b-95da-4d7f-aa33-c1d939e68ba5-tmp-dir\") pod \"noten-model-cf06cb-kserve-5d65f6d79f-xwz2r\" (UID: \"a07a202b-95da-4d7f-aa33-c1d939e68ba5\") " pod="ai-tenant-e2e-noten-cf06cb/noten-model-cf06cb-kserve-5d65f6d79f-xwz2r"
Sep 08 02:15:44.386527 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:44.386400    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a07a202b-95da-4d7f-aa33-c1d939e68ba5-tls-certs\") pod \"noten-model-cf06cb-kserve-5d65f6d79f-xwz2r\" (UID: \"a07a202b-95da-4d7f-aa33-c1d939e68ba5\") " pod="ai-tenant-e2e-noten-cf06cb/noten-model-cf06cb-kserve-5d65f6d79f-xwz2r"
Sep 08 02:15:44.386643 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:44.386620    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a07a202b-95da-4d7f-aa33-c1d939e68ba5-home\") pod \"noten-model-cf06cb-kserve-5d65f6d79f-xwz2r\" (UID: \"a07a202b-95da-4d7f-aa33-c1d939e68ba5\") " pod="ai-tenant-e2e-noten-cf06cb/noten-model-cf06cb-kserve-5d65f6d79f-xwz2r"
Sep 08 02:15:44.386711 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:44.386663    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8pdf\" (UniqueName: \"kubernetes.io/projected/a07a202b-95da-4d7f-aa33-c1d939e68ba5-kube-api-access-x8pdf\") pod \"noten-model-cf06cb-kserve-5d65f6d79f-xwz2r\" (UID: \"a07a202b-95da-4d7f-aa33-c1d939e68ba5\") " pod="ai-tenant-e2e-noten-cf06cb/noten-model-cf06cb-kserve-5d65f6d79f-xwz2r"
Sep 08 02:15:44.386828 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:44.386801    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a07a202b-95da-4d7f-aa33-c1d939e68ba5-tmp-dir\") pod \"noten-model-cf06cb-kserve-5d65f6d79f-xwz2r\" (UID: \"a07a202b-95da-4d7f-aa33-c1d939e68ba5\") " pod="ai-tenant-e2e-noten-cf06cb/noten-model-cf06cb-kserve-5d65f6d79f-xwz2r"
Sep 08 02:15:44.386964 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:44.386807    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a07a202b-95da-4d7f-aa33-c1d939e68ba5-model-cache\") pod \"noten-model-cf06cb-kserve-5d65f6d79f-xwz2r\" (UID: \"a07a202b-95da-4d7f-aa33-c1d939e68ba5\") " pod="ai-tenant-e2e-noten-cf06cb/noten-model-cf06cb-kserve-5d65f6d79f-xwz2r"
Sep 08 02:15:44.388584 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:44.388559    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a07a202b-95da-4d7f-aa33-c1d939e68ba5-dshm\") pod \"noten-model-cf06cb-kserve-5d65f6d79f-xwz2r\" (UID: \"a07a202b-95da-4d7f-aa33-c1d939e68ba5\") " pod="ai-tenant-e2e-noten-cf06cb/noten-model-cf06cb-kserve-5d65f6d79f-xwz2r"
Sep 08 02:15:44.389000 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:44.388980    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a07a202b-95da-4d7f-aa33-c1d939e68ba5-tls-certs\") pod \"noten-model-cf06cb-kserve-5d65f6d79f-xwz2r\" (UID: \"a07a202b-95da-4d7f-aa33-c1d939e68ba5\") " pod="ai-tenant-e2e-noten-cf06cb/noten-model-cf06cb-kserve-5d65f6d79f-xwz2r"
Sep 08 02:15:44.393937 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:44.393911    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8pdf\" (UniqueName: \"kubernetes.io/projected/a07a202b-95da-4d7f-aa33-c1d939e68ba5-kube-api-access-x8pdf\") pod \"noten-model-cf06cb-kserve-5d65f6d79f-xwz2r\" (UID: \"a07a202b-95da-4d7f-aa33-c1d939e68ba5\") " pod="ai-tenant-e2e-noten-cf06cb/noten-model-cf06cb-kserve-5d65f6d79f-xwz2r"
Sep 08 02:15:44.521400 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:44.521303    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-cf06cb/noten-model-cf06cb-kserve-5d65f6d79f-xwz2r"
Sep 08 02:15:44.790615 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:44.790588    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-cf06cb/noten-model-cf06cb-kserve-5d65f6d79f-xwz2r"]
Sep 08 02:15:44.792573 ip-10-0-142-78 kubenswrapper[2547]: W0908 02:15:44.792537    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda07a202b_95da_4d7f_aa33_c1d939e68ba5.slice/crio-131a43a8685b0ba5d304e63fefe458533f1df495b342e36836a29d29576ebac4 WatchSource:0}: Error finding container 131a43a8685b0ba5d304e63fefe458533f1df495b342e36836a29d29576ebac4: Status 404 returned error can't find the container with id 131a43a8685b0ba5d304e63fefe458533f1df495b342e36836a29d29576ebac4
Sep 08 02:15:45.259764 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:45.259725    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-cf06cb/noten-model-cf06cb-kserve-5d65f6d79f-xwz2r" event={"ID":"a07a202b-95da-4d7f-aa33-c1d939e68ba5","Type":"ContainerStarted","Data":"3f690e8ecf1d43bdcece3c116ada6367707085088cfa344e46dea760dc19eba5"}
Sep 08 02:15:45.259764 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:45.259768    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-cf06cb/noten-model-cf06cb-kserve-5d65f6d79f-xwz2r" event={"ID":"a07a202b-95da-4d7f-aa33-c1d939e68ba5","Type":"ContainerStarted","Data":"131a43a8685b0ba5d304e63fefe458533f1df495b342e36836a29d29576ebac4"}
Sep 08 02:15:45.275430 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:45.275372    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-cf06cb/noten-model-cf06cb-kserve-5d65f6d79f-xwz2r" podStartSLOduration=1.27535505 podStartE2EDuration="1.27535505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:15:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:15:45.273381417 +0000 UTC m=+1016.797156498" watchObservedRunningTime="2026-09-08 02:15:45.27535505 +0000 UTC m=+1016.799130129"
Sep 08 02:15:48.194074 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:48.194036    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-cf06cb/noten-model-cf06cb-kserve-5d65f6d79f-xwz2r"]
Sep 08 02:15:49.165741 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:49.165700    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-8d800a/test-model-8d800a-kserve-7f8cd79b9f-kv4m7"]
Sep 08 02:15:49.233155 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:49.233109    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-8fa74948-6b998c9754-xw5cs"
Sep 08 02:15:49.233563 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:49.233287    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-8fa74948-7f464b7cd9-75cxz"
Sep 08 02:15:54.522263 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:54.522208    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-cf06cb/noten-model-cf06cb-kserve-5d65f6d79f-xwz2r"
Sep 08 02:15:59.923688 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:59.923643    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5d27de-29813895-hj94w"]
Sep 08 02:15:59.925246 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:59.925223    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5d27de-29813895-hj94w"]
Sep 08 02:15:59.947897 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:59.947854    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5d27de-58656496c5-qzvbz"]
Sep 08 02:15:59.948148 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:59.948090    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-5d27de-58656496c5-qzvbz" podUID="d1dabb5b-1a34-4dc3-a7c5-eaa6db21ce32" containerName="payload-processing" containerID="cri-o://e5b937ae156ff34b0b4492beddcb479329dcbee74fac44939a4131d399884ff7" gracePeriod=30
Sep 08 02:15:59.955590 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:59.954526    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5d27de-78684b5958-vtg65"]
Sep 08 02:15:59.955590 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:15:59.954805    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5d27de-78684b5958-vtg65" podUID="d6d26402-535b-4f5d-8cff-91b3bc126699" containerName="payload-pre-processing" containerID="cri-o://43a776e44312ae6e2bfc39107d915a25a6ca73b700b22e89a6b6648a1afa0605" gracePeriod=30
Sep 08 02:16:00.227982 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:00.227960    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-5d27de-58656496c5-qzvbz"
Sep 08 02:16:00.230714 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:00.230692    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5d27de-78684b5958-vtg65"
Sep 08 02:16:00.324193 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:00.324123    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1dabb5b-1a34-4dc3-a7c5-eaa6db21ce32-plugins-config-volume\") pod \"d1dabb5b-1a34-4dc3-a7c5-eaa6db21ce32\" (UID: \"d1dabb5b-1a34-4dc3-a7c5-eaa6db21ce32\") "
Sep 08 02:16:00.324362 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:00.324228    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wzftx\" (UniqueName: \"kubernetes.io/projected/d6d26402-535b-4f5d-8cff-91b3bc126699-kube-api-access-wzftx\") pod \"d6d26402-535b-4f5d-8cff-91b3bc126699\" (UID: \"d6d26402-535b-4f5d-8cff-91b3bc126699\") "
Sep 08 02:16:00.324362 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:00.324280    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6d26402-535b-4f5d-8cff-91b3bc126699-plugins-config-volume\") pod \"d6d26402-535b-4f5d-8cff-91b3bc126699\" (UID: \"d6d26402-535b-4f5d-8cff-91b3bc126699\") "
Sep 08 02:16:00.324362 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:00.324337    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2hdk2\" (UniqueName: \"kubernetes.io/projected/d1dabb5b-1a34-4dc3-a7c5-eaa6db21ce32-kube-api-access-2hdk2\") pod \"d1dabb5b-1a34-4dc3-a7c5-eaa6db21ce32\" (UID: \"d1dabb5b-1a34-4dc3-a7c5-eaa6db21ce32\") "
Sep 08 02:16:00.324626 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:00.324601    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1dabb5b-1a34-4dc3-a7c5-eaa6db21ce32-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d1dabb5b-1a34-4dc3-a7c5-eaa6db21ce32" (UID: "d1dabb5b-1a34-4dc3-a7c5-eaa6db21ce32"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 02:16:00.324743 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:00.324709    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6d26402-535b-4f5d-8cff-91b3bc126699-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d6d26402-535b-4f5d-8cff-91b3bc126699" (UID: "d6d26402-535b-4f5d-8cff-91b3bc126699"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 02:16:00.326555 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:00.326514    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1dabb5b-1a34-4dc3-a7c5-eaa6db21ce32-kube-api-access-2hdk2" (OuterVolumeSpecName: "kube-api-access-2hdk2") pod "d1dabb5b-1a34-4dc3-a7c5-eaa6db21ce32" (UID: "d1dabb5b-1a34-4dc3-a7c5-eaa6db21ce32"). InnerVolumeSpecName "kube-api-access-2hdk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 02:16:00.326555 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:00.326549    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6d26402-535b-4f5d-8cff-91b3bc126699-kube-api-access-wzftx" (OuterVolumeSpecName: "kube-api-access-wzftx") pod "d6d26402-535b-4f5d-8cff-91b3bc126699" (UID: "d6d26402-535b-4f5d-8cff-91b3bc126699"). InnerVolumeSpecName "kube-api-access-wzftx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 02:16:00.331693 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:00.331663    2547 generic.go:356] "Generic (PLEG): container finished" podID="d6d26402-535b-4f5d-8cff-91b3bc126699" containerID="43a776e44312ae6e2bfc39107d915a25a6ca73b700b22e89a6b6648a1afa0605" exitCode=0
Sep 08 02:16:00.331810 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:00.331718    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5d27de-78684b5958-vtg65"
Sep 08 02:16:00.331810 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:00.331743    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5d27de-78684b5958-vtg65" event={"ID":"d6d26402-535b-4f5d-8cff-91b3bc126699","Type":"ContainerDied","Data":"43a776e44312ae6e2bfc39107d915a25a6ca73b700b22e89a6b6648a1afa0605"}
Sep 08 02:16:00.331938 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:00.331840    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5d27de-78684b5958-vtg65" event={"ID":"d6d26402-535b-4f5d-8cff-91b3bc126699","Type":"ContainerDied","Data":"500b2364064f581d4089d8d9a505fa595afe1846d2072391c3d9f940f0511d29"}
Sep 08 02:16:00.331938 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:00.331875    2547 scope.go:117] "RemoveContainer" containerID="43a776e44312ae6e2bfc39107d915a25a6ca73b700b22e89a6b6648a1afa0605"
Sep 08 02:16:00.333358 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:00.333335    2547 generic.go:356] "Generic (PLEG): container finished" podID="d1dabb5b-1a34-4dc3-a7c5-eaa6db21ce32" containerID="e5b937ae156ff34b0b4492beddcb479329dcbee74fac44939a4131d399884ff7" exitCode=0
Sep 08 02:16:00.333455 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:00.333377    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-5d27de-58656496c5-qzvbz" event={"ID":"d1dabb5b-1a34-4dc3-a7c5-eaa6db21ce32","Type":"ContainerDied","Data":"e5b937ae156ff34b0b4492beddcb479329dcbee74fac44939a4131d399884ff7"}
Sep 08 02:16:00.333455 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:00.333387    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-5d27de-58656496c5-qzvbz"
Sep 08 02:16:00.333455 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:00.333418    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-5d27de-58656496c5-qzvbz" event={"ID":"d1dabb5b-1a34-4dc3-a7c5-eaa6db21ce32","Type":"ContainerDied","Data":"2f6ba7333fb07053fad03526d03555bbb96d6fb181fe6e1b1bd496cf256ddefb"}
Sep 08 02:16:00.341792 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:00.341771    2547 scope.go:117] "RemoveContainer" containerID="43a776e44312ae6e2bfc39107d915a25a6ca73b700b22e89a6b6648a1afa0605"
Sep 08 02:16:00.342120 ip-10-0-142-78 kubenswrapper[2547]: E0908 02:16:00.342095    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43a776e44312ae6e2bfc39107d915a25a6ca73b700b22e89a6b6648a1afa0605\": container with ID starting with 43a776e44312ae6e2bfc39107d915a25a6ca73b700b22e89a6b6648a1afa0605 not found: ID does not exist" containerID="43a776e44312ae6e2bfc39107d915a25a6ca73b700b22e89a6b6648a1afa0605"
Sep 08 02:16:00.342238 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:00.342129    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43a776e44312ae6e2bfc39107d915a25a6ca73b700b22e89a6b6648a1afa0605"} err="failed to get container status \"43a776e44312ae6e2bfc39107d915a25a6ca73b700b22e89a6b6648a1afa0605\": rpc error: code = NotFound desc = could not find container \"43a776e44312ae6e2bfc39107d915a25a6ca73b700b22e89a6b6648a1afa0605\": container with ID starting with 43a776e44312ae6e2bfc39107d915a25a6ca73b700b22e89a6b6648a1afa0605 not found: ID does not exist"
Sep 08 02:16:00.342346 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:00.342241    2547 scope.go:117] "RemoveContainer" containerID="e5b937ae156ff34b0b4492beddcb479329dcbee74fac44939a4131d399884ff7"
Sep 08 02:16:00.350189 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:00.350168    2547 scope.go:117] "RemoveContainer" containerID="e5b937ae156ff34b0b4492beddcb479329dcbee74fac44939a4131d399884ff7"
Sep 08 02:16:00.350474 ip-10-0-142-78 kubenswrapper[2547]: E0908 02:16:00.350453    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5b937ae156ff34b0b4492beddcb479329dcbee74fac44939a4131d399884ff7\": container with ID starting with e5b937ae156ff34b0b4492beddcb479329dcbee74fac44939a4131d399884ff7 not found: ID does not exist" containerID="e5b937ae156ff34b0b4492beddcb479329dcbee74fac44939a4131d399884ff7"
Sep 08 02:16:00.350538 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:00.350481    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5b937ae156ff34b0b4492beddcb479329dcbee74fac44939a4131d399884ff7"} err="failed to get container status \"e5b937ae156ff34b0b4492beddcb479329dcbee74fac44939a4131d399884ff7\": rpc error: code = NotFound desc = could not find container \"e5b937ae156ff34b0b4492beddcb479329dcbee74fac44939a4131d399884ff7\": container with ID starting with e5b937ae156ff34b0b4492beddcb479329dcbee74fac44939a4131d399884ff7 not found: ID does not exist"
Sep 08 02:16:00.356270 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:00.356243    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5d27de-78684b5958-vtg65"]
Sep 08 02:16:00.357901 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:00.357880    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5d27de-78684b5958-vtg65"]
Sep 08 02:16:00.366892 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:00.366862    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5d27de-58656496c5-qzvbz"]
Sep 08 02:16:00.370666 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:00.370632    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5d27de-58656496c5-qzvbz"]
Sep 08 02:16:00.425669 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:00.425632    2547 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1dabb5b-1a34-4dc3-a7c5-eaa6db21ce32-plugins-config-volume\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:16:00.425669 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:00.425666    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzftx\" (UniqueName: \"kubernetes.io/projected/d6d26402-535b-4f5d-8cff-91b3bc126699-kube-api-access-wzftx\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:16:00.425874 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:00.425683    2547 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6d26402-535b-4f5d-8cff-91b3bc126699-plugins-config-volume\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:16:00.425874 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:00.425692    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hdk2\" (UniqueName: \"kubernetes.io/projected/d1dabb5b-1a34-4dc3-a7c5-eaa6db21ce32-kube-api-access-2hdk2\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:16:00.994105 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:00.994069    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9948a4f7-6893-45ef-85df-9d339b51cdb2" path="/var/lib/kubelet/pods/9948a4f7-6893-45ef-85df-9d339b51cdb2/volumes"
Sep 08 02:16:00.994483 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:00.994427    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1dabb5b-1a34-4dc3-a7c5-eaa6db21ce32" path="/var/lib/kubelet/pods/d1dabb5b-1a34-4dc3-a7c5-eaa6db21ce32/volumes"
Sep 08 02:16:00.994750 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:00.994736    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6d26402-535b-4f5d-8cff-91b3bc126699" path="/var/lib/kubelet/pods/d6d26402-535b-4f5d-8cff-91b3bc126699/volumes"
Sep 08 02:16:03.213993 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:03.213948    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-cf06cb/noten-model-cf06cb-kserve-5d65f6d79f-xwz2r" podUID="a07a202b-95da-4d7f-aa33-c1d939e68ba5" containerName="main" containerID="cri-o://3f690e8ecf1d43bdcece3c116ada6367707085088cfa344e46dea760dc19eba5" gracePeriod=45
Sep 08 02:16:03.350405 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:03.350371    2547 generic.go:356] "Generic (PLEG): container finished" podID="a07a202b-95da-4d7f-aa33-c1d939e68ba5" containerID="3f690e8ecf1d43bdcece3c116ada6367707085088cfa344e46dea760dc19eba5" exitCode=0
Sep 08 02:16:03.350551 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:03.350421    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-cf06cb/noten-model-cf06cb-kserve-5d65f6d79f-xwz2r" event={"ID":"a07a202b-95da-4d7f-aa33-c1d939e68ba5","Type":"ContainerDied","Data":"3f690e8ecf1d43bdcece3c116ada6367707085088cfa344e46dea760dc19eba5"}
Sep 08 02:16:03.471874 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:03.471805    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-cf06cb/noten-model-cf06cb-kserve-5d65f6d79f-xwz2r"
Sep 08 02:16:03.550631 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:03.550592    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a07a202b-95da-4d7f-aa33-c1d939e68ba5-tls-certs\") pod \"a07a202b-95da-4d7f-aa33-c1d939e68ba5\" (UID: \"a07a202b-95da-4d7f-aa33-c1d939e68ba5\") "
Sep 08 02:16:03.550839 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:03.550658    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a07a202b-95da-4d7f-aa33-c1d939e68ba5-tmp-dir\") pod \"a07a202b-95da-4d7f-aa33-c1d939e68ba5\" (UID: \"a07a202b-95da-4d7f-aa33-c1d939e68ba5\") "
Sep 08 02:16:03.550839 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:03.550687    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a07a202b-95da-4d7f-aa33-c1d939e68ba5-dshm\") pod \"a07a202b-95da-4d7f-aa33-c1d939e68ba5\" (UID: \"a07a202b-95da-4d7f-aa33-c1d939e68ba5\") "
Sep 08 02:16:03.550839 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:03.550721    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a07a202b-95da-4d7f-aa33-c1d939e68ba5-model-cache\") pod \"a07a202b-95da-4d7f-aa33-c1d939e68ba5\" (UID: \"a07a202b-95da-4d7f-aa33-c1d939e68ba5\") "
Sep 08 02:16:03.550839 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:03.550752    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x8pdf\" (UniqueName: \"kubernetes.io/projected/a07a202b-95da-4d7f-aa33-c1d939e68ba5-kube-api-access-x8pdf\") pod \"a07a202b-95da-4d7f-aa33-c1d939e68ba5\" (UID: \"a07a202b-95da-4d7f-aa33-c1d939e68ba5\") "
Sep 08 02:16:03.550839 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:03.550775    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a07a202b-95da-4d7f-aa33-c1d939e68ba5-home\") pod \"a07a202b-95da-4d7f-aa33-c1d939e68ba5\" (UID: \"a07a202b-95da-4d7f-aa33-c1d939e68ba5\") "
Sep 08 02:16:03.551086 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:03.550997    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a07a202b-95da-4d7f-aa33-c1d939e68ba5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a07a202b-95da-4d7f-aa33-c1d939e68ba5" (UID: "a07a202b-95da-4d7f-aa33-c1d939e68ba5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 02:16:03.551086 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:03.551025    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a07a202b-95da-4d7f-aa33-c1d939e68ba5-model-cache" (OuterVolumeSpecName: "model-cache") pod "a07a202b-95da-4d7f-aa33-c1d939e68ba5" (UID: "a07a202b-95da-4d7f-aa33-c1d939e68ba5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 02:16:03.551086 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:03.551078    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a07a202b-95da-4d7f-aa33-c1d939e68ba5-home" (OuterVolumeSpecName: "home") pod "a07a202b-95da-4d7f-aa33-c1d939e68ba5" (UID: "a07a202b-95da-4d7f-aa33-c1d939e68ba5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 02:16:03.552899 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:03.552863    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a07a202b-95da-4d7f-aa33-c1d939e68ba5-dshm" (OuterVolumeSpecName: "dshm") pod "a07a202b-95da-4d7f-aa33-c1d939e68ba5" (UID: "a07a202b-95da-4d7f-aa33-c1d939e68ba5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 02:16:03.552992 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:03.552954    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a07a202b-95da-4d7f-aa33-c1d939e68ba5-kube-api-access-x8pdf" (OuterVolumeSpecName: "kube-api-access-x8pdf") pod "a07a202b-95da-4d7f-aa33-c1d939e68ba5" (UID: "a07a202b-95da-4d7f-aa33-c1d939e68ba5"). InnerVolumeSpecName "kube-api-access-x8pdf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 02:16:03.552992 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:03.552958    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a07a202b-95da-4d7f-aa33-c1d939e68ba5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a07a202b-95da-4d7f-aa33-c1d939e68ba5" (UID: "a07a202b-95da-4d7f-aa33-c1d939e68ba5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 02:16:03.651669 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:03.651628    2547 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a07a202b-95da-4d7f-aa33-c1d939e68ba5-tls-certs\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:16:03.651669 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:03.651663    2547 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a07a202b-95da-4d7f-aa33-c1d939e68ba5-tmp-dir\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:16:03.651669 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:03.651672    2547 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a07a202b-95da-4d7f-aa33-c1d939e68ba5-dshm\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:16:03.651669 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:03.651680    2547 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a07a202b-95da-4d7f-aa33-c1d939e68ba5-model-cache\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:16:03.651934 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:03.651689    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8pdf\" (UniqueName: \"kubernetes.io/projected/a07a202b-95da-4d7f-aa33-c1d939e68ba5-kube-api-access-x8pdf\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:16:03.651934 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:03.651697    2547 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a07a202b-95da-4d7f-aa33-c1d939e68ba5-home\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:16:04.185690 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:04.185650    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-8d800a/test-model-8d800a-kserve-7f8cd79b9f-kv4m7" podUID="be861d49-5caf-4a99-b5bf-2b53f85dd905" containerName="main" containerID="cri-o://b30cffb48d7756ed8640d91a4d725405021c35f6a058ec267d52257e18aa251e" gracePeriod=45
Sep 08 02:16:04.357158 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:04.357107    2547 generic.go:356] "Generic (PLEG): container finished" podID="be861d49-5caf-4a99-b5bf-2b53f85dd905" containerID="b30cffb48d7756ed8640d91a4d725405021c35f6a058ec267d52257e18aa251e" exitCode=0
Sep 08 02:16:04.357568 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:04.357172    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-8d800a/test-model-8d800a-kserve-7f8cd79b9f-kv4m7" event={"ID":"be861d49-5caf-4a99-b5bf-2b53f85dd905","Type":"ContainerDied","Data":"b30cffb48d7756ed8640d91a4d725405021c35f6a058ec267d52257e18aa251e"}
Sep 08 02:16:04.358506 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:04.358487    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-cf06cb/noten-model-cf06cb-kserve-5d65f6d79f-xwz2r"
Sep 08 02:16:04.358506 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:04.358490    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-cf06cb/noten-model-cf06cb-kserve-5d65f6d79f-xwz2r" event={"ID":"a07a202b-95da-4d7f-aa33-c1d939e68ba5","Type":"ContainerDied","Data":"131a43a8685b0ba5d304e63fefe458533f1df495b342e36836a29d29576ebac4"}
Sep 08 02:16:04.358676 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:04.358541    2547 scope.go:117] "RemoveContainer" containerID="3f690e8ecf1d43bdcece3c116ada6367707085088cfa344e46dea760dc19eba5"
Sep 08 02:16:04.381813 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:04.381760    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-cf06cb/noten-model-cf06cb-kserve-5d65f6d79f-xwz2r"]
Sep 08 02:16:04.384737 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:04.384711    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-cf06cb/noten-model-cf06cb-kserve-5d65f6d79f-xwz2r"]
Sep 08 02:16:04.435624 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:04.435600    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-8d800a/test-model-8d800a-kserve-7f8cd79b9f-kv4m7"
Sep 08 02:16:04.557953 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:04.557851    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be861d49-5caf-4a99-b5bf-2b53f85dd905-home\") pod \"be861d49-5caf-4a99-b5bf-2b53f85dd905\" (UID: \"be861d49-5caf-4a99-b5bf-2b53f85dd905\") "
Sep 08 02:16:04.557953 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:04.557893    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be861d49-5caf-4a99-b5bf-2b53f85dd905-dshm\") pod \"be861d49-5caf-4a99-b5bf-2b53f85dd905\" (UID: \"be861d49-5caf-4a99-b5bf-2b53f85dd905\") "
Sep 08 02:16:04.557953 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:04.557949    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be861d49-5caf-4a99-b5bf-2b53f85dd905-tmp-dir\") pod \"be861d49-5caf-4a99-b5bf-2b53f85dd905\" (UID: \"be861d49-5caf-4a99-b5bf-2b53f85dd905\") "
Sep 08 02:16:04.558274 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:04.558007    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be861d49-5caf-4a99-b5bf-2b53f85dd905-tls-certs\") pod \"be861d49-5caf-4a99-b5bf-2b53f85dd905\" (UID: \"be861d49-5caf-4a99-b5bf-2b53f85dd905\") "
Sep 08 02:16:04.558274 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:04.558037    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x57p4\" (UniqueName: \"kubernetes.io/projected/be861d49-5caf-4a99-b5bf-2b53f85dd905-kube-api-access-x57p4\") pod \"be861d49-5caf-4a99-b5bf-2b53f85dd905\" (UID: \"be861d49-5caf-4a99-b5bf-2b53f85dd905\") "
Sep 08 02:16:04.558274 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:04.558077    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be861d49-5caf-4a99-b5bf-2b53f85dd905-model-cache\") pod \"be861d49-5caf-4a99-b5bf-2b53f85dd905\" (UID: \"be861d49-5caf-4a99-b5bf-2b53f85dd905\") "
Sep 08 02:16:04.558419 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:04.558267    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be861d49-5caf-4a99-b5bf-2b53f85dd905-home" (OuterVolumeSpecName: "home") pod "be861d49-5caf-4a99-b5bf-2b53f85dd905" (UID: "be861d49-5caf-4a99-b5bf-2b53f85dd905"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 02:16:04.558419 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:04.558296    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be861d49-5caf-4a99-b5bf-2b53f85dd905-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "be861d49-5caf-4a99-b5bf-2b53f85dd905" (UID: "be861d49-5caf-4a99-b5bf-2b53f85dd905"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 02:16:04.558419 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:04.558370    2547 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be861d49-5caf-4a99-b5bf-2b53f85dd905-tmp-dir\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:16:04.558419 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:04.558389    2547 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be861d49-5caf-4a99-b5bf-2b53f85dd905-home\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:16:04.558567 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:04.558536    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be861d49-5caf-4a99-b5bf-2b53f85dd905-model-cache" (OuterVolumeSpecName: "model-cache") pod "be861d49-5caf-4a99-b5bf-2b53f85dd905" (UID: "be861d49-5caf-4a99-b5bf-2b53f85dd905"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 02:16:04.560362 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:04.560340    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be861d49-5caf-4a99-b5bf-2b53f85dd905-dshm" (OuterVolumeSpecName: "dshm") pod "be861d49-5caf-4a99-b5bf-2b53f85dd905" (UID: "be861d49-5caf-4a99-b5bf-2b53f85dd905"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 02:16:04.560628 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:04.560606    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be861d49-5caf-4a99-b5bf-2b53f85dd905-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "be861d49-5caf-4a99-b5bf-2b53f85dd905" (UID: "be861d49-5caf-4a99-b5bf-2b53f85dd905"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 02:16:04.560699 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:04.560627    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be861d49-5caf-4a99-b5bf-2b53f85dd905-kube-api-access-x57p4" (OuterVolumeSpecName: "kube-api-access-x57p4") pod "be861d49-5caf-4a99-b5bf-2b53f85dd905" (UID: "be861d49-5caf-4a99-b5bf-2b53f85dd905"). InnerVolumeSpecName "kube-api-access-x57p4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 02:16:04.659435 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:04.659394    2547 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be861d49-5caf-4a99-b5bf-2b53f85dd905-model-cache\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:16:04.659435 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:04.659427    2547 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be861d49-5caf-4a99-b5bf-2b53f85dd905-dshm\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:16:04.659435 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:04.659436    2547 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be861d49-5caf-4a99-b5bf-2b53f85dd905-tls-certs\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:16:04.659435 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:04.659444    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x57p4\" (UniqueName: \"kubernetes.io/projected/be861d49-5caf-4a99-b5bf-2b53f85dd905-kube-api-access-x57p4\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:16:04.994349 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:04.994315    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a07a202b-95da-4d7f-aa33-c1d939e68ba5" path="/var/lib/kubelet/pods/a07a202b-95da-4d7f-aa33-c1d939e68ba5/volumes"
Sep 08 02:16:05.364361 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:05.364266    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-8d800a/test-model-8d800a-kserve-7f8cd79b9f-kv4m7" event={"ID":"be861d49-5caf-4a99-b5bf-2b53f85dd905","Type":"ContainerDied","Data":"6752e064ecbec7bb291b257a6bc52dc2bc42377b6faff2d7efa751c2bfa5bb01"}
Sep 08 02:16:05.364361 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:05.364278    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-8d800a/test-model-8d800a-kserve-7f8cd79b9f-kv4m7"
Sep 08 02:16:05.364912 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:05.364318    2547 scope.go:117] "RemoveContainer" containerID="b30cffb48d7756ed8640d91a4d725405021c35f6a058ec267d52257e18aa251e"
Sep 08 02:16:05.380360 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:05.380329    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-8d800a/test-model-8d800a-kserve-7f8cd79b9f-kv4m7"]
Sep 08 02:16:05.383762 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:05.383733    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-8d800a/test-model-8d800a-kserve-7f8cd79b9f-kv4m7"]
Sep 08 02:16:06.994016 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:06.993978    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be861d49-5caf-4a99-b5bf-2b53f85dd905" path="/var/lib/kubelet/pods/be861d49-5caf-4a99-b5bf-2b53f85dd905/volumes"
Sep 08 02:16:07.829970 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:07.829930    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8fa74948-6v4lx"]
Sep 08 02:16:07.830292 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:07.830279    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a07a202b-95da-4d7f-aa33-c1d939e68ba5" containerName="main"
Sep 08 02:16:07.830332 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:07.830293    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="a07a202b-95da-4d7f-aa33-c1d939e68ba5" containerName="main"
Sep 08 02:16:07.830332 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:07.830301    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6d26402-535b-4f5d-8cff-91b3bc126699" containerName="payload-pre-processing"
Sep 08 02:16:07.830332 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:07.830307    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6d26402-535b-4f5d-8cff-91b3bc126699" containerName="payload-pre-processing"
Sep 08 02:16:07.830332 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:07.830319    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1dabb5b-1a34-4dc3-a7c5-eaa6db21ce32" containerName="payload-processing"
Sep 08 02:16:07.830332 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:07.830325    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1dabb5b-1a34-4dc3-a7c5-eaa6db21ce32" containerName="payload-processing"
Sep 08 02:16:07.830497 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:07.830336    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be861d49-5caf-4a99-b5bf-2b53f85dd905" containerName="main"
Sep 08 02:16:07.830497 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:07.830341    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="be861d49-5caf-4a99-b5bf-2b53f85dd905" containerName="main"
Sep 08 02:16:07.830497 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:07.830391    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="a07a202b-95da-4d7f-aa33-c1d939e68ba5" containerName="main"
Sep 08 02:16:07.830497 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:07.830400    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1dabb5b-1a34-4dc3-a7c5-eaa6db21ce32" containerName="payload-processing"
Sep 08 02:16:07.830497 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:07.830408    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="be861d49-5caf-4a99-b5bf-2b53f85dd905" containerName="main"
Sep 08 02:16:07.830497 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:07.830415    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6d26402-535b-4f5d-8cff-91b3bc126699" containerName="payload-pre-processing"
Sep 08 02:16:07.835033 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:07.835010    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8fa74948-6v4lx"
Sep 08 02:16:07.837476 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:07.837442    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 02:16:07.838314 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:07.838286    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8mbwc\""
Sep 08 02:16:07.840871 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:07.840814    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8fa74948-6v4lx"]
Sep 08 02:16:07.887287 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:07.887243    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aac04da1-ec35-4b47-8efe-f2d34fcccbf4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-8fa74948-6v4lx\" (UID: \"aac04da1-ec35-4b47-8efe-f2d34fcccbf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8fa74948-6v4lx"
Sep 08 02:16:07.988606 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:07.988564    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aac04da1-ec35-4b47-8efe-f2d34fcccbf4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-8fa74948-6v4lx\" (UID: \"aac04da1-ec35-4b47-8efe-f2d34fcccbf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8fa74948-6v4lx"
Sep 08 02:16:07.989235 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:07.989215    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aac04da1-ec35-4b47-8efe-f2d34fcccbf4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-8fa74948-6v4lx\" (UID: \"aac04da1-ec35-4b47-8efe-f2d34fcccbf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8fa74948-6v4lx"
Sep 08 02:16:08.147231 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:08.147198    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8fa74948-6v4lx"
Sep 08 02:16:08.417473 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:08.417393    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8fa74948-6v4lx"]
Sep 08 02:16:08.421239 ip-10-0-142-78 kubenswrapper[2547]: W0908 02:16:08.421207    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaac04da1_ec35_4b47_8efe_f2d34fcccbf4.slice/crio-04c4e09c3b684eb04d90221f8bbc11ccf13616a976da3aa4be0483ac1b9a7507 WatchSource:0}: Error finding container 04c4e09c3b684eb04d90221f8bbc11ccf13616a976da3aa4be0483ac1b9a7507: Status 404 returned error can't find the container with id 04c4e09c3b684eb04d90221f8bbc11ccf13616a976da3aa4be0483ac1b9a7507
Sep 08 02:16:09.386998 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:09.386901    2547 generic.go:356] "Generic (PLEG): container finished" podID="aac04da1-ec35-4b47-8efe-f2d34fcccbf4" containerID="962d9a8e3f65f08273877d357d19a2e24594610dafa4f7134b9977d3bced9bf8" exitCode=0
Sep 08 02:16:09.387416 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:09.386992    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8fa74948-6v4lx" event={"ID":"aac04da1-ec35-4b47-8efe-f2d34fcccbf4","Type":"ContainerDied","Data":"962d9a8e3f65f08273877d357d19a2e24594610dafa4f7134b9977d3bced9bf8"}
Sep 08 02:16:09.387416 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:09.387075    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8fa74948-6v4lx" event={"ID":"aac04da1-ec35-4b47-8efe-f2d34fcccbf4","Type":"ContainerStarted","Data":"04c4e09c3b684eb04d90221f8bbc11ccf13616a976da3aa4be0483ac1b9a7507"}
Sep 08 02:16:10.514540 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:10.514511    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8fa74948-6v4lx"
Sep 08 02:16:10.610476 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:10.610439    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aac04da1-ec35-4b47-8efe-f2d34fcccbf4-maas-api-service-ca\") pod \"aac04da1-ec35-4b47-8efe-f2d34fcccbf4\" (UID: \"aac04da1-ec35-4b47-8efe-f2d34fcccbf4\") "
Sep 08 02:16:10.610842 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:10.610820    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aac04da1-ec35-4b47-8efe-f2d34fcccbf4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "aac04da1-ec35-4b47-8efe-f2d34fcccbf4" (UID: "aac04da1-ec35-4b47-8efe-f2d34fcccbf4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 02:16:10.711514 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:10.711419    2547 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aac04da1-ec35-4b47-8efe-f2d34fcccbf4-maas-api-service-ca\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:16:11.396311 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:11.396283    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8fa74948-6v4lx"
Sep 08 02:16:11.396451 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:11.396329    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8fa74948-6v4lx" event={"ID":"aac04da1-ec35-4b47-8efe-f2d34fcccbf4","Type":"ContainerDied","Data":"04c4e09c3b684eb04d90221f8bbc11ccf13616a976da3aa4be0483ac1b9a7507"}
Sep 08 02:16:11.396451 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:11.396372    2547 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="04c4e09c3b684eb04d90221f8bbc11ccf13616a976da3aa4be0483ac1b9a7507"
Sep 08 02:16:17.418461 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:17.418413    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8d800a-6b455db645-cxmzp"]
Sep 08 02:16:17.419019 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:17.418718    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-8d800a-6b455db645-cxmzp" podUID="0318e043-46c2-435c-ba36-a07110b8e456" containerName="payload-processing" containerID="cri-o://d6257168b5093ff502ce5537899287ab84824059eab8f18c1f591b6929a4ed6b" gracePeriod=30
Sep 08 02:16:17.428570 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:17.427865    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8d800a-7fd6f4554-wz878"]
Sep 08 02:16:17.428570 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:17.428309    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8d800a-7fd6f4554-wz878" podUID="5d8fcaa0-2ffb-4931-931e-2b0d3f06d083" containerName="payload-pre-processing" containerID="cri-o://56c4ead1c154da2f43a1be9e896a70960e52900756c7bfd45850116eec4ef6a9" gracePeriod=30
Sep 08 02:16:17.691341 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:17.691316    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8d800a-6b455db645-cxmzp"
Sep 08 02:16:17.694279 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:17.694258    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8d800a-7fd6f4554-wz878"
Sep 08 02:16:17.767394 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:17.767351    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lmjlf\" (UniqueName: \"kubernetes.io/projected/5d8fcaa0-2ffb-4931-931e-2b0d3f06d083-kube-api-access-lmjlf\") pod \"5d8fcaa0-2ffb-4931-931e-2b0d3f06d083\" (UID: \"5d8fcaa0-2ffb-4931-931e-2b0d3f06d083\") "
Sep 08 02:16:17.767561 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:17.767476    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0318e043-46c2-435c-ba36-a07110b8e456-plugins-config-volume\") pod \"0318e043-46c2-435c-ba36-a07110b8e456\" (UID: \"0318e043-46c2-435c-ba36-a07110b8e456\") "
Sep 08 02:16:17.767561 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:17.767501    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sk8dt\" (UniqueName: \"kubernetes.io/projected/0318e043-46c2-435c-ba36-a07110b8e456-kube-api-access-sk8dt\") pod \"0318e043-46c2-435c-ba36-a07110b8e456\" (UID: \"0318e043-46c2-435c-ba36-a07110b8e456\") "
Sep 08 02:16:17.767561 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:17.767554    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d8fcaa0-2ffb-4931-931e-2b0d3f06d083-plugins-config-volume\") pod \"5d8fcaa0-2ffb-4931-931e-2b0d3f06d083\" (UID: \"5d8fcaa0-2ffb-4931-931e-2b0d3f06d083\") "
Sep 08 02:16:17.767878 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:17.767852    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0318e043-46c2-435c-ba36-a07110b8e456-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0318e043-46c2-435c-ba36-a07110b8e456" (UID: "0318e043-46c2-435c-ba36-a07110b8e456"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 02:16:17.767957 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:17.767926    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d8fcaa0-2ffb-4931-931e-2b0d3f06d083-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5d8fcaa0-2ffb-4931-931e-2b0d3f06d083" (UID: "5d8fcaa0-2ffb-4931-931e-2b0d3f06d083"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 02:16:17.769566 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:17.769542    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d8fcaa0-2ffb-4931-931e-2b0d3f06d083-kube-api-access-lmjlf" (OuterVolumeSpecName: "kube-api-access-lmjlf") pod "5d8fcaa0-2ffb-4931-931e-2b0d3f06d083" (UID: "5d8fcaa0-2ffb-4931-931e-2b0d3f06d083"). InnerVolumeSpecName "kube-api-access-lmjlf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 02:16:17.769703 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:17.769676    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0318e043-46c2-435c-ba36-a07110b8e456-kube-api-access-sk8dt" (OuterVolumeSpecName: "kube-api-access-sk8dt") pod "0318e043-46c2-435c-ba36-a07110b8e456" (UID: "0318e043-46c2-435c-ba36-a07110b8e456"). InnerVolumeSpecName "kube-api-access-sk8dt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 02:16:17.868231 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:17.868187    2547 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d8fcaa0-2ffb-4931-931e-2b0d3f06d083-plugins-config-volume\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:16:17.868231 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:17.868216    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmjlf\" (UniqueName: \"kubernetes.io/projected/5d8fcaa0-2ffb-4931-931e-2b0d3f06d083-kube-api-access-lmjlf\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:16:17.868231 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:17.868226    2547 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0318e043-46c2-435c-ba36-a07110b8e456-plugins-config-volume\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:16:17.868231 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:17.868235    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sk8dt\" (UniqueName: \"kubernetes.io/projected/0318e043-46c2-435c-ba36-a07110b8e456-kube-api-access-sk8dt\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:16:18.431010 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:18.430969    2547 generic.go:356] "Generic (PLEG): container finished" podID="0318e043-46c2-435c-ba36-a07110b8e456" containerID="d6257168b5093ff502ce5537899287ab84824059eab8f18c1f591b6929a4ed6b" exitCode=0
Sep 08 02:16:18.431499 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:18.431038    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8d800a-6b455db645-cxmzp"
Sep 08 02:16:18.431499 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:18.431050    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8d800a-6b455db645-cxmzp" event={"ID":"0318e043-46c2-435c-ba36-a07110b8e456","Type":"ContainerDied","Data":"d6257168b5093ff502ce5537899287ab84824059eab8f18c1f591b6929a4ed6b"}
Sep 08 02:16:18.431499 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:18.431105    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8d800a-6b455db645-cxmzp" event={"ID":"0318e043-46c2-435c-ba36-a07110b8e456","Type":"ContainerDied","Data":"c615d5505b4f17084678de3e93dff99caac433de3413987145dd06d5d9ceb789"}
Sep 08 02:16:18.431499 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:18.431150    2547 scope.go:117] "RemoveContainer" containerID="d6257168b5093ff502ce5537899287ab84824059eab8f18c1f591b6929a4ed6b"
Sep 08 02:16:18.432802 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:18.432506    2547 generic.go:356] "Generic (PLEG): container finished" podID="5d8fcaa0-2ffb-4931-931e-2b0d3f06d083" containerID="56c4ead1c154da2f43a1be9e896a70960e52900756c7bfd45850116eec4ef6a9" exitCode=0
Sep 08 02:16:18.432802 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:18.432566    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8d800a-7fd6f4554-wz878"
Sep 08 02:16:18.432802 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:18.432562    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8d800a-7fd6f4554-wz878" event={"ID":"5d8fcaa0-2ffb-4931-931e-2b0d3f06d083","Type":"ContainerDied","Data":"56c4ead1c154da2f43a1be9e896a70960e52900756c7bfd45850116eec4ef6a9"}
Sep 08 02:16:18.432802 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:18.432635    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8d800a-7fd6f4554-wz878" event={"ID":"5d8fcaa0-2ffb-4931-931e-2b0d3f06d083","Type":"ContainerDied","Data":"3f9d22a6cdbb6ee914ae48f2e0371d7b873e4ceca00065ca56d5834b00706180"}
Sep 08 02:16:18.445256 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:18.445238    2547 scope.go:117] "RemoveContainer" containerID="d6257168b5093ff502ce5537899287ab84824059eab8f18c1f591b6929a4ed6b"
Sep 08 02:16:18.445604 ip-10-0-142-78 kubenswrapper[2547]: E0908 02:16:18.445582    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6257168b5093ff502ce5537899287ab84824059eab8f18c1f591b6929a4ed6b\": container with ID starting with d6257168b5093ff502ce5537899287ab84824059eab8f18c1f591b6929a4ed6b not found: ID does not exist" containerID="d6257168b5093ff502ce5537899287ab84824059eab8f18c1f591b6929a4ed6b"
Sep 08 02:16:18.445683 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:18.445617    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6257168b5093ff502ce5537899287ab84824059eab8f18c1f591b6929a4ed6b"} err="failed to get container status \"d6257168b5093ff502ce5537899287ab84824059eab8f18c1f591b6929a4ed6b\": rpc error: code = NotFound desc = could not find container \"d6257168b5093ff502ce5537899287ab84824059eab8f18c1f591b6929a4ed6b\": container with ID starting with d6257168b5093ff502ce5537899287ab84824059eab8f18c1f591b6929a4ed6b not found: ID does not exist"
Sep 08 02:16:18.445683 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:18.445639    2547 scope.go:117] "RemoveContainer" containerID="56c4ead1c154da2f43a1be9e896a70960e52900756c7bfd45850116eec4ef6a9"
Sep 08 02:16:18.453100 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:18.453080    2547 scope.go:117] "RemoveContainer" containerID="56c4ead1c154da2f43a1be9e896a70960e52900756c7bfd45850116eec4ef6a9"
Sep 08 02:16:18.453393 ip-10-0-142-78 kubenswrapper[2547]: E0908 02:16:18.453374    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56c4ead1c154da2f43a1be9e896a70960e52900756c7bfd45850116eec4ef6a9\": container with ID starting with 56c4ead1c154da2f43a1be9e896a70960e52900756c7bfd45850116eec4ef6a9 not found: ID does not exist" containerID="56c4ead1c154da2f43a1be9e896a70960e52900756c7bfd45850116eec4ef6a9"
Sep 08 02:16:18.453445 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:18.453400    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56c4ead1c154da2f43a1be9e896a70960e52900756c7bfd45850116eec4ef6a9"} err="failed to get container status \"56c4ead1c154da2f43a1be9e896a70960e52900756c7bfd45850116eec4ef6a9\": rpc error: code = NotFound desc = could not find container \"56c4ead1c154da2f43a1be9e896a70960e52900756c7bfd45850116eec4ef6a9\": container with ID starting with 56c4ead1c154da2f43a1be9e896a70960e52900756c7bfd45850116eec4ef6a9 not found: ID does not exist"
Sep 08 02:16:18.456535 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:18.456508    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8d800a-7fd6f4554-wz878"]
Sep 08 02:16:18.459898 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:18.459875    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8d800a-7fd6f4554-wz878"]
Sep 08 02:16:18.469609 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:18.469579    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8d800a-6b455db645-cxmzp"]
Sep 08 02:16:18.473001 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:18.472977    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8d800a-6b455db645-cxmzp"]
Sep 08 02:16:18.634405 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:18.634369    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8fa74948-7f464b7cd9-75cxz"]
Sep 08 02:16:18.634731 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:18.634635    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-8fa74948-7f464b7cd9-75cxz" podUID="bf419798-9b6d-465a-8bee-5301c8a52031" containerName="payload-processing" containerID="cri-o://11f04e1f3c3c651e89993e33471edc7f149c3530ec7195961dae636466fa88af" gracePeriod=30
Sep 08 02:16:18.639280 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:18.639252    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8fa74948-6b998c9754-xw5cs"]
Sep 08 02:16:18.639517 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:18.639491    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-8fa74948-6b998c9754-xw5cs" podUID="293a49f1-6902-4192-80cd-105f9594bf89" containerName="payload-pre-processing" containerID="cri-o://74a8cf9633e1570c02f9d0cc265e962e93b6470f002ef9eb8ab2fb22459fe1e6" gracePeriod=30
Sep 08 02:16:18.852068 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:18.852027    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8fa74948-6v4lx"]
Sep 08 02:16:18.865282 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:18.865251    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8fa74948-6v4lx"]
Sep 08 02:16:18.909786 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:18.909762    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8fa74948-7f464b7cd9-75cxz"
Sep 08 02:16:18.912476 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:18.912457    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8fa74948-6b998c9754-xw5cs"
Sep 08 02:16:18.977591 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:18.977501    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wrv5b\" (UniqueName: \"kubernetes.io/projected/293a49f1-6902-4192-80cd-105f9594bf89-kube-api-access-wrv5b\") pod \"293a49f1-6902-4192-80cd-105f9594bf89\" (UID: \"293a49f1-6902-4192-80cd-105f9594bf89\") "
Sep 08 02:16:18.977591 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:18.977536    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf419798-9b6d-465a-8bee-5301c8a52031-plugins-config-volume\") pod \"bf419798-9b6d-465a-8bee-5301c8a52031\" (UID: \"bf419798-9b6d-465a-8bee-5301c8a52031\") "
Sep 08 02:16:18.977591 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:18.977567    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/293a49f1-6902-4192-80cd-105f9594bf89-plugins-config-volume\") pod \"293a49f1-6902-4192-80cd-105f9594bf89\" (UID: \"293a49f1-6902-4192-80cd-105f9594bf89\") "
Sep 08 02:16:18.977878 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:18.977624    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7zlm4\" (UniqueName: \"kubernetes.io/projected/bf419798-9b6d-465a-8bee-5301c8a52031-kube-api-access-7zlm4\") pod \"bf419798-9b6d-465a-8bee-5301c8a52031\" (UID: \"bf419798-9b6d-465a-8bee-5301c8a52031\") "
Sep 08 02:16:18.977961 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:18.977912    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/293a49f1-6902-4192-80cd-105f9594bf89-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "293a49f1-6902-4192-80cd-105f9594bf89" (UID: "293a49f1-6902-4192-80cd-105f9594bf89"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 02:16:18.978102 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:18.978068    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf419798-9b6d-465a-8bee-5301c8a52031-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bf419798-9b6d-465a-8bee-5301c8a52031" (UID: "bf419798-9b6d-465a-8bee-5301c8a52031"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 02:16:18.979740 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:18.979717    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf419798-9b6d-465a-8bee-5301c8a52031-kube-api-access-7zlm4" (OuterVolumeSpecName: "kube-api-access-7zlm4") pod "bf419798-9b6d-465a-8bee-5301c8a52031" (UID: "bf419798-9b6d-465a-8bee-5301c8a52031"). InnerVolumeSpecName "kube-api-access-7zlm4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 02:16:18.979831 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:18.979755    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/293a49f1-6902-4192-80cd-105f9594bf89-kube-api-access-wrv5b" (OuterVolumeSpecName: "kube-api-access-wrv5b") pod "293a49f1-6902-4192-80cd-105f9594bf89" (UID: "293a49f1-6902-4192-80cd-105f9594bf89"). InnerVolumeSpecName "kube-api-access-wrv5b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 02:16:18.993568 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:18.993541    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0318e043-46c2-435c-ba36-a07110b8e456" path="/var/lib/kubelet/pods/0318e043-46c2-435c-ba36-a07110b8e456/volumes"
Sep 08 02:16:18.993883 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:18.993870    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d8fcaa0-2ffb-4931-931e-2b0d3f06d083" path="/var/lib/kubelet/pods/5d8fcaa0-2ffb-4931-931e-2b0d3f06d083/volumes"
Sep 08 02:16:18.994194 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:18.994181    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aac04da1-ec35-4b47-8efe-f2d34fcccbf4" path="/var/lib/kubelet/pods/aac04da1-ec35-4b47-8efe-f2d34fcccbf4/volumes"
Sep 08 02:16:19.078233 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:19.078199    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrv5b\" (UniqueName: \"kubernetes.io/projected/293a49f1-6902-4192-80cd-105f9594bf89-kube-api-access-wrv5b\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:16:19.078233 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:19.078227    2547 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf419798-9b6d-465a-8bee-5301c8a52031-plugins-config-volume\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:16:19.078233 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:19.078237    2547 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/293a49f1-6902-4192-80cd-105f9594bf89-plugins-config-volume\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:16:19.078458 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:19.078245    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zlm4\" (UniqueName: \"kubernetes.io/projected/bf419798-9b6d-465a-8bee-5301c8a52031-kube-api-access-7zlm4\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:16:19.439652 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:19.439619    2547 generic.go:356] "Generic (PLEG): container finished" podID="293a49f1-6902-4192-80cd-105f9594bf89" containerID="74a8cf9633e1570c02f9d0cc265e962e93b6470f002ef9eb8ab2fb22459fe1e6" exitCode=0
Sep 08 02:16:19.440030 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:19.439669    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8fa74948-6b998c9754-xw5cs"
Sep 08 02:16:19.440030 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:19.439703    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8fa74948-6b998c9754-xw5cs" event={"ID":"293a49f1-6902-4192-80cd-105f9594bf89","Type":"ContainerDied","Data":"74a8cf9633e1570c02f9d0cc265e962e93b6470f002ef9eb8ab2fb22459fe1e6"}
Sep 08 02:16:19.440030 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:19.439758    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8fa74948-6b998c9754-xw5cs" event={"ID":"293a49f1-6902-4192-80cd-105f9594bf89","Type":"ContainerDied","Data":"5adb8c0a8d4f6f58fadf3bea1b3814478d9a823e3ae3cc9d5644f304ebb971ef"}
Sep 08 02:16:19.440030 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:19.439787    2547 scope.go:117] "RemoveContainer" containerID="74a8cf9633e1570c02f9d0cc265e962e93b6470f002ef9eb8ab2fb22459fe1e6"
Sep 08 02:16:19.442185 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:19.442164    2547 generic.go:356] "Generic (PLEG): container finished" podID="bf419798-9b6d-465a-8bee-5301c8a52031" containerID="11f04e1f3c3c651e89993e33471edc7f149c3530ec7195961dae636466fa88af" exitCode=0
Sep 08 02:16:19.442295 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:19.442197    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8fa74948-7f464b7cd9-75cxz" event={"ID":"bf419798-9b6d-465a-8bee-5301c8a52031","Type":"ContainerDied","Data":"11f04e1f3c3c651e89993e33471edc7f149c3530ec7195961dae636466fa88af"}
Sep 08 02:16:19.442295 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:19.442221    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8fa74948-7f464b7cd9-75cxz"
Sep 08 02:16:19.442295 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:19.442247    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8fa74948-7f464b7cd9-75cxz" event={"ID":"bf419798-9b6d-465a-8bee-5301c8a52031","Type":"ContainerDied","Data":"503e3aa8fd80948e34215a00bbf3268ccffefdba487ff67e98f95314a872a8ed"}
Sep 08 02:16:19.448391 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:19.448371    2547 scope.go:117] "RemoveContainer" containerID="74a8cf9633e1570c02f9d0cc265e962e93b6470f002ef9eb8ab2fb22459fe1e6"
Sep 08 02:16:19.448674 ip-10-0-142-78 kubenswrapper[2547]: E0908 02:16:19.448656    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74a8cf9633e1570c02f9d0cc265e962e93b6470f002ef9eb8ab2fb22459fe1e6\": container with ID starting with 74a8cf9633e1570c02f9d0cc265e962e93b6470f002ef9eb8ab2fb22459fe1e6 not found: ID does not exist" containerID="74a8cf9633e1570c02f9d0cc265e962e93b6470f002ef9eb8ab2fb22459fe1e6"
Sep 08 02:16:19.448757 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:19.448686    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74a8cf9633e1570c02f9d0cc265e962e93b6470f002ef9eb8ab2fb22459fe1e6"} err="failed to get container status \"74a8cf9633e1570c02f9d0cc265e962e93b6470f002ef9eb8ab2fb22459fe1e6\": rpc error: code = NotFound desc = could not find container \"74a8cf9633e1570c02f9d0cc265e962e93b6470f002ef9eb8ab2fb22459fe1e6\": container with ID starting with 74a8cf9633e1570c02f9d0cc265e962e93b6470f002ef9eb8ab2fb22459fe1e6 not found: ID does not exist"
Sep 08 02:16:19.448757 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:19.448710    2547 scope.go:117] "RemoveContainer" containerID="11f04e1f3c3c651e89993e33471edc7f149c3530ec7195961dae636466fa88af"
Sep 08 02:16:19.455611 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:19.455588    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8fa74948-6b998c9754-xw5cs"]
Sep 08 02:16:19.458097 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:19.458078    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8fa74948-6b998c9754-xw5cs"]
Sep 08 02:16:19.458187 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:19.458155    2547 scope.go:117] "RemoveContainer" containerID="11f04e1f3c3c651e89993e33471edc7f149c3530ec7195961dae636466fa88af"
Sep 08 02:16:19.458484 ip-10-0-142-78 kubenswrapper[2547]: E0908 02:16:19.458463    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11f04e1f3c3c651e89993e33471edc7f149c3530ec7195961dae636466fa88af\": container with ID starting with 11f04e1f3c3c651e89993e33471edc7f149c3530ec7195961dae636466fa88af not found: ID does not exist" containerID="11f04e1f3c3c651e89993e33471edc7f149c3530ec7195961dae636466fa88af"
Sep 08 02:16:19.458530 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:19.458491    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11f04e1f3c3c651e89993e33471edc7f149c3530ec7195961dae636466fa88af"} err="failed to get container status \"11f04e1f3c3c651e89993e33471edc7f149c3530ec7195961dae636466fa88af\": rpc error: code = NotFound desc = could not find container \"11f04e1f3c3c651e89993e33471edc7f149c3530ec7195961dae636466fa88af\": container with ID starting with 11f04e1f3c3c651e89993e33471edc7f149c3530ec7195961dae636466fa88af not found: ID does not exist"
Sep 08 02:16:19.470085 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:19.470059    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8fa74948-7f464b7cd9-75cxz"]
Sep 08 02:16:19.472347 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:19.472323    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8fa74948-7f464b7cd9-75cxz"]
Sep 08 02:16:20.993643 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:20.993608    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="293a49f1-6902-4192-80cd-105f9594bf89" path="/var/lib/kubelet/pods/293a49f1-6902-4192-80cd-105f9594bf89/volumes"
Sep 08 02:16:20.994018 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:20.994001    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf419798-9b6d-465a-8bee-5301c8a52031" path="/var/lib/kubelet/pods/bf419798-9b6d-465a-8bee-5301c8a52031/volumes"
Sep 08 02:16:30.111609 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:30.111570    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fcdd46-64c469749d-jc7xm"]
Sep 08 02:16:30.111988 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:30.111869    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf419798-9b6d-465a-8bee-5301c8a52031" containerName="payload-processing"
Sep 08 02:16:30.111988 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:30.111879    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf419798-9b6d-465a-8bee-5301c8a52031" containerName="payload-processing"
Sep 08 02:16:30.111988 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:30.111889    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aac04da1-ec35-4b47-8efe-f2d34fcccbf4" containerName="revoke-keys"
Sep 08 02:16:30.111988 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:30.111896    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="aac04da1-ec35-4b47-8efe-f2d34fcccbf4" containerName="revoke-keys"
Sep 08 02:16:30.111988 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:30.111906    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="293a49f1-6902-4192-80cd-105f9594bf89" containerName="payload-pre-processing"
Sep 08 02:16:30.111988 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:30.111911    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="293a49f1-6902-4192-80cd-105f9594bf89" containerName="payload-pre-processing"
Sep 08 02:16:30.111988 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:30.111918    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d8fcaa0-2ffb-4931-931e-2b0d3f06d083" containerName="payload-pre-processing"
Sep 08 02:16:30.111988 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:30.111923    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d8fcaa0-2ffb-4931-931e-2b0d3f06d083" containerName="payload-pre-processing"
Sep 08 02:16:30.111988 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:30.111933    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0318e043-46c2-435c-ba36-a07110b8e456" containerName="payload-processing"
Sep 08 02:16:30.111988 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:30.111939    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="0318e043-46c2-435c-ba36-a07110b8e456" containerName="payload-processing"
Sep 08 02:16:30.111988 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:30.111988    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="aac04da1-ec35-4b47-8efe-f2d34fcccbf4" containerName="revoke-keys"
Sep 08 02:16:30.111988 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:30.111995    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d8fcaa0-2ffb-4931-931e-2b0d3f06d083" containerName="payload-pre-processing"
Sep 08 02:16:30.112383 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:30.112001    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf419798-9b6d-465a-8bee-5301c8a52031" containerName="payload-processing"
Sep 08 02:16:30.112383 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:30.112008    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="293a49f1-6902-4192-80cd-105f9594bf89" containerName="payload-pre-processing"
Sep 08 02:16:30.112383 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:30.112016    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="0318e043-46c2-435c-ba36-a07110b8e456" containerName="payload-processing"
Sep 08 02:16:30.116296 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:30.116277    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fcdd46-64c469749d-jc7xm"
Sep 08 02:16:30.118399 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:30.118376    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-fcdd46-dockercfg-p5dxt\""
Sep 08 02:16:30.118399 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:30.118396    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-fcdd46\""
Sep 08 02:16:30.127921 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:30.127892    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fcdd46-64c469749d-jc7xm"]
Sep 08 02:16:30.151311 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:30.151268    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-fcdd46-5488fbfb5-rjwkz"]
Sep 08 02:16:30.154310 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:30.154285    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-fcdd46-5488fbfb5-rjwkz"
Sep 08 02:16:30.165477 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:30.165449    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-fcdd46-5488fbfb5-rjwkz"]
Sep 08 02:16:30.273016 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:30.272982    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb5bb8a7-37b7-4581-be4c-49892f01db9e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-fcdd46-5488fbfb5-rjwkz\" (UID: \"cb5bb8a7-37b7-4581-be4c-49892f01db9e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-fcdd46-5488fbfb5-rjwkz"
Sep 08 02:16:30.273220 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:30.273033    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7b93571-26fc-4e6f-89b5-48926f61dbf8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-fcdd46-64c469749d-jc7xm\" (UID: \"a7b93571-26fc-4e6f-89b5-48926f61dbf8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fcdd46-64c469749d-jc7xm"
Sep 08 02:16:30.273220 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:30.273102    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnvf5\" (UniqueName: \"kubernetes.io/projected/a7b93571-26fc-4e6f-89b5-48926f61dbf8-kube-api-access-lnvf5\") pod \"payload-pre-processing-e2e-shared-a-w3-fcdd46-64c469749d-jc7xm\" (UID: \"a7b93571-26fc-4e6f-89b5-48926f61dbf8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fcdd46-64c469749d-jc7xm"
Sep 08 02:16:30.273220 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:30.273177    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7f7w\" (UniqueName: \"kubernetes.io/projected/cb5bb8a7-37b7-4581-be4c-49892f01db9e-kube-api-access-z7f7w\") pod \"payload-processing-e2e-shared-a-w3-fcdd46-5488fbfb5-rjwkz\" (UID: \"cb5bb8a7-37b7-4581-be4c-49892f01db9e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-fcdd46-5488fbfb5-rjwkz"
Sep 08 02:16:30.374677 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:30.374567    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb5bb8a7-37b7-4581-be4c-49892f01db9e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-fcdd46-5488fbfb5-rjwkz\" (UID: \"cb5bb8a7-37b7-4581-be4c-49892f01db9e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-fcdd46-5488fbfb5-rjwkz"
Sep 08 02:16:30.374677 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:30.374630    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7b93571-26fc-4e6f-89b5-48926f61dbf8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-fcdd46-64c469749d-jc7xm\" (UID: \"a7b93571-26fc-4e6f-89b5-48926f61dbf8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fcdd46-64c469749d-jc7xm"
Sep 08 02:16:30.374677 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:30.374671    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnvf5\" (UniqueName: \"kubernetes.io/projected/a7b93571-26fc-4e6f-89b5-48926f61dbf8-kube-api-access-lnvf5\") pod \"payload-pre-processing-e2e-shared-a-w3-fcdd46-64c469749d-jc7xm\" (UID: \"a7b93571-26fc-4e6f-89b5-48926f61dbf8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fcdd46-64c469749d-jc7xm"
Sep 08 02:16:30.374968 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:30.374692    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7f7w\" (UniqueName: \"kubernetes.io/projected/cb5bb8a7-37b7-4581-be4c-49892f01db9e-kube-api-access-z7f7w\") pod \"payload-processing-e2e-shared-a-w3-fcdd46-5488fbfb5-rjwkz\" (UID: \"cb5bb8a7-37b7-4581-be4c-49892f01db9e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-fcdd46-5488fbfb5-rjwkz"
Sep 08 02:16:30.375464 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:30.375436    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb5bb8a7-37b7-4581-be4c-49892f01db9e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-fcdd46-5488fbfb5-rjwkz\" (UID: \"cb5bb8a7-37b7-4581-be4c-49892f01db9e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-fcdd46-5488fbfb5-rjwkz"
Sep 08 02:16:30.375554 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:30.375437    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7b93571-26fc-4e6f-89b5-48926f61dbf8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-fcdd46-64c469749d-jc7xm\" (UID: \"a7b93571-26fc-4e6f-89b5-48926f61dbf8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fcdd46-64c469749d-jc7xm"
Sep 08 02:16:30.383813 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:30.383789    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnvf5\" (UniqueName: \"kubernetes.io/projected/a7b93571-26fc-4e6f-89b5-48926f61dbf8-kube-api-access-lnvf5\") pod \"payload-pre-processing-e2e-shared-a-w3-fcdd46-64c469749d-jc7xm\" (UID: \"a7b93571-26fc-4e6f-89b5-48926f61dbf8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fcdd46-64c469749d-jc7xm"
Sep 08 02:16:30.383927 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:30.383850    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7f7w\" (UniqueName: \"kubernetes.io/projected/cb5bb8a7-37b7-4581-be4c-49892f01db9e-kube-api-access-z7f7w\") pod \"payload-processing-e2e-shared-a-w3-fcdd46-5488fbfb5-rjwkz\" (UID: \"cb5bb8a7-37b7-4581-be4c-49892f01db9e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-fcdd46-5488fbfb5-rjwkz"
Sep 08 02:16:30.426720 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:30.426681    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fcdd46-64c469749d-jc7xm"
Sep 08 02:16:30.465593 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:30.465488    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-fcdd46-5488fbfb5-rjwkz"
Sep 08 02:16:30.701298 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:30.701270    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fcdd46-64c469749d-jc7xm"]
Sep 08 02:16:30.703318 ip-10-0-142-78 kubenswrapper[2547]: W0908 02:16:30.703291    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7b93571_26fc_4e6f_89b5_48926f61dbf8.slice/crio-cd3ec2ee04edaa97ac94d77f49a9c9e9fb9e088ada4602ccdb5cac1f3d6774e6 WatchSource:0}: Error finding container cd3ec2ee04edaa97ac94d77f49a9c9e9fb9e088ada4602ccdb5cac1f3d6774e6: Status 404 returned error can't find the container with id cd3ec2ee04edaa97ac94d77f49a9c9e9fb9e088ada4602ccdb5cac1f3d6774e6
Sep 08 02:16:30.744206 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:30.744103    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-fcdd46-5488fbfb5-rjwkz"]
Sep 08 02:16:30.746699 ip-10-0-142-78 kubenswrapper[2547]: W0908 02:16:30.746577    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb5bb8a7_37b7_4581_be4c_49892f01db9e.slice/crio-405f4360748ab43ac1c997a766b92b3e8e7980a31fbc9a717ad01a6bdf8063a6 WatchSource:0}: Error finding container 405f4360748ab43ac1c997a766b92b3e8e7980a31fbc9a717ad01a6bdf8063a6: Status 404 returned error can't find the container with id 405f4360748ab43ac1c997a766b92b3e8e7980a31fbc9a717ad01a6bdf8063a6
Sep 08 02:16:31.499556 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:31.499519    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fcdd46-64c469749d-jc7xm" event={"ID":"a7b93571-26fc-4e6f-89b5-48926f61dbf8","Type":"ContainerStarted","Data":"f135d722b948270d3184216daaeedf567683ce712639dbc674eea0c06908e2ee"}
Sep 08 02:16:31.500035 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:31.499577    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fcdd46-64c469749d-jc7xm" event={"ID":"a7b93571-26fc-4e6f-89b5-48926f61dbf8","Type":"ContainerStarted","Data":"cd3ec2ee04edaa97ac94d77f49a9c9e9fb9e088ada4602ccdb5cac1f3d6774e6"}
Sep 08 02:16:31.500035 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:31.499605    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fcdd46-64c469749d-jc7xm"
Sep 08 02:16:31.501279 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:31.501256    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-fcdd46-5488fbfb5-rjwkz" event={"ID":"cb5bb8a7-37b7-4581-be4c-49892f01db9e","Type":"ContainerStarted","Data":"aedf16d8279776cd25072d0245eefa74907c3978081770ec144648e53d0b005b"}
Sep 08 02:16:31.501397 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:31.501297    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-fcdd46-5488fbfb5-rjwkz" event={"ID":"cb5bb8a7-37b7-4581-be4c-49892f01db9e","Type":"ContainerStarted","Data":"405f4360748ab43ac1c997a766b92b3e8e7980a31fbc9a717ad01a6bdf8063a6"}
Sep 08 02:16:31.501397 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:31.501317    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-fcdd46-5488fbfb5-rjwkz"
Sep 08 02:16:31.516272 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:31.516223    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fcdd46-64c469749d-jc7xm" podStartSLOduration=1.516208306 podStartE2EDuration="1.516208306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:16:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:16:31.514010228 +0000 UTC m=+1063.037785348" watchObservedRunningTime="2026-09-08 02:16:31.516208306 +0000 UTC m=+1063.039983397"
Sep 08 02:16:31.534918 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:31.534863    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-fcdd46-5488fbfb5-rjwkz" podStartSLOduration=1.53484673 podStartE2EDuration="1.53484673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:16:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:16:31.531156876 +0000 UTC m=+1063.054931947" watchObservedRunningTime="2026-09-08 02:16:31.53484673 +0000 UTC m=+1063.058621810"
Sep 08 02:16:42.509770 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:42.509680    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fcdd46-64c469749d-jc7xm"
Sep 08 02:16:42.510204 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:42.509858    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-fcdd46-5488fbfb5-rjwkz"
Sep 08 02:16:44.574871 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:44.574833    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-670bc3-6fc959454b-jqpgb"]
Sep 08 02:16:44.578334 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:44.578313    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-670bc3-6fc959454b-jqpgb"
Sep 08 02:16:44.580530 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:44.580509    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-670bc3\""
Sep 08 02:16:44.580633 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:44.580509    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-670bc3-dockercfg-b4vgx\""
Sep 08 02:16:44.588402 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:44.588372    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-670bc3-6fc959454b-jqpgb"]
Sep 08 02:16:44.589194 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:44.589175    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8dc52e3-ab0b-44cd-810f-bac422df9d46-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-670bc3-6fc959454b-jqpgb\" (UID: \"f8dc52e3-ab0b-44cd-810f-bac422df9d46\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-670bc3-6fc959454b-jqpgb"
Sep 08 02:16:44.589296 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:44.589217    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzt9h\" (UniqueName: \"kubernetes.io/projected/f8dc52e3-ab0b-44cd-810f-bac422df9d46-kube-api-access-rzt9h\") pod \"payload-pre-processing-e2e-shared-a-w2-670bc3-6fc959454b-jqpgb\" (UID: \"f8dc52e3-ab0b-44cd-810f-bac422df9d46\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-670bc3-6fc959454b-jqpgb"
Sep 08 02:16:44.616240 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:44.616209    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-670bc3-76644ff8c-jqf99"]
Sep 08 02:16:44.619418 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:44.619398    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-670bc3-76644ff8c-jqf99"
Sep 08 02:16:44.629868 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:44.629837    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-670bc3-76644ff8c-jqf99"]
Sep 08 02:16:44.689761 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:44.689718    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzt9h\" (UniqueName: \"kubernetes.io/projected/f8dc52e3-ab0b-44cd-810f-bac422df9d46-kube-api-access-rzt9h\") pod \"payload-pre-processing-e2e-shared-a-w2-670bc3-6fc959454b-jqpgb\" (UID: \"f8dc52e3-ab0b-44cd-810f-bac422df9d46\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-670bc3-6fc959454b-jqpgb"
Sep 08 02:16:44.689937 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:44.689802    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60468f93-bb6e-470c-8c1a-cd6212d58c49-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-670bc3-76644ff8c-jqf99\" (UID: \"60468f93-bb6e-470c-8c1a-cd6212d58c49\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-670bc3-76644ff8c-jqf99"
Sep 08 02:16:44.689937 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:44.689837    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwbmj\" (UniqueName: \"kubernetes.io/projected/60468f93-bb6e-470c-8c1a-cd6212d58c49-kube-api-access-jwbmj\") pod \"payload-processing-e2e-shared-a-w2-670bc3-76644ff8c-jqf99\" (UID: \"60468f93-bb6e-470c-8c1a-cd6212d58c49\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-670bc3-76644ff8c-jqf99"
Sep 08 02:16:44.689937 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:44.689896    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8dc52e3-ab0b-44cd-810f-bac422df9d46-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-670bc3-6fc959454b-jqpgb\" (UID: \"f8dc52e3-ab0b-44cd-810f-bac422df9d46\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-670bc3-6fc959454b-jqpgb"
Sep 08 02:16:44.690708 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:44.690676    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8dc52e3-ab0b-44cd-810f-bac422df9d46-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-670bc3-6fc959454b-jqpgb\" (UID: \"f8dc52e3-ab0b-44cd-810f-bac422df9d46\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-670bc3-6fc959454b-jqpgb"
Sep 08 02:16:44.696872 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:44.696846    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzt9h\" (UniqueName: \"kubernetes.io/projected/f8dc52e3-ab0b-44cd-810f-bac422df9d46-kube-api-access-rzt9h\") pod \"payload-pre-processing-e2e-shared-a-w2-670bc3-6fc959454b-jqpgb\" (UID: \"f8dc52e3-ab0b-44cd-810f-bac422df9d46\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-670bc3-6fc959454b-jqpgb"
Sep 08 02:16:44.790624 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:44.790581    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60468f93-bb6e-470c-8c1a-cd6212d58c49-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-670bc3-76644ff8c-jqf99\" (UID: \"60468f93-bb6e-470c-8c1a-cd6212d58c49\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-670bc3-76644ff8c-jqf99"
Sep 08 02:16:44.790624 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:44.790628    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwbmj\" (UniqueName: \"kubernetes.io/projected/60468f93-bb6e-470c-8c1a-cd6212d58c49-kube-api-access-jwbmj\") pod \"payload-processing-e2e-shared-a-w2-670bc3-76644ff8c-jqf99\" (UID: \"60468f93-bb6e-470c-8c1a-cd6212d58c49\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-670bc3-76644ff8c-jqf99"
Sep 08 02:16:44.791385 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:44.791364    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60468f93-bb6e-470c-8c1a-cd6212d58c49-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-670bc3-76644ff8c-jqf99\" (UID: \"60468f93-bb6e-470c-8c1a-cd6212d58c49\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-670bc3-76644ff8c-jqf99"
Sep 08 02:16:44.799374 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:44.799343    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwbmj\" (UniqueName: \"kubernetes.io/projected/60468f93-bb6e-470c-8c1a-cd6212d58c49-kube-api-access-jwbmj\") pod \"payload-processing-e2e-shared-a-w2-670bc3-76644ff8c-jqf99\" (UID: \"60468f93-bb6e-470c-8c1a-cd6212d58c49\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-670bc3-76644ff8c-jqf99"
Sep 08 02:16:44.889657 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:44.889630    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-670bc3-6fc959454b-jqpgb"
Sep 08 02:16:44.931665 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:44.931631    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-670bc3-76644ff8c-jqf99"
Sep 08 02:16:45.158509 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:45.158438    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-670bc3-6fc959454b-jqpgb"]
Sep 08 02:16:45.161340 ip-10-0-142-78 kubenswrapper[2547]: W0908 02:16:45.161314    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8dc52e3_ab0b_44cd_810f_bac422df9d46.slice/crio-bcecbc046ca9c514c9acfd486006c5958f1027f5afd7141aa794c22f93f09c29 WatchSource:0}: Error finding container bcecbc046ca9c514c9acfd486006c5958f1027f5afd7141aa794c22f93f09c29: Status 404 returned error can't find the container with id bcecbc046ca9c514c9acfd486006c5958f1027f5afd7141aa794c22f93f09c29
Sep 08 02:16:45.211815 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:45.211786    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-670bc3-76644ff8c-jqf99"]
Sep 08 02:16:45.213400 ip-10-0-142-78 kubenswrapper[2547]: W0908 02:16:45.213378    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60468f93_bb6e_470c_8c1a_cd6212d58c49.slice/crio-dcc20f76b839470bf2a6427324c56d0335f5f4efa41860b1539cca66e31e594a WatchSource:0}: Error finding container dcc20f76b839470bf2a6427324c56d0335f5f4efa41860b1539cca66e31e594a: Status 404 returned error can't find the container with id dcc20f76b839470bf2a6427324c56d0335f5f4efa41860b1539cca66e31e594a
Sep 08 02:16:45.568997 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:45.568884    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-670bc3-6fc959454b-jqpgb" event={"ID":"f8dc52e3-ab0b-44cd-810f-bac422df9d46","Type":"ContainerStarted","Data":"f4e27b68961eb4f81c3aefcd2e79a0c34f1c95989e0cfc49f869005db446eafc"}
Sep 08 02:16:45.568997 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:45.568944    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-670bc3-6fc959454b-jqpgb" event={"ID":"f8dc52e3-ab0b-44cd-810f-bac422df9d46","Type":"ContainerStarted","Data":"bcecbc046ca9c514c9acfd486006c5958f1027f5afd7141aa794c22f93f09c29"}
Sep 08 02:16:45.568997 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:45.568972    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-670bc3-6fc959454b-jqpgb"
Sep 08 02:16:45.570675 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:45.570638    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-670bc3-76644ff8c-jqf99" event={"ID":"60468f93-bb6e-470c-8c1a-cd6212d58c49","Type":"ContainerStarted","Data":"9329c7ecafdfc2cb42b3936d2250a0190c166bc7f1f150c821cfb9be4d248f94"}
Sep 08 02:16:45.570816 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:45.570699    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-670bc3-76644ff8c-jqf99" event={"ID":"60468f93-bb6e-470c-8c1a-cd6212d58c49","Type":"ContainerStarted","Data":"dcc20f76b839470bf2a6427324c56d0335f5f4efa41860b1539cca66e31e594a"}
Sep 08 02:16:45.570816 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:45.570731    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-670bc3-76644ff8c-jqf99"
Sep 08 02:16:45.586212 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:45.586161    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-670bc3-6fc959454b-jqpgb" podStartSLOduration=1.586123731 podStartE2EDuration="1.586123731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:16:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:16:45.585048595 +0000 UTC m=+1077.108823676" watchObservedRunningTime="2026-09-08 02:16:45.586123731 +0000 UTC m=+1077.109898807"
Sep 08 02:16:45.604496 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:45.604441    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-670bc3-76644ff8c-jqf99" podStartSLOduration=1.604425733 podStartE2EDuration="1.604425733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:16:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:16:45.603567971 +0000 UTC m=+1077.127343052" watchObservedRunningTime="2026-09-08 02:16:45.604425733 +0000 UTC m=+1077.128200812"
Sep 08 02:16:56.577756 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:56.577720    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-670bc3-6fc959454b-jqpgb"
Sep 08 02:16:56.578177 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:56.577805    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-670bc3-76644ff8c-jqf99"
Sep 08 02:16:58.548126 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:58.548085    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-527ac8-5fb9794fb6-rhqvt"]
Sep 08 02:16:58.551058 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:58.551041    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-527ac8-5fb9794fb6-rhqvt"
Sep 08 02:16:58.553214 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:58.553190    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-527ac8-dockercfg-6tmq7\""
Sep 08 02:16:58.553328 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:58.553221    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-527ac8\""
Sep 08 02:16:58.560493 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:58.560466    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-527ac8-5fb9794fb6-rhqvt"]
Sep 08 02:16:58.565095 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:58.565071    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-527ac8-7cc5bfb4d9-hzk4w"]
Sep 08 02:16:58.567317 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:58.567297    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-527ac8-7cc5bfb4d9-hzk4w"
Sep 08 02:16:58.581707 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:58.581681    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-527ac8-7cc5bfb4d9-hzk4w"]
Sep 08 02:16:58.604441 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:58.604408    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6b57m\" (UniqueName: \"kubernetes.io/projected/8bedf1b4-79ee-40a3-aeac-28908af7f972-kube-api-access-6b57m\") pod \"payload-pre-processing-e2e-shared-b-w3-527ac8-5fb9794fb6-rhqvt\" (UID: \"8bedf1b4-79ee-40a3-aeac-28908af7f972\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-527ac8-5fb9794fb6-rhqvt"
Sep 08 02:16:58.604612 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:58.604463    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bedf1b4-79ee-40a3-aeac-28908af7f972-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-527ac8-5fb9794fb6-rhqvt\" (UID: \"8bedf1b4-79ee-40a3-aeac-28908af7f972\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-527ac8-5fb9794fb6-rhqvt"
Sep 08 02:16:58.604612 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:58.604483    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wl9lb\" (UniqueName: \"kubernetes.io/projected/5ef60051-508f-41e4-9293-d4d04b7f47a5-kube-api-access-wl9lb\") pod \"payload-processing-e2e-shared-b-w3-527ac8-7cc5bfb4d9-hzk4w\" (UID: \"5ef60051-508f-41e4-9293-d4d04b7f47a5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-527ac8-7cc5bfb4d9-hzk4w"
Sep 08 02:16:58.604612 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:58.604553    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ef60051-508f-41e4-9293-d4d04b7f47a5-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-527ac8-7cc5bfb4d9-hzk4w\" (UID: \"5ef60051-508f-41e4-9293-d4d04b7f47a5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-527ac8-7cc5bfb4d9-hzk4w"
Sep 08 02:16:58.705305 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:58.705270    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6b57m\" (UniqueName: \"kubernetes.io/projected/8bedf1b4-79ee-40a3-aeac-28908af7f972-kube-api-access-6b57m\") pod \"payload-pre-processing-e2e-shared-b-w3-527ac8-5fb9794fb6-rhqvt\" (UID: \"8bedf1b4-79ee-40a3-aeac-28908af7f972\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-527ac8-5fb9794fb6-rhqvt"
Sep 08 02:16:58.705495 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:58.705325    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bedf1b4-79ee-40a3-aeac-28908af7f972-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-527ac8-5fb9794fb6-rhqvt\" (UID: \"8bedf1b4-79ee-40a3-aeac-28908af7f972\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-527ac8-5fb9794fb6-rhqvt"
Sep 08 02:16:58.705495 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:58.705344    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wl9lb\" (UniqueName: \"kubernetes.io/projected/5ef60051-508f-41e4-9293-d4d04b7f47a5-kube-api-access-wl9lb\") pod \"payload-processing-e2e-shared-b-w3-527ac8-7cc5bfb4d9-hzk4w\" (UID: \"5ef60051-508f-41e4-9293-d4d04b7f47a5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-527ac8-7cc5bfb4d9-hzk4w"
Sep 08 02:16:58.705495 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:58.705365    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ef60051-508f-41e4-9293-d4d04b7f47a5-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-527ac8-7cc5bfb4d9-hzk4w\" (UID: \"5ef60051-508f-41e4-9293-d4d04b7f47a5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-527ac8-7cc5bfb4d9-hzk4w"
Sep 08 02:16:58.706101 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:58.706081    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bedf1b4-79ee-40a3-aeac-28908af7f972-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-527ac8-5fb9794fb6-rhqvt\" (UID: \"8bedf1b4-79ee-40a3-aeac-28908af7f972\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-527ac8-5fb9794fb6-rhqvt"
Sep 08 02:16:58.706159 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:58.706100    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ef60051-508f-41e4-9293-d4d04b7f47a5-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-527ac8-7cc5bfb4d9-hzk4w\" (UID: \"5ef60051-508f-41e4-9293-d4d04b7f47a5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-527ac8-7cc5bfb4d9-hzk4w"
Sep 08 02:16:58.713157 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:58.713108    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wl9lb\" (UniqueName: \"kubernetes.io/projected/5ef60051-508f-41e4-9293-d4d04b7f47a5-kube-api-access-wl9lb\") pod \"payload-processing-e2e-shared-b-w3-527ac8-7cc5bfb4d9-hzk4w\" (UID: \"5ef60051-508f-41e4-9293-d4d04b7f47a5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-527ac8-7cc5bfb4d9-hzk4w"
Sep 08 02:16:58.713338 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:58.713316    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6b57m\" (UniqueName: \"kubernetes.io/projected/8bedf1b4-79ee-40a3-aeac-28908af7f972-kube-api-access-6b57m\") pod \"payload-pre-processing-e2e-shared-b-w3-527ac8-5fb9794fb6-rhqvt\" (UID: \"8bedf1b4-79ee-40a3-aeac-28908af7f972\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-527ac8-5fb9794fb6-rhqvt"
Sep 08 02:16:58.862097 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:58.862017    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-527ac8-5fb9794fb6-rhqvt"
Sep 08 02:16:58.879349 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:58.879319    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-527ac8-7cc5bfb4d9-hzk4w"
Sep 08 02:16:59.132446 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:59.132421    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-527ac8-5fb9794fb6-rhqvt"]
Sep 08 02:16:59.134304 ip-10-0-142-78 kubenswrapper[2547]: W0908 02:16:59.134278    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bedf1b4_79ee_40a3_aeac_28908af7f972.slice/crio-275f3a00898a11647f40d1e17747c26c52654e5e1e826d97b9a0befbd3c62e69 WatchSource:0}: Error finding container 275f3a00898a11647f40d1e17747c26c52654e5e1e826d97b9a0befbd3c62e69: Status 404 returned error can't find the container with id 275f3a00898a11647f40d1e17747c26c52654e5e1e826d97b9a0befbd3c62e69
Sep 08 02:16:59.154961 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:59.154938    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-527ac8-7cc5bfb4d9-hzk4w"]
Sep 08 02:16:59.176276 ip-10-0-142-78 kubenswrapper[2547]: W0908 02:16:59.176244    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ef60051_508f_41e4_9293_d4d04b7f47a5.slice/crio-4a647b31afc1f0d7b2ccd125eedab9fb3011fb76580835bdcebc4e964108f772 WatchSource:0}: Error finding container 4a647b31afc1f0d7b2ccd125eedab9fb3011fb76580835bdcebc4e964108f772: Status 404 returned error can't find the container with id 4a647b31afc1f0d7b2ccd125eedab9fb3011fb76580835bdcebc4e964108f772
Sep 08 02:16:59.640482 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:59.640443    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-527ac8-7cc5bfb4d9-hzk4w" event={"ID":"5ef60051-508f-41e4-9293-d4d04b7f47a5","Type":"ContainerStarted","Data":"c04363fbbb9a358ba8295e7bb0518aca221d4c3a7fd1c62a445a42b776044331"}
Sep 08 02:16:59.640947 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:59.640508    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-527ac8-7cc5bfb4d9-hzk4w" event={"ID":"5ef60051-508f-41e4-9293-d4d04b7f47a5","Type":"ContainerStarted","Data":"4a647b31afc1f0d7b2ccd125eedab9fb3011fb76580835bdcebc4e964108f772"}
Sep 08 02:16:59.640947 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:59.640543    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-527ac8-7cc5bfb4d9-hzk4w"
Sep 08 02:16:59.642158 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:59.642106    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-527ac8-5fb9794fb6-rhqvt" event={"ID":"8bedf1b4-79ee-40a3-aeac-28908af7f972","Type":"ContainerStarted","Data":"77c575680def701a662b6547c4175667568fb1f14238fa72c105618c8f7d888d"}
Sep 08 02:16:59.642292 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:59.642176    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-527ac8-5fb9794fb6-rhqvt"
Sep 08 02:16:59.642292 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:59.642201    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-527ac8-5fb9794fb6-rhqvt" event={"ID":"8bedf1b4-79ee-40a3-aeac-28908af7f972","Type":"ContainerStarted","Data":"275f3a00898a11647f40d1e17747c26c52654e5e1e826d97b9a0befbd3c62e69"}
Sep 08 02:16:59.654919 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:59.654871    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-527ac8-7cc5bfb4d9-hzk4w" podStartSLOduration=1.6548543690000002 podStartE2EDuration="1.654854369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:16:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:16:59.654801066 +0000 UTC m=+1091.178576147" watchObservedRunningTime="2026-09-08 02:16:59.654854369 +0000 UTC m=+1091.178629467"
Sep 08 02:16:59.667971 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:16:59.667906    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-527ac8-5fb9794fb6-rhqvt" podStartSLOduration=1.6678905080000002 podStartE2EDuration="1.667890508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:16:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:16:59.667709287 +0000 UTC m=+1091.191484369" watchObservedRunningTime="2026-09-08 02:16:59.667890508 +0000 UTC m=+1091.191665588"
Sep 08 02:17:10.649902 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:17:10.649870    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-527ac8-7cc5bfb4d9-hzk4w"
Sep 08 02:17:10.650294 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:17:10.650036    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-527ac8-5fb9794fb6-rhqvt"
Sep 08 02:17:12.666056 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:17:12.666018    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1d1413-d9c688cd-mcjkg"]
Sep 08 02:17:12.670293 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:17:12.670275    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1d1413-d9c688cd-mcjkg"
Sep 08 02:17:12.672582 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:17:12.672555    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-1d1413\""
Sep 08 02:17:12.672705 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:17:12.672588    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-1d1413-dockercfg-gjwnr\""
Sep 08 02:17:12.679517 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:17:12.679492    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1d1413-d9c688cd-mcjkg"]
Sep 08 02:17:12.704962 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:17:12.704934    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-1d1413-6998765b7f-m47zs"]
Sep 08 02:17:12.708238 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:17:12.708221    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-1d1413-6998765b7f-m47zs"
Sep 08 02:17:12.719057 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:17:12.719033    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-1d1413-6998765b7f-m47zs"]
Sep 08 02:17:12.729547 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:17:12.729516    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrfx6\" (UniqueName: \"kubernetes.io/projected/e0a235cc-d789-472a-94b1-0d62c74aa161-kube-api-access-vrfx6\") pod \"payload-pre-processing-e2e-shared-b-w2-1d1413-d9c688cd-mcjkg\" (UID: \"e0a235cc-d789-472a-94b1-0d62c74aa161\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1d1413-d9c688cd-mcjkg"
Sep 08 02:17:12.729670 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:17:12.729565    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0a235cc-d789-472a-94b1-0d62c74aa161-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-1d1413-d9c688cd-mcjkg\" (UID: \"e0a235cc-d789-472a-94b1-0d62c74aa161\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1d1413-d9c688cd-mcjkg"
Sep 08 02:17:12.830597 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:17:12.830560    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrfx6\" (UniqueName: \"kubernetes.io/projected/e0a235cc-d789-472a-94b1-0d62c74aa161-kube-api-access-vrfx6\") pod \"payload-pre-processing-e2e-shared-b-w2-1d1413-d9c688cd-mcjkg\" (UID: \"e0a235cc-d789-472a-94b1-0d62c74aa161\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1d1413-d9c688cd-mcjkg"
Sep 08 02:17:12.830776 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:17:12.830616    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0a235cc-d789-472a-94b1-0d62c74aa161-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-1d1413-d9c688cd-mcjkg\" (UID: \"e0a235cc-d789-472a-94b1-0d62c74aa161\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1d1413-d9c688cd-mcjkg"
Sep 08 02:17:12.830776 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:17:12.830670    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98cdb5bb-741c-4f68-b21a-9b75ec91c30e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-1d1413-6998765b7f-m47zs\" (UID: \"98cdb5bb-741c-4f68-b21a-9b75ec91c30e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-1d1413-6998765b7f-m47zs"
Sep 08 02:17:12.830776 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:17:12.830720    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znxbc\" (UniqueName: \"kubernetes.io/projected/98cdb5bb-741c-4f68-b21a-9b75ec91c30e-kube-api-access-znxbc\") pod \"payload-processing-e2e-shared-b-w2-1d1413-6998765b7f-m47zs\" (UID: \"98cdb5bb-741c-4f68-b21a-9b75ec91c30e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-1d1413-6998765b7f-m47zs"
Sep 08 02:17:12.831367 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:17:12.831348    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0a235cc-d789-472a-94b1-0d62c74aa161-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-1d1413-d9c688cd-mcjkg\" (UID: \"e0a235cc-d789-472a-94b1-0d62c74aa161\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1d1413-d9c688cd-mcjkg"
Sep 08 02:17:12.839921 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:17:12.839902    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrfx6\" (UniqueName: \"kubernetes.io/projected/e0a235cc-d789-472a-94b1-0d62c74aa161-kube-api-access-vrfx6\") pod \"payload-pre-processing-e2e-shared-b-w2-1d1413-d9c688cd-mcjkg\" (UID: \"e0a235cc-d789-472a-94b1-0d62c74aa161\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1d1413-d9c688cd-mcjkg"
Sep 08 02:17:12.931948 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:17:12.931850    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98cdb5bb-741c-4f68-b21a-9b75ec91c30e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-1d1413-6998765b7f-m47zs\" (UID: \"98cdb5bb-741c-4f68-b21a-9b75ec91c30e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-1d1413-6998765b7f-m47zs"
Sep 08 02:17:12.931948 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:17:12.931911    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znxbc\" (UniqueName: \"kubernetes.io/projected/98cdb5bb-741c-4f68-b21a-9b75ec91c30e-kube-api-access-znxbc\") pod \"payload-processing-e2e-shared-b-w2-1d1413-6998765b7f-m47zs\" (UID: \"98cdb5bb-741c-4f68-b21a-9b75ec91c30e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-1d1413-6998765b7f-m47zs"
Sep 08 02:17:12.932619 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:17:12.932596    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98cdb5bb-741c-4f68-b21a-9b75ec91c30e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-1d1413-6998765b7f-m47zs\" (UID: \"98cdb5bb-741c-4f68-b21a-9b75ec91c30e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-1d1413-6998765b7f-m47zs"
Sep 08 02:17:12.939876 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:17:12.939851    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znxbc\" (UniqueName: \"kubernetes.io/projected/98cdb5bb-741c-4f68-b21a-9b75ec91c30e-kube-api-access-znxbc\") pod \"payload-processing-e2e-shared-b-w2-1d1413-6998765b7f-m47zs\" (UID: \"98cdb5bb-741c-4f68-b21a-9b75ec91c30e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-1d1413-6998765b7f-m47zs"
Sep 08 02:17:12.981779 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:17:12.981749    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1d1413-d9c688cd-mcjkg"
Sep 08 02:17:13.018277 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:17:13.018238    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-1d1413-6998765b7f-m47zs"
Sep 08 02:17:13.259518 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:17:13.259491    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1d1413-d9c688cd-mcjkg"]
Sep 08 02:17:13.261008 ip-10-0-142-78 kubenswrapper[2547]: W0908 02:17:13.260982    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0a235cc_d789_472a_94b1_0d62c74aa161.slice/crio-b3004e4a538a66b469eb419d5f5f7b010645ae9ae612fa12871bb2e5c852d9a7 WatchSource:0}: Error finding container b3004e4a538a66b469eb419d5f5f7b010645ae9ae612fa12871bb2e5c852d9a7: Status 404 returned error can't find the container with id b3004e4a538a66b469eb419d5f5f7b010645ae9ae612fa12871bb2e5c852d9a7
Sep 08 02:17:13.299237 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:17:13.299205    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-1d1413-6998765b7f-m47zs"]
Sep 08 02:17:13.316409 ip-10-0-142-78 kubenswrapper[2547]: W0908 02:17:13.316367    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98cdb5bb_741c_4f68_b21a_9b75ec91c30e.slice/crio-6e84dcb60a64e2ec42fc8536a89999c412a81325a6bc579e01c7006c83fa9de4 WatchSource:0}: Error finding container 6e84dcb60a64e2ec42fc8536a89999c412a81325a6bc579e01c7006c83fa9de4: Status 404 returned error can't find the container with id 6e84dcb60a64e2ec42fc8536a89999c412a81325a6bc579e01c7006c83fa9de4
Sep 08 02:17:13.711571 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:17:13.711529    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-1d1413-6998765b7f-m47zs" event={"ID":"98cdb5bb-741c-4f68-b21a-9b75ec91c30e","Type":"ContainerStarted","Data":"ef13f08e5262d75ff831a9367fb158c18ee374efd8e82a5e86a3d5efa09d0dbd"}
Sep 08 02:17:13.712023 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:17:13.711591    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-1d1413-6998765b7f-m47zs" event={"ID":"98cdb5bb-741c-4f68-b21a-9b75ec91c30e","Type":"ContainerStarted","Data":"6e84dcb60a64e2ec42fc8536a89999c412a81325a6bc579e01c7006c83fa9de4"}
Sep 08 02:17:13.712023 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:17:13.711648    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-1d1413-6998765b7f-m47zs"
Sep 08 02:17:13.713112 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:17:13.713092    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1d1413-d9c688cd-mcjkg" event={"ID":"e0a235cc-d789-472a-94b1-0d62c74aa161","Type":"ContainerStarted","Data":"5ea7252d98629bb392f43f1e846bdc25d70d56d02f57ea817f3fd6eec0ddad22"}
Sep 08 02:17:13.713248 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:17:13.713147    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1d1413-d9c688cd-mcjkg" event={"ID":"e0a235cc-d789-472a-94b1-0d62c74aa161","Type":"ContainerStarted","Data":"b3004e4a538a66b469eb419d5f5f7b010645ae9ae612fa12871bb2e5c852d9a7"}
Sep 08 02:17:13.713248 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:17:13.713177    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1d1413-d9c688cd-mcjkg"
Sep 08 02:17:13.729269 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:17:13.729215    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-1d1413-6998765b7f-m47zs" podStartSLOduration=1.729196898 podStartE2EDuration="1.729196898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:17:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:17:13.728588519 +0000 UTC m=+1105.252363626" watchObservedRunningTime="2026-09-08 02:17:13.729196898 +0000 UTC m=+1105.252971976"
Sep 08 02:17:13.744814 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:17:13.744763    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1d1413-d9c688cd-mcjkg" podStartSLOduration=1.744745014 podStartE2EDuration="1.744745014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:17:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:17:13.744227906 +0000 UTC m=+1105.268002985" watchObservedRunningTime="2026-09-08 02:17:13.744745014 +0000 UTC m=+1105.268520114"
Sep 08 02:17:24.719903 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:17:24.719868    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-1d1413-6998765b7f-m47zs"
Sep 08 02:17:24.720373 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:17:24.719953    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1d1413-d9c688cd-mcjkg"
Sep 08 02:17:54.892724 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:17:54.892687    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-967ab780-58d56f969d-p7dmb"]
Sep 08 02:17:54.896201 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:17:54.896182    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-967ab780-58d56f969d-p7dmb"
Sep 08 02:17:54.898459 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:17:54.898437    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-967ab780\""
Sep 08 02:17:54.898561 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:17:54.898482    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-967ab780-dockercfg-vvwfw\""
Sep 08 02:17:54.907325 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:17:54.907291    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-967ab780-58d56f969d-p7dmb"]
Sep 08 02:17:54.993033 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:17:54.993004    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89d05519-3e99-4dd8-8f80-b24eb233ddaa-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-967ab780-58d56f969d-p7dmb\" (UID: \"89d05519-3e99-4dd8-8f80-b24eb233ddaa\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-967ab780-58d56f969d-p7dmb"
Sep 08 02:17:54.993237 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:17:54.993056    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fx7tj\" (UniqueName: \"kubernetes.io/projected/89d05519-3e99-4dd8-8f80-b24eb233ddaa-kube-api-access-fx7tj\") pod \"payload-pre-processing-e2e-mt-967ab780-58d56f969d-p7dmb\" (UID: \"89d05519-3e99-4dd8-8f80-b24eb233ddaa\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-967ab780-58d56f969d-p7dmb"
Sep 08 02:17:55.094225 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:17:55.094180    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89d05519-3e99-4dd8-8f80-b24eb233ddaa-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-967ab780-58d56f969d-p7dmb\" (UID: \"89d05519-3e99-4dd8-8f80-b24eb233ddaa\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-967ab780-58d56f969d-p7dmb"
Sep 08 02:17:55.094437 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:17:55.094286    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx7tj\" (UniqueName: \"kubernetes.io/projected/89d05519-3e99-4dd8-8f80-b24eb233ddaa-kube-api-access-fx7tj\") pod \"payload-pre-processing-e2e-mt-967ab780-58d56f969d-p7dmb\" (UID: \"89d05519-3e99-4dd8-8f80-b24eb233ddaa\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-967ab780-58d56f969d-p7dmb"
Sep 08 02:17:55.095021 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:17:55.094999    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89d05519-3e99-4dd8-8f80-b24eb233ddaa-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-967ab780-58d56f969d-p7dmb\" (UID: \"89d05519-3e99-4dd8-8f80-b24eb233ddaa\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-967ab780-58d56f969d-p7dmb"
Sep 08 02:17:55.102242 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:17:55.102213    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fx7tj\" (UniqueName: \"kubernetes.io/projected/89d05519-3e99-4dd8-8f80-b24eb233ddaa-kube-api-access-fx7tj\") pod \"payload-pre-processing-e2e-mt-967ab780-58d56f969d-p7dmb\" (UID: \"89d05519-3e99-4dd8-8f80-b24eb233ddaa\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-967ab780-58d56f969d-p7dmb"
Sep 08 02:17:55.207299 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:17:55.207206    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-967ab780-58d56f969d-p7dmb"
Sep 08 02:17:55.476554 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:17:55.476529    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-967ab780-58d56f969d-p7dmb"]
Sep 08 02:17:55.478409 ip-10-0-142-78 kubenswrapper[2547]: W0908 02:17:55.478380    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89d05519_3e99_4dd8_8f80_b24eb233ddaa.slice/crio-f45eff483f41af8bee7cedf59ecf67386dded1017c30f5311d9af90bcc082f25 WatchSource:0}: Error finding container f45eff483f41af8bee7cedf59ecf67386dded1017c30f5311d9af90bcc082f25: Status 404 returned error can't find the container with id f45eff483f41af8bee7cedf59ecf67386dded1017c30f5311d9af90bcc082f25
Sep 08 02:17:55.910858 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:17:55.910821    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-967ab780-58d56f969d-p7dmb" event={"ID":"89d05519-3e99-4dd8-8f80-b24eb233ddaa","Type":"ContainerStarted","Data":"872850ae23f8e49ca3d52f4e726acd6bc69b25fa4a3c5849355b49c3d2a9c687"}
Sep 08 02:17:55.911580 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:17:55.910873    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-967ab780-58d56f969d-p7dmb" event={"ID":"89d05519-3e99-4dd8-8f80-b24eb233ddaa","Type":"ContainerStarted","Data":"f45eff483f41af8bee7cedf59ecf67386dded1017c30f5311d9af90bcc082f25"}
Sep 08 02:17:55.911580 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:17:55.910926    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-967ab780-58d56f969d-p7dmb"
Sep 08 02:17:55.927726 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:17:55.927673    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-967ab780-58d56f969d-p7dmb" podStartSLOduration=1.927656538 podStartE2EDuration="1.927656538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:17:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:17:55.9258658 +0000 UTC m=+1147.449640880" watchObservedRunningTime="2026-09-08 02:17:55.927656538 +0000 UTC m=+1147.451431618"
Sep 08 02:18:06.917757 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:18:06.917724    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-967ab780-58d56f969d-p7dmb"
Sep 08 02:18:07.304865 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:18:07.304780    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-527ac8/auth-test-model-527ac8-kserve-64d86fff7f-pp528"]
Sep 08 02:18:07.308306 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:18:07.308285    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-527ac8/auth-test-model-527ac8-kserve-64d86fff7f-pp528"
Sep 08 02:18:07.310624 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:18:07.310602    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-527ac8\"/\"openshift-service-ca.crt\""
Sep 08 02:18:07.311344 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:18:07.311327    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-527ac8\"/\"default-dockercfg-zjlj4\""
Sep 08 02:18:07.311413 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:18:07.311348    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-527ac8\"/\"kube-root-ca.crt\""
Sep 08 02:18:07.311413 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:18:07.311336    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-527ac8\"/\"auth-test-model-527ac8-kserve-self-signed-certs\""
Sep 08 02:18:07.317675 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:18:07.317648    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-527ac8/auth-test-model-527ac8-kserve-64d86fff7f-pp528"]
Sep 08 02:18:07.406054 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:18:07.406012    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d433b308-951f-4ea2-be77-ab55d7a8b628-tmp-dir\") pod \"auth-test-model-527ac8-kserve-64d86fff7f-pp528\" (UID: \"d433b308-951f-4ea2-be77-ab55d7a8b628\") " pod="ai-tenant-e2e-shared-b-w3-527ac8/auth-test-model-527ac8-kserve-64d86fff7f-pp528"
Sep 08 02:18:07.406281 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:18:07.406162    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d433b308-951f-4ea2-be77-ab55d7a8b628-home\") pod \"auth-test-model-527ac8-kserve-64d86fff7f-pp528\" (UID: \"d433b308-951f-4ea2-be77-ab55d7a8b628\") " pod="ai-tenant-e2e-shared-b-w3-527ac8/auth-test-model-527ac8-kserve-64d86fff7f-pp528"
Sep 08 02:18:07.406281 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:18:07.406206    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t66tj\" (UniqueName: \"kubernetes.io/projected/d433b308-951f-4ea2-be77-ab55d7a8b628-kube-api-access-t66tj\") pod \"auth-test-model-527ac8-kserve-64d86fff7f-pp528\" (UID: \"d433b308-951f-4ea2-be77-ab55d7a8b628\") " pod="ai-tenant-e2e-shared-b-w3-527ac8/auth-test-model-527ac8-kserve-64d86fff7f-pp528"
Sep 08 02:18:07.406408 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:18:07.406278    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d433b308-951f-4ea2-be77-ab55d7a8b628-tls-certs\") pod \"auth-test-model-527ac8-kserve-64d86fff7f-pp528\" (UID: \"d433b308-951f-4ea2-be77-ab55d7a8b628\") " pod="ai-tenant-e2e-shared-b-w3-527ac8/auth-test-model-527ac8-kserve-64d86fff7f-pp528"
Sep 08 02:18:07.406408 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:18:07.406330    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d433b308-951f-4ea2-be77-ab55d7a8b628-dshm\") pod \"auth-test-model-527ac8-kserve-64d86fff7f-pp528\" (UID: \"d433b308-951f-4ea2-be77-ab55d7a8b628\") " pod="ai-tenant-e2e-shared-b-w3-527ac8/auth-test-model-527ac8-kserve-64d86fff7f-pp528"
Sep 08 02:18:07.406408 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:18:07.406365    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d433b308-951f-4ea2-be77-ab55d7a8b628-model-cache\") pod \"auth-test-model-527ac8-kserve-64d86fff7f-pp528\" (UID: \"d433b308-951f-4ea2-be77-ab55d7a8b628\") " pod="ai-tenant-e2e-shared-b-w3-527ac8/auth-test-model-527ac8-kserve-64d86fff7f-pp528"
Sep 08 02:18:07.506990 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:18:07.506940    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d433b308-951f-4ea2-be77-ab55d7a8b628-home\") pod \"auth-test-model-527ac8-kserve-64d86fff7f-pp528\" (UID: \"d433b308-951f-4ea2-be77-ab55d7a8b628\") " pod="ai-tenant-e2e-shared-b-w3-527ac8/auth-test-model-527ac8-kserve-64d86fff7f-pp528"
Sep 08 02:18:07.506990 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:18:07.506992    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t66tj\" (UniqueName: \"kubernetes.io/projected/d433b308-951f-4ea2-be77-ab55d7a8b628-kube-api-access-t66tj\") pod \"auth-test-model-527ac8-kserve-64d86fff7f-pp528\" (UID: \"d433b308-951f-4ea2-be77-ab55d7a8b628\") " pod="ai-tenant-e2e-shared-b-w3-527ac8/auth-test-model-527ac8-kserve-64d86fff7f-pp528"
Sep 08 02:18:07.507305 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:18:07.507018    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d433b308-951f-4ea2-be77-ab55d7a8b628-tls-certs\") pod \"auth-test-model-527ac8-kserve-64d86fff7f-pp528\" (UID: \"d433b308-951f-4ea2-be77-ab55d7a8b628\") " pod="ai-tenant-e2e-shared-b-w3-527ac8/auth-test-model-527ac8-kserve-64d86fff7f-pp528"
Sep 08 02:18:07.507305 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:18:07.507174    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d433b308-951f-4ea2-be77-ab55d7a8b628-dshm\") pod \"auth-test-model-527ac8-kserve-64d86fff7f-pp528\" (UID: \"d433b308-951f-4ea2-be77-ab55d7a8b628\") " pod="ai-tenant-e2e-shared-b-w3-527ac8/auth-test-model-527ac8-kserve-64d86fff7f-pp528"
Sep 08 02:18:07.507305 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:18:07.507215    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d433b308-951f-4ea2-be77-ab55d7a8b628-model-cache\") pod \"auth-test-model-527ac8-kserve-64d86fff7f-pp528\" (UID: \"d433b308-951f-4ea2-be77-ab55d7a8b628\") " pod="ai-tenant-e2e-shared-b-w3-527ac8/auth-test-model-527ac8-kserve-64d86fff7f-pp528"
Sep 08 02:18:07.507305 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:18:07.507245    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d433b308-951f-4ea2-be77-ab55d7a8b628-tmp-dir\") pod \"auth-test-model-527ac8-kserve-64d86fff7f-pp528\" (UID: \"d433b308-951f-4ea2-be77-ab55d7a8b628\") " pod="ai-tenant-e2e-shared-b-w3-527ac8/auth-test-model-527ac8-kserve-64d86fff7f-pp528"
Sep 08 02:18:07.507521 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:18:07.507473    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d433b308-951f-4ea2-be77-ab55d7a8b628-home\") pod \"auth-test-model-527ac8-kserve-64d86fff7f-pp528\" (UID: \"d433b308-951f-4ea2-be77-ab55d7a8b628\") " pod="ai-tenant-e2e-shared-b-w3-527ac8/auth-test-model-527ac8-kserve-64d86fff7f-pp528"
Sep 08 02:18:07.507563 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:18:07.507549    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d433b308-951f-4ea2-be77-ab55d7a8b628-tmp-dir\") pod \"auth-test-model-527ac8-kserve-64d86fff7f-pp528\" (UID: \"d433b308-951f-4ea2-be77-ab55d7a8b628\") " pod="ai-tenant-e2e-shared-b-w3-527ac8/auth-test-model-527ac8-kserve-64d86fff7f-pp528"
Sep 08 02:18:07.507603 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:18:07.507571    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d433b308-951f-4ea2-be77-ab55d7a8b628-model-cache\") pod \"auth-test-model-527ac8-kserve-64d86fff7f-pp528\" (UID: \"d433b308-951f-4ea2-be77-ab55d7a8b628\") " pod="ai-tenant-e2e-shared-b-w3-527ac8/auth-test-model-527ac8-kserve-64d86fff7f-pp528"
Sep 08 02:18:07.509388 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:18:07.509360    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d433b308-951f-4ea2-be77-ab55d7a8b628-dshm\") pod \"auth-test-model-527ac8-kserve-64d86fff7f-pp528\" (UID: \"d433b308-951f-4ea2-be77-ab55d7a8b628\") " pod="ai-tenant-e2e-shared-b-w3-527ac8/auth-test-model-527ac8-kserve-64d86fff7f-pp528"
Sep 08 02:18:07.509537 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:18:07.509520    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d433b308-951f-4ea2-be77-ab55d7a8b628-tls-certs\") pod \"auth-test-model-527ac8-kserve-64d86fff7f-pp528\" (UID: \"d433b308-951f-4ea2-be77-ab55d7a8b628\") " pod="ai-tenant-e2e-shared-b-w3-527ac8/auth-test-model-527ac8-kserve-64d86fff7f-pp528"
Sep 08 02:18:07.514270 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:18:07.514251    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t66tj\" (UniqueName: \"kubernetes.io/projected/d433b308-951f-4ea2-be77-ab55d7a8b628-kube-api-access-t66tj\") pod \"auth-test-model-527ac8-kserve-64d86fff7f-pp528\" (UID: \"d433b308-951f-4ea2-be77-ab55d7a8b628\") " pod="ai-tenant-e2e-shared-b-w3-527ac8/auth-test-model-527ac8-kserve-64d86fff7f-pp528"
Sep 08 02:18:07.618030 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:18:07.617934    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-527ac8/auth-test-model-527ac8-kserve-64d86fff7f-pp528"
Sep 08 02:18:07.891077 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:18:07.890988    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-527ac8/auth-test-model-527ac8-kserve-64d86fff7f-pp528"]
Sep 08 02:18:07.893585 ip-10-0-142-78 kubenswrapper[2547]: W0908 02:18:07.893558    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd433b308_951f_4ea2_be77_ab55d7a8b628.slice/crio-2a7cf0e05fc239bb4c2a7d5ea32763c44d47050c0eba1a83f24e3b524368d575 WatchSource:0}: Error finding container 2a7cf0e05fc239bb4c2a7d5ea32763c44d47050c0eba1a83f24e3b524368d575: Status 404 returned error can't find the container with id 2a7cf0e05fc239bb4c2a7d5ea32763c44d47050c0eba1a83f24e3b524368d575
Sep 08 02:18:07.968330 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:18:07.968292    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-527ac8/auth-test-model-527ac8-kserve-64d86fff7f-pp528" event={"ID":"d433b308-951f-4ea2-be77-ab55d7a8b628","Type":"ContainerStarted","Data":"d40574098d092e70addc791f4a566aa2a7b4414df62464e21861380c3d75cb77"}
Sep 08 02:18:07.968767 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:18:07.968344    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-527ac8/auth-test-model-527ac8-kserve-64d86fff7f-pp528" event={"ID":"d433b308-951f-4ea2-be77-ab55d7a8b628","Type":"ContainerStarted","Data":"2a7cf0e05fc239bb4c2a7d5ea32763c44d47050c0eba1a83f24e3b524368d575"}
Sep 08 02:18:07.984288 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:18:07.984229    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-527ac8/auth-test-model-527ac8-kserve-64d86fff7f-pp528" podStartSLOduration=0.984212175 podStartE2EDuration="984.212175ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:18:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:18:07.983027041 +0000 UTC m=+1159.506802121" watchObservedRunningTime="2026-09-08 02:18:07.984212175 +0000 UTC m=+1159.507987252"
Sep 08 02:18:17.618682 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:18:17.618571    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-527ac8/auth-test-model-527ac8-kserve-64d86fff7f-pp528"
Sep 08 02:18:17.618682 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:18:17.618641    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-527ac8/auth-test-model-527ac8-kserve-64d86fff7f-pp528"
Sep 08 02:18:17.631500 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:18:17.631470    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-527ac8/auth-test-model-527ac8-kserve-64d86fff7f-pp528"
Sep 08 02:18:18.030166 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:18:18.030116    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-527ac8/auth-test-model-527ac8-kserve-64d86fff7f-pp528"
Sep 08 02:18:24.312409 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:18:24.312360    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-967ab780/e2e-lifecycle-model-967ab780-kserve-5d87fddf6c-ckhfn"]
Sep 08 02:18:24.315757 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:18:24.315738    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-967ab780/e2e-lifecycle-model-967ab780-kserve-5d87fddf6c-ckhfn"
Sep 08 02:18:24.317883 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:18:24.317854    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-967ab780\"/\"openshift-service-ca.crt\""
Sep 08 02:18:24.318038 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:18:24.317881    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-967ab780\"/\"kube-root-ca.crt\""
Sep 08 02:18:24.318728 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:18:24.318707    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-967ab780\"/\"default-dockercfg-gwq4t\""
Sep 08 02:18:24.318814 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:18:24.318744    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-967ab780\"/\"e2e-lifecycle-model-967ab780-kserve-self-signed-certs\""
Sep 08 02:18:24.325100 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:18:24.325070    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-967ab780/e2e-lifecycle-model-967ab780-kserve-5d87fddf6c-ckhfn"]
Sep 08 02:18:24.467032 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:18:24.466991    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78758944-4e05-4da6-be65-0f893a296f5b-tmp-dir\") pod \"e2e-lifecycle-model-967ab780-kserve-5d87fddf6c-ckhfn\" (UID: \"78758944-4e05-4da6-be65-0f893a296f5b\") " pod="ai-tenant-e2e-mt-967ab780/e2e-lifecycle-model-967ab780-kserve-5d87fddf6c-ckhfn"
Sep 08 02:18:24.467032 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:18:24.467036    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78758944-4e05-4da6-be65-0f893a296f5b-model-cache\") pod \"e2e-lifecycle-model-967ab780-kserve-5d87fddf6c-ckhfn\" (UID: \"78758944-4e05-4da6-be65-0f893a296f5b\") " pod="ai-tenant-e2e-mt-967ab780/e2e-lifecycle-model-967ab780-kserve-5d87fddf6c-ckhfn"
Sep 08 02:18:24.467305 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:18:24.467106    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgkhs\" (UniqueName: \"kubernetes.io/projected/78758944-4e05-4da6-be65-0f893a296f5b-kube-api-access-hgkhs\") pod \"e2e-lifecycle-model-967ab780-kserve-5d87fddf6c-ckhfn\" (UID: \"78758944-4e05-4da6-be65-0f893a296f5b\") " pod="ai-tenant-e2e-mt-967ab780/e2e-lifecycle-model-967ab780-kserve-5d87fddf6c-ckhfn"
Sep 08 02:18:24.467305 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:18:24.467233    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78758944-4e05-4da6-be65-0f893a296f5b-dshm\") pod \"e2e-lifecycle-model-967ab780-kserve-5d87fddf6c-ckhfn\" (UID: \"78758944-4e05-4da6-be65-0f893a296f5b\") " pod="ai-tenant-e2e-mt-967ab780/e2e-lifecycle-model-967ab780-kserve-5d87fddf6c-ckhfn"
Sep 08 02:18:24.467305 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:18:24.467268    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78758944-4e05-4da6-be65-0f893a296f5b-home\") pod \"e2e-lifecycle-model-967ab780-kserve-5d87fddf6c-ckhfn\" (UID: \"78758944-4e05-4da6-be65-0f893a296f5b\") " pod="ai-tenant-e2e-mt-967ab780/e2e-lifecycle-model-967ab780-kserve-5d87fddf6c-ckhfn"
Sep 08 02:18:24.467305 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:18:24.467292    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78758944-4e05-4da6-be65-0f893a296f5b-tls-certs\") pod \"e2e-lifecycle-model-967ab780-kserve-5d87fddf6c-ckhfn\" (UID: \"78758944-4e05-4da6-be65-0f893a296f5b\") " pod="ai-tenant-e2e-mt-967ab780/e2e-lifecycle-model-967ab780-kserve-5d87fddf6c-ckhfn"
Sep 08 02:18:24.568619 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:18:24.568518    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78758944-4e05-4da6-be65-0f893a296f5b-dshm\") pod \"e2e-lifecycle-model-967ab780-kserve-5d87fddf6c-ckhfn\" (UID: \"78758944-4e05-4da6-be65-0f893a296f5b\") " pod="ai-tenant-e2e-mt-967ab780/e2e-lifecycle-model-967ab780-kserve-5d87fddf6c-ckhfn"
Sep 08 02:18:24.568619 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:18:24.568588    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78758944-4e05-4da6-be65-0f893a296f5b-home\") pod \"e2e-lifecycle-model-967ab780-kserve-5d87fddf6c-ckhfn\" (UID: \"78758944-4e05-4da6-be65-0f893a296f5b\") " pod="ai-tenant-e2e-mt-967ab780/e2e-lifecycle-model-967ab780-kserve-5d87fddf6c-ckhfn"
Sep 08 02:18:24.568619 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:18:24.568613    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78758944-4e05-4da6-be65-0f893a296f5b-tls-certs\") pod \"e2e-lifecycle-model-967ab780-kserve-5d87fddf6c-ckhfn\" (UID: \"78758944-4e05-4da6-be65-0f893a296f5b\") " pod="ai-tenant-e2e-mt-967ab780/e2e-lifecycle-model-967ab780-kserve-5d87fddf6c-ckhfn"
Sep 08 02:18:24.568942 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:18:24.568667    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78758944-4e05-4da6-be65-0f893a296f5b-tmp-dir\") pod \"e2e-lifecycle-model-967ab780-kserve-5d87fddf6c-ckhfn\" (UID: \"78758944-4e05-4da6-be65-0f893a296f5b\") " pod="ai-tenant-e2e-mt-967ab780/e2e-lifecycle-model-967ab780-kserve-5d87fddf6c-ckhfn"
Sep 08 02:18:24.568942 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:18:24.568709    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78758944-4e05-4da6-be65-0f893a296f5b-model-cache\") pod \"e2e-lifecycle-model-967ab780-kserve-5d87fddf6c-ckhfn\" (UID: \"78758944-4e05-4da6-be65-0f893a296f5b\") " pod="ai-tenant-e2e-mt-967ab780/e2e-lifecycle-model-967ab780-kserve-5d87fddf6c-ckhfn"
Sep 08 02:18:24.568942 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:18:24.568754    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgkhs\" (UniqueName: \"kubernetes.io/projected/78758944-4e05-4da6-be65-0f893a296f5b-kube-api-access-hgkhs\") pod \"e2e-lifecycle-model-967ab780-kserve-5d87fddf6c-ckhfn\" (UID: \"78758944-4e05-4da6-be65-0f893a296f5b\") " pod="ai-tenant-e2e-mt-967ab780/e2e-lifecycle-model-967ab780-kserve-5d87fddf6c-ckhfn"
Sep 08 02:18:24.569099 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:18:24.569040    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78758944-4e05-4da6-be65-0f893a296f5b-home\") pod \"e2e-lifecycle-model-967ab780-kserve-5d87fddf6c-ckhfn\" (UID: \"78758944-4e05-4da6-be65-0f893a296f5b\") " pod="ai-tenant-e2e-mt-967ab780/e2e-lifecycle-model-967ab780-kserve-5d87fddf6c-ckhfn"
Sep 08 02:18:24.569099 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:18:24.569068    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78758944-4e05-4da6-be65-0f893a296f5b-tmp-dir\") pod \"e2e-lifecycle-model-967ab780-kserve-5d87fddf6c-ckhfn\" (UID: \"78758944-4e05-4da6-be65-0f893a296f5b\") " pod="ai-tenant-e2e-mt-967ab780/e2e-lifecycle-model-967ab780-kserve-5d87fddf6c-ckhfn"
Sep 08 02:18:24.569255 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:18:24.569223    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78758944-4e05-4da6-be65-0f893a296f5b-model-cache\") pod \"e2e-lifecycle-model-967ab780-kserve-5d87fddf6c-ckhfn\" (UID: \"78758944-4e05-4da6-be65-0f893a296f5b\") " pod="ai-tenant-e2e-mt-967ab780/e2e-lifecycle-model-967ab780-kserve-5d87fddf6c-ckhfn"
Sep 08 02:18:24.570914 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:18:24.570894    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78758944-4e05-4da6-be65-0f893a296f5b-dshm\") pod \"e2e-lifecycle-model-967ab780-kserve-5d87fddf6c-ckhfn\" (UID: \"78758944-4e05-4da6-be65-0f893a296f5b\") " pod="ai-tenant-e2e-mt-967ab780/e2e-lifecycle-model-967ab780-kserve-5d87fddf6c-ckhfn"
Sep 08 02:18:24.571016 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:18:24.571008    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78758944-4e05-4da6-be65-0f893a296f5b-tls-certs\") pod \"e2e-lifecycle-model-967ab780-kserve-5d87fddf6c-ckhfn\" (UID: \"78758944-4e05-4da6-be65-0f893a296f5b\") " pod="ai-tenant-e2e-mt-967ab780/e2e-lifecycle-model-967ab780-kserve-5d87fddf6c-ckhfn"
Sep 08 02:18:24.577815 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:18:24.577785    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgkhs\" (UniqueName: \"kubernetes.io/projected/78758944-4e05-4da6-be65-0f893a296f5b-kube-api-access-hgkhs\") pod \"e2e-lifecycle-model-967ab780-kserve-5d87fddf6c-ckhfn\" (UID: \"78758944-4e05-4da6-be65-0f893a296f5b\") " pod="ai-tenant-e2e-mt-967ab780/e2e-lifecycle-model-967ab780-kserve-5d87fddf6c-ckhfn"
Sep 08 02:18:24.625755 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:18:24.625714    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-967ab780/e2e-lifecycle-model-967ab780-kserve-5d87fddf6c-ckhfn"
Sep 08 02:18:24.898520 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:18:24.898470    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-967ab780/e2e-lifecycle-model-967ab780-kserve-5d87fddf6c-ckhfn"]
Sep 08 02:18:24.901848 ip-10-0-142-78 kubenswrapper[2547]: W0908 02:18:24.901815    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod78758944_4e05_4da6_be65_0f893a296f5b.slice/crio-553b9eeac1b081334813f45749c780a1b251333982496d161d1267f9f45f6e32 WatchSource:0}: Error finding container 553b9eeac1b081334813f45749c780a1b251333982496d161d1267f9f45f6e32: Status 404 returned error can't find the container with id 553b9eeac1b081334813f45749c780a1b251333982496d161d1267f9f45f6e32
Sep 08 02:18:25.050882 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:18:25.050843    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-967ab780/e2e-lifecycle-model-967ab780-kserve-5d87fddf6c-ckhfn" event={"ID":"78758944-4e05-4da6-be65-0f893a296f5b","Type":"ContainerStarted","Data":"0f0bcbc41cbe91331b309572bd552d3d2cadce1e33251811064b4249fd2b0cc5"}
Sep 08 02:18:25.051057 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:18:25.050890    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-967ab780/e2e-lifecycle-model-967ab780-kserve-5d87fddf6c-ckhfn" event={"ID":"78758944-4e05-4da6-be65-0f893a296f5b","Type":"ContainerStarted","Data":"553b9eeac1b081334813f45749c780a1b251333982496d161d1267f9f45f6e32"}
Sep 08 02:18:25.066055 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:18:25.065998    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-967ab780/e2e-lifecycle-model-967ab780-kserve-5d87fddf6c-ckhfn" podStartSLOduration=1.065981815 podStartE2EDuration="1.065981815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:18:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:18:25.064539019 +0000 UTC m=+1176.588314102" watchObservedRunningTime="2026-09-08 02:18:25.065981815 +0000 UTC m=+1176.589756894"
Sep 08 02:18:34.626176 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:18:34.626115    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-967ab780/e2e-lifecycle-model-967ab780-kserve-5d87fddf6c-ckhfn"
Sep 08 02:18:34.626176 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:18:34.626181    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-967ab780/e2e-lifecycle-model-967ab780-kserve-5d87fddf6c-ckhfn"
Sep 08 02:18:34.638876 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:18:34.638848    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-967ab780/e2e-lifecycle-model-967ab780-kserve-5d87fddf6c-ckhfn"
Sep 08 02:18:35.110268 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:18:35.110238    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-967ab780/e2e-lifecycle-model-967ab780-kserve-5d87fddf6c-ckhfn"
Sep 08 02:19:07.573300 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:19:07.573265    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-967ab780-58d56f969d-p7dmb"]
Sep 08 02:19:07.573859 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:19:07.573482    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-967ab780-58d56f969d-p7dmb" podUID="89d05519-3e99-4dd8-8f80-b24eb233ddaa" containerName="payload-pre-processing" containerID="cri-o://872850ae23f8e49ca3d52f4e726acd6bc69b25fa4a3c5849355b49c3d2a9c687" gracePeriod=30
Sep 08 02:19:07.834926 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:19:07.834859    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-967ab780-58d56f969d-p7dmb"
Sep 08 02:19:07.965037 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:19:07.964998    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fx7tj\" (UniqueName: \"kubernetes.io/projected/89d05519-3e99-4dd8-8f80-b24eb233ddaa-kube-api-access-fx7tj\") pod \"89d05519-3e99-4dd8-8f80-b24eb233ddaa\" (UID: \"89d05519-3e99-4dd8-8f80-b24eb233ddaa\") "
Sep 08 02:19:07.965252 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:19:07.965078    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89d05519-3e99-4dd8-8f80-b24eb233ddaa-plugins-config-volume\") pod \"89d05519-3e99-4dd8-8f80-b24eb233ddaa\" (UID: \"89d05519-3e99-4dd8-8f80-b24eb233ddaa\") "
Sep 08 02:19:07.965481 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:19:07.965457    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89d05519-3e99-4dd8-8f80-b24eb233ddaa-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "89d05519-3e99-4dd8-8f80-b24eb233ddaa" (UID: "89d05519-3e99-4dd8-8f80-b24eb233ddaa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 02:19:07.967150 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:19:07.967114    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89d05519-3e99-4dd8-8f80-b24eb233ddaa-kube-api-access-fx7tj" (OuterVolumeSpecName: "kube-api-access-fx7tj") pod "89d05519-3e99-4dd8-8f80-b24eb233ddaa" (UID: "89d05519-3e99-4dd8-8f80-b24eb233ddaa"). InnerVolumeSpecName "kube-api-access-fx7tj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 02:19:08.066512 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:19:08.066470    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fx7tj\" (UniqueName: \"kubernetes.io/projected/89d05519-3e99-4dd8-8f80-b24eb233ddaa-kube-api-access-fx7tj\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:19:08.066512 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:19:08.066505    2547 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89d05519-3e99-4dd8-8f80-b24eb233ddaa-plugins-config-volume\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:19:08.256311 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:19:08.256271    2547 generic.go:356] "Generic (PLEG): container finished" podID="89d05519-3e99-4dd8-8f80-b24eb233ddaa" containerID="872850ae23f8e49ca3d52f4e726acd6bc69b25fa4a3c5849355b49c3d2a9c687" exitCode=0
Sep 08 02:19:08.256480 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:19:08.256316    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-967ab780-58d56f969d-p7dmb"
Sep 08 02:19:08.256480 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:19:08.256361    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-967ab780-58d56f969d-p7dmb" event={"ID":"89d05519-3e99-4dd8-8f80-b24eb233ddaa","Type":"ContainerDied","Data":"872850ae23f8e49ca3d52f4e726acd6bc69b25fa4a3c5849355b49c3d2a9c687"}
Sep 08 02:19:08.256480 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:19:08.256417    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-967ab780-58d56f969d-p7dmb" event={"ID":"89d05519-3e99-4dd8-8f80-b24eb233ddaa","Type":"ContainerDied","Data":"f45eff483f41af8bee7cedf59ecf67386dded1017c30f5311d9af90bcc082f25"}
Sep 08 02:19:08.256480 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:19:08.256448    2547 scope.go:117] "RemoveContainer" containerID="872850ae23f8e49ca3d52f4e726acd6bc69b25fa4a3c5849355b49c3d2a9c687"
Sep 08 02:19:08.265663 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:19:08.265645    2547 scope.go:117] "RemoveContainer" containerID="872850ae23f8e49ca3d52f4e726acd6bc69b25fa4a3c5849355b49c3d2a9c687"
Sep 08 02:19:08.265923 ip-10-0-142-78 kubenswrapper[2547]: E0908 02:19:08.265905    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"872850ae23f8e49ca3d52f4e726acd6bc69b25fa4a3c5849355b49c3d2a9c687\": container with ID starting with 872850ae23f8e49ca3d52f4e726acd6bc69b25fa4a3c5849355b49c3d2a9c687 not found: ID does not exist" containerID="872850ae23f8e49ca3d52f4e726acd6bc69b25fa4a3c5849355b49c3d2a9c687"
Sep 08 02:19:08.265969 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:19:08.265933    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"872850ae23f8e49ca3d52f4e726acd6bc69b25fa4a3c5849355b49c3d2a9c687"} err="failed to get container status \"872850ae23f8e49ca3d52f4e726acd6bc69b25fa4a3c5849355b49c3d2a9c687\": rpc error: code = NotFound desc = could not find container \"872850ae23f8e49ca3d52f4e726acd6bc69b25fa4a3c5849355b49c3d2a9c687\": container with ID starting with 872850ae23f8e49ca3d52f4e726acd6bc69b25fa4a3c5849355b49c3d2a9c687 not found: ID does not exist"
Sep 08 02:19:08.276570 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:19:08.276542    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-967ab780-58d56f969d-p7dmb"]
Sep 08 02:19:08.279988 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:19:08.279963    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-967ab780-58d56f969d-p7dmb"]
Sep 08 02:19:08.993846 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:19:08.993808    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89d05519-3e99-4dd8-8f80-b24eb233ddaa" path="/var/lib/kubelet/pods/89d05519-3e99-4dd8-8f80-b24eb233ddaa/volumes"
Sep 08 02:19:15.628005 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:19:15.627969    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-967ab780/e2e-lifecycle-model-967ab780-kserve-5d87fddf6c-ckhfn"]
Sep 08 02:19:30.647894 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:19:30.647826    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-967ab780/e2e-lifecycle-model-967ab780-kserve-5d87fddf6c-ckhfn" podUID="78758944-4e05-4da6-be65-0f893a296f5b" containerName="main" containerID="cri-o://0f0bcbc41cbe91331b309572bd552d3d2cadce1e33251811064b4249fd2b0cc5" gracePeriod=45
Sep 08 02:19:30.905768 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:19:30.905704    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-967ab780/e2e-lifecycle-model-967ab780-kserve-5d87fddf6c-ckhfn"
Sep 08 02:19:30.971838 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:19:30.971804    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hgkhs\" (UniqueName: \"kubernetes.io/projected/78758944-4e05-4da6-be65-0f893a296f5b-kube-api-access-hgkhs\") pod \"78758944-4e05-4da6-be65-0f893a296f5b\" (UID: \"78758944-4e05-4da6-be65-0f893a296f5b\") "
Sep 08 02:19:30.971838 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:19:30.971843    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78758944-4e05-4da6-be65-0f893a296f5b-dshm\") pod \"78758944-4e05-4da6-be65-0f893a296f5b\" (UID: \"78758944-4e05-4da6-be65-0f893a296f5b\") "
Sep 08 02:19:30.972299 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:19:30.971889    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78758944-4e05-4da6-be65-0f893a296f5b-tls-certs\") pod \"78758944-4e05-4da6-be65-0f893a296f5b\" (UID: \"78758944-4e05-4da6-be65-0f893a296f5b\") "
Sep 08 02:19:30.972299 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:19:30.971920    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78758944-4e05-4da6-be65-0f893a296f5b-tmp-dir\") pod \"78758944-4e05-4da6-be65-0f893a296f5b\" (UID: \"78758944-4e05-4da6-be65-0f893a296f5b\") "
Sep 08 02:19:30.972299 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:19:30.971997    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78758944-4e05-4da6-be65-0f893a296f5b-home\") pod \"78758944-4e05-4da6-be65-0f893a296f5b\" (UID: \"78758944-4e05-4da6-be65-0f893a296f5b\") "
Sep 08 02:19:30.972299 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:19:30.972023    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78758944-4e05-4da6-be65-0f893a296f5b-model-cache\") pod \"78758944-4e05-4da6-be65-0f893a296f5b\" (UID: \"78758944-4e05-4da6-be65-0f893a296f5b\") "
Sep 08 02:19:30.972527 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:19:30.972303    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78758944-4e05-4da6-be65-0f893a296f5b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "78758944-4e05-4da6-be65-0f893a296f5b" (UID: "78758944-4e05-4da6-be65-0f893a296f5b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 02:19:30.972527 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:19:30.972312    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78758944-4e05-4da6-be65-0f893a296f5b-home" (OuterVolumeSpecName: "home") pod "78758944-4e05-4da6-be65-0f893a296f5b" (UID: "78758944-4e05-4da6-be65-0f893a296f5b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 02:19:30.972527 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:19:30.972474    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78758944-4e05-4da6-be65-0f893a296f5b-model-cache" (OuterVolumeSpecName: "model-cache") pod "78758944-4e05-4da6-be65-0f893a296f5b" (UID: "78758944-4e05-4da6-be65-0f893a296f5b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 02:19:30.974193 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:19:30.974121    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78758944-4e05-4da6-be65-0f893a296f5b-kube-api-access-hgkhs" (OuterVolumeSpecName: "kube-api-access-hgkhs") pod "78758944-4e05-4da6-be65-0f893a296f5b" (UID: "78758944-4e05-4da6-be65-0f893a296f5b"). InnerVolumeSpecName "kube-api-access-hgkhs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 02:19:30.974458 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:19:30.974436    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78758944-4e05-4da6-be65-0f893a296f5b-dshm" (OuterVolumeSpecName: "dshm") pod "78758944-4e05-4da6-be65-0f893a296f5b" (UID: "78758944-4e05-4da6-be65-0f893a296f5b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 02:19:30.974505 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:19:30.974469    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78758944-4e05-4da6-be65-0f893a296f5b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "78758944-4e05-4da6-be65-0f893a296f5b" (UID: "78758944-4e05-4da6-be65-0f893a296f5b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 02:19:31.072955 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:19:31.072916    2547 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78758944-4e05-4da6-be65-0f893a296f5b-home\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:19:31.072955 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:19:31.072949    2547 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78758944-4e05-4da6-be65-0f893a296f5b-model-cache\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:19:31.072955 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:19:31.072962    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgkhs\" (UniqueName: \"kubernetes.io/projected/78758944-4e05-4da6-be65-0f893a296f5b-kube-api-access-hgkhs\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:19:31.073228 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:19:31.072972    2547 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78758944-4e05-4da6-be65-0f893a296f5b-dshm\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:19:31.073228 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:19:31.072981    2547 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78758944-4e05-4da6-be65-0f893a296f5b-tls-certs\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:19:31.073228 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:19:31.072988    2547 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78758944-4e05-4da6-be65-0f893a296f5b-tmp-dir\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:19:31.369919 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:19:31.369833    2547 generic.go:356] "Generic (PLEG): container finished" podID="78758944-4e05-4da6-be65-0f893a296f5b" containerID="0f0bcbc41cbe91331b309572bd552d3d2cadce1e33251811064b4249fd2b0cc5" exitCode=0
Sep 08 02:19:31.369919 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:19:31.369878    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-967ab780/e2e-lifecycle-model-967ab780-kserve-5d87fddf6c-ckhfn"
Sep 08 02:19:31.370100 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:19:31.369914    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-967ab780/e2e-lifecycle-model-967ab780-kserve-5d87fddf6c-ckhfn" event={"ID":"78758944-4e05-4da6-be65-0f893a296f5b","Type":"ContainerDied","Data":"0f0bcbc41cbe91331b309572bd552d3d2cadce1e33251811064b4249fd2b0cc5"}
Sep 08 02:19:31.370100 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:19:31.369957    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-967ab780/e2e-lifecycle-model-967ab780-kserve-5d87fddf6c-ckhfn" event={"ID":"78758944-4e05-4da6-be65-0f893a296f5b","Type":"ContainerDied","Data":"553b9eeac1b081334813f45749c780a1b251333982496d161d1267f9f45f6e32"}
Sep 08 02:19:31.370100 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:19:31.369981    2547 scope.go:117] "RemoveContainer" containerID="0f0bcbc41cbe91331b309572bd552d3d2cadce1e33251811064b4249fd2b0cc5"
Sep 08 02:19:31.378295 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:19:31.378274    2547 scope.go:117] "RemoveContainer" containerID="0f0bcbc41cbe91331b309572bd552d3d2cadce1e33251811064b4249fd2b0cc5"
Sep 08 02:19:31.378582 ip-10-0-142-78 kubenswrapper[2547]: E0908 02:19:31.378564    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f0bcbc41cbe91331b309572bd552d3d2cadce1e33251811064b4249fd2b0cc5\": container with ID starting with 0f0bcbc41cbe91331b309572bd552d3d2cadce1e33251811064b4249fd2b0cc5 not found: ID does not exist" containerID="0f0bcbc41cbe91331b309572bd552d3d2cadce1e33251811064b4249fd2b0cc5"
Sep 08 02:19:31.378623 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:19:31.378593    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f0bcbc41cbe91331b309572bd552d3d2cadce1e33251811064b4249fd2b0cc5"} err="failed to get container status \"0f0bcbc41cbe91331b309572bd552d3d2cadce1e33251811064b4249fd2b0cc5\": rpc error: code = NotFound desc = could not find container \"0f0bcbc41cbe91331b309572bd552d3d2cadce1e33251811064b4249fd2b0cc5\": container with ID starting with 0f0bcbc41cbe91331b309572bd552d3d2cadce1e33251811064b4249fd2b0cc5 not found: ID does not exist"
Sep 08 02:19:31.384696 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:19:31.384668    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-967ab780/e2e-lifecycle-model-967ab780-kserve-5d87fddf6c-ckhfn"]
Sep 08 02:19:31.388456 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:19:31.388432    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-967ab780/e2e-lifecycle-model-967ab780-kserve-5d87fddf6c-ckhfn"]
Sep 08 02:19:32.994123 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:19:32.994089    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78758944-4e05-4da6-be65-0f893a296f5b" path="/var/lib/kubelet/pods/78758944-4e05-4da6-be65-0f893a296f5b/volumes"
Sep 08 02:20:13.306445 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:20:13.306408    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-527ac8/auth-test-model-527ac8-kserve-64d86fff7f-pp528"]
Sep 08 02:20:18.502545 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:20:18.502502    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-fcdd46/sub-test-model-fcdd46-kserve-5b968649fc-krgrv"]
Sep 08 02:20:18.503076 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:20:18.503056    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78758944-4e05-4da6-be65-0f893a296f5b" containerName="main"
Sep 08 02:20:18.503180 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:20:18.503079    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="78758944-4e05-4da6-be65-0f893a296f5b" containerName="main"
Sep 08 02:20:18.503180 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:20:18.503111    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89d05519-3e99-4dd8-8f80-b24eb233ddaa" containerName="payload-pre-processing"
Sep 08 02:20:18.503180 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:20:18.503121    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="89d05519-3e99-4dd8-8f80-b24eb233ddaa" containerName="payload-pre-processing"
Sep 08 02:20:18.503339 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:20:18.503248    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="89d05519-3e99-4dd8-8f80-b24eb233ddaa" containerName="payload-pre-processing"
Sep 08 02:20:18.503339 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:20:18.503263    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="78758944-4e05-4da6-be65-0f893a296f5b" containerName="main"
Sep 08 02:20:18.506130 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:20:18.506111    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-fcdd46/sub-test-model-fcdd46-kserve-5b968649fc-krgrv"
Sep 08 02:20:18.508482 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:20:18.508455    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-fcdd46\"/\"kube-root-ca.crt\""
Sep 08 02:20:18.509385 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:20:18.509364    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-fcdd46\"/\"default-dockercfg-z4qcn\""
Sep 08 02:20:18.509385 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:20:18.509376    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-fcdd46\"/\"openshift-service-ca.crt\""
Sep 08 02:20:18.509549 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:20:18.509395    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-fcdd46\"/\"sub-test-model-fcdd46-kserve-self-signed-certs\""
Sep 08 02:20:18.516446 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:20:18.516424    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-fcdd46/sub-test-model-fcdd46-kserve-5b968649fc-krgrv"]
Sep 08 02:20:18.597596 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:20:18.597550    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvrs5\" (UniqueName: \"kubernetes.io/projected/9f31a2d6-de1c-4efa-a301-33622285044e-kube-api-access-jvrs5\") pod \"sub-test-model-fcdd46-kserve-5b968649fc-krgrv\" (UID: \"9f31a2d6-de1c-4efa-a301-33622285044e\") " pod="ai-tenant-e2e-shared-a-w3-fcdd46/sub-test-model-fcdd46-kserve-5b968649fc-krgrv"
Sep 08 02:20:18.597882 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:20:18.597643    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f31a2d6-de1c-4efa-a301-33622285044e-dshm\") pod \"sub-test-model-fcdd46-kserve-5b968649fc-krgrv\" (UID: \"9f31a2d6-de1c-4efa-a301-33622285044e\") " pod="ai-tenant-e2e-shared-a-w3-fcdd46/sub-test-model-fcdd46-kserve-5b968649fc-krgrv"
Sep 08 02:20:18.597882 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:20:18.597666    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f31a2d6-de1c-4efa-a301-33622285044e-tmp-dir\") pod \"sub-test-model-fcdd46-kserve-5b968649fc-krgrv\" (UID: \"9f31a2d6-de1c-4efa-a301-33622285044e\") " pod="ai-tenant-e2e-shared-a-w3-fcdd46/sub-test-model-fcdd46-kserve-5b968649fc-krgrv"
Sep 08 02:20:18.597882 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:20:18.597701    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f31a2d6-de1c-4efa-a301-33622285044e-model-cache\") pod \"sub-test-model-fcdd46-kserve-5b968649fc-krgrv\" (UID: \"9f31a2d6-de1c-4efa-a301-33622285044e\") " pod="ai-tenant-e2e-shared-a-w3-fcdd46/sub-test-model-fcdd46-kserve-5b968649fc-krgrv"
Sep 08 02:20:18.597882 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:20:18.597812    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f31a2d6-de1c-4efa-a301-33622285044e-tls-certs\") pod \"sub-test-model-fcdd46-kserve-5b968649fc-krgrv\" (UID: \"9f31a2d6-de1c-4efa-a301-33622285044e\") " pod="ai-tenant-e2e-shared-a-w3-fcdd46/sub-test-model-fcdd46-kserve-5b968649fc-krgrv"
Sep 08 02:20:18.598053 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:20:18.597882    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f31a2d6-de1c-4efa-a301-33622285044e-home\") pod \"sub-test-model-fcdd46-kserve-5b968649fc-krgrv\" (UID: \"9f31a2d6-de1c-4efa-a301-33622285044e\") " pod="ai-tenant-e2e-shared-a-w3-fcdd46/sub-test-model-fcdd46-kserve-5b968649fc-krgrv"
Sep 08 02:20:18.699060 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:20:18.699026    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f31a2d6-de1c-4efa-a301-33622285044e-home\") pod \"sub-test-model-fcdd46-kserve-5b968649fc-krgrv\" (UID: \"9f31a2d6-de1c-4efa-a301-33622285044e\") " pod="ai-tenant-e2e-shared-a-w3-fcdd46/sub-test-model-fcdd46-kserve-5b968649fc-krgrv"
Sep 08 02:20:18.699266 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:20:18.699066    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvrs5\" (UniqueName: \"kubernetes.io/projected/9f31a2d6-de1c-4efa-a301-33622285044e-kube-api-access-jvrs5\") pod \"sub-test-model-fcdd46-kserve-5b968649fc-krgrv\" (UID: \"9f31a2d6-de1c-4efa-a301-33622285044e\") " pod="ai-tenant-e2e-shared-a-w3-fcdd46/sub-test-model-fcdd46-kserve-5b968649fc-krgrv"
Sep 08 02:20:18.699266 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:20:18.699159    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f31a2d6-de1c-4efa-a301-33622285044e-dshm\") pod \"sub-test-model-fcdd46-kserve-5b968649fc-krgrv\" (UID: \"9f31a2d6-de1c-4efa-a301-33622285044e\") " pod="ai-tenant-e2e-shared-a-w3-fcdd46/sub-test-model-fcdd46-kserve-5b968649fc-krgrv"
Sep 08 02:20:18.699266 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:20:18.699187    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f31a2d6-de1c-4efa-a301-33622285044e-tmp-dir\") pod \"sub-test-model-fcdd46-kserve-5b968649fc-krgrv\" (UID: \"9f31a2d6-de1c-4efa-a301-33622285044e\") " pod="ai-tenant-e2e-shared-a-w3-fcdd46/sub-test-model-fcdd46-kserve-5b968649fc-krgrv"
Sep 08 02:20:18.699266 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:20:18.699224    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f31a2d6-de1c-4efa-a301-33622285044e-model-cache\") pod \"sub-test-model-fcdd46-kserve-5b968649fc-krgrv\" (UID: \"9f31a2d6-de1c-4efa-a301-33622285044e\") " pod="ai-tenant-e2e-shared-a-w3-fcdd46/sub-test-model-fcdd46-kserve-5b968649fc-krgrv"
Sep 08 02:20:18.699472 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:20:18.699283    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f31a2d6-de1c-4efa-a301-33622285044e-tls-certs\") pod \"sub-test-model-fcdd46-kserve-5b968649fc-krgrv\" (UID: \"9f31a2d6-de1c-4efa-a301-33622285044e\") " pod="ai-tenant-e2e-shared-a-w3-fcdd46/sub-test-model-fcdd46-kserve-5b968649fc-krgrv"
Sep 08 02:20:18.699554 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:20:18.699531    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f31a2d6-de1c-4efa-a301-33622285044e-home\") pod \"sub-test-model-fcdd46-kserve-5b968649fc-krgrv\" (UID: \"9f31a2d6-de1c-4efa-a301-33622285044e\") " pod="ai-tenant-e2e-shared-a-w3-fcdd46/sub-test-model-fcdd46-kserve-5b968649fc-krgrv"
Sep 08 02:20:18.699628 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:20:18.699610    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f31a2d6-de1c-4efa-a301-33622285044e-tmp-dir\") pod \"sub-test-model-fcdd46-kserve-5b968649fc-krgrv\" (UID: \"9f31a2d6-de1c-4efa-a301-33622285044e\") " pod="ai-tenant-e2e-shared-a-w3-fcdd46/sub-test-model-fcdd46-kserve-5b968649fc-krgrv"
Sep 08 02:20:18.699690 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:20:18.699668    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f31a2d6-de1c-4efa-a301-33622285044e-model-cache\") pod \"sub-test-model-fcdd46-kserve-5b968649fc-krgrv\" (UID: \"9f31a2d6-de1c-4efa-a301-33622285044e\") " pod="ai-tenant-e2e-shared-a-w3-fcdd46/sub-test-model-fcdd46-kserve-5b968649fc-krgrv"
Sep 08 02:20:18.701513 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:20:18.701490    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f31a2d6-de1c-4efa-a301-33622285044e-dshm\") pod \"sub-test-model-fcdd46-kserve-5b968649fc-krgrv\" (UID: \"9f31a2d6-de1c-4efa-a301-33622285044e\") " pod="ai-tenant-e2e-shared-a-w3-fcdd46/sub-test-model-fcdd46-kserve-5b968649fc-krgrv"
Sep 08 02:20:18.701629 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:20:18.701612    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f31a2d6-de1c-4efa-a301-33622285044e-tls-certs\") pod \"sub-test-model-fcdd46-kserve-5b968649fc-krgrv\" (UID: \"9f31a2d6-de1c-4efa-a301-33622285044e\") " pod="ai-tenant-e2e-shared-a-w3-fcdd46/sub-test-model-fcdd46-kserve-5b968649fc-krgrv"
Sep 08 02:20:18.706886 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:20:18.706864    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvrs5\" (UniqueName: \"kubernetes.io/projected/9f31a2d6-de1c-4efa-a301-33622285044e-kube-api-access-jvrs5\") pod \"sub-test-model-fcdd46-kserve-5b968649fc-krgrv\" (UID: \"9f31a2d6-de1c-4efa-a301-33622285044e\") " pod="ai-tenant-e2e-shared-a-w3-fcdd46/sub-test-model-fcdd46-kserve-5b968649fc-krgrv"
Sep 08 02:20:18.817272 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:20:18.817178    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-fcdd46/sub-test-model-fcdd46-kserve-5b968649fc-krgrv"
Sep 08 02:20:19.087557 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:20:19.087533    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-fcdd46/sub-test-model-fcdd46-kserve-5b968649fc-krgrv"]
Sep 08 02:20:19.089654 ip-10-0-142-78 kubenswrapper[2547]: W0908 02:20:19.089627    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9f31a2d6_de1c_4efa_a301_33622285044e.slice/crio-4c4e697c9b2dbd53996395a0119ab797319619351d59614d3a8d426958344c82 WatchSource:0}: Error finding container 4c4e697c9b2dbd53996395a0119ab797319619351d59614d3a8d426958344c82: Status 404 returned error can't find the container with id 4c4e697c9b2dbd53996395a0119ab797319619351d59614d3a8d426958344c82
Sep 08 02:20:19.608453 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:20:19.608413    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-fcdd46/sub-test-model-fcdd46-kserve-5b968649fc-krgrv" event={"ID":"9f31a2d6-de1c-4efa-a301-33622285044e","Type":"ContainerStarted","Data":"6c3285bdb97f77213d5286b68b3d52ede5e5a17c06c8ace2f9b0b9d994e442e3"}
Sep 08 02:20:19.608944 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:20:19.608461    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-fcdd46/sub-test-model-fcdd46-kserve-5b968649fc-krgrv" event={"ID":"9f31a2d6-de1c-4efa-a301-33622285044e","Type":"ContainerStarted","Data":"4c4e697c9b2dbd53996395a0119ab797319619351d59614d3a8d426958344c82"}
Sep 08 02:20:19.627526 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:20:19.627472    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-fcdd46/sub-test-model-fcdd46-kserve-5b968649fc-krgrv" podStartSLOduration=1.627456582 podStartE2EDuration="1.627456582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:20:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:20:19.627012831 +0000 UTC m=+1291.150787911" watchObservedRunningTime="2026-09-08 02:20:19.627456582 +0000 UTC m=+1291.151231662"
Sep 08 02:20:28.326391 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:20:28.326353    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-527ac8/auth-test-model-527ac8-kserve-64d86fff7f-pp528" podUID="d433b308-951f-4ea2-be77-ab55d7a8b628" containerName="main" containerID="cri-o://d40574098d092e70addc791f4a566aa2a7b4414df62464e21861380c3d75cb77" gracePeriod=45
Sep 08 02:20:28.567164 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:20:28.567125    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-527ac8/auth-test-model-527ac8-kserve-64d86fff7f-pp528"
Sep 08 02:20:28.653216 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:20:28.653181    2547 generic.go:356] "Generic (PLEG): container finished" podID="d433b308-951f-4ea2-be77-ab55d7a8b628" containerID="d40574098d092e70addc791f4a566aa2a7b4414df62464e21861380c3d75cb77" exitCode=0
Sep 08 02:20:28.653391 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:20:28.653227    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-527ac8/auth-test-model-527ac8-kserve-64d86fff7f-pp528"
Sep 08 02:20:28.653391 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:20:28.653276    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-527ac8/auth-test-model-527ac8-kserve-64d86fff7f-pp528" event={"ID":"d433b308-951f-4ea2-be77-ab55d7a8b628","Type":"ContainerDied","Data":"d40574098d092e70addc791f4a566aa2a7b4414df62464e21861380c3d75cb77"}
Sep 08 02:20:28.653391 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:20:28.653319    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-527ac8/auth-test-model-527ac8-kserve-64d86fff7f-pp528" event={"ID":"d433b308-951f-4ea2-be77-ab55d7a8b628","Type":"ContainerDied","Data":"2a7cf0e05fc239bb4c2a7d5ea32763c44d47050c0eba1a83f24e3b524368d575"}
Sep 08 02:20:28.653391 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:20:28.653337    2547 scope.go:117] "RemoveContainer" containerID="d40574098d092e70addc791f4a566aa2a7b4414df62464e21861380c3d75cb77"
Sep 08 02:20:28.661608 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:20:28.661586    2547 scope.go:117] "RemoveContainer" containerID="d40574098d092e70addc791f4a566aa2a7b4414df62464e21861380c3d75cb77"
Sep 08 02:20:28.661905 ip-10-0-142-78 kubenswrapper[2547]: E0908 02:20:28.661886    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d40574098d092e70addc791f4a566aa2a7b4414df62464e21861380c3d75cb77\": container with ID starting with d40574098d092e70addc791f4a566aa2a7b4414df62464e21861380c3d75cb77 not found: ID does not exist" containerID="d40574098d092e70addc791f4a566aa2a7b4414df62464e21861380c3d75cb77"
Sep 08 02:20:28.661955 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:20:28.661916    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d40574098d092e70addc791f4a566aa2a7b4414df62464e21861380c3d75cb77"} err="failed to get container status \"d40574098d092e70addc791f4a566aa2a7b4414df62464e21861380c3d75cb77\": rpc error: code = NotFound desc = could not find container \"d40574098d092e70addc791f4a566aa2a7b4414df62464e21861380c3d75cb77\": container with ID starting with d40574098d092e70addc791f4a566aa2a7b4414df62464e21861380c3d75cb77 not found: ID does not exist"
Sep 08 02:20:28.686727 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:20:28.686687    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d433b308-951f-4ea2-be77-ab55d7a8b628-tls-certs\") pod \"d433b308-951f-4ea2-be77-ab55d7a8b628\" (UID: \"d433b308-951f-4ea2-be77-ab55d7a8b628\") "
Sep 08 02:20:28.686909 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:20:28.686767    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d433b308-951f-4ea2-be77-ab55d7a8b628-home\") pod \"d433b308-951f-4ea2-be77-ab55d7a8b628\" (UID: \"d433b308-951f-4ea2-be77-ab55d7a8b628\") "
Sep 08 02:20:28.686909 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:20:28.686839    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t66tj\" (UniqueName: \"kubernetes.io/projected/d433b308-951f-4ea2-be77-ab55d7a8b628-kube-api-access-t66tj\") pod \"d433b308-951f-4ea2-be77-ab55d7a8b628\" (UID: \"d433b308-951f-4ea2-be77-ab55d7a8b628\") "
Sep 08 02:20:28.686909 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:20:28.686883    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d433b308-951f-4ea2-be77-ab55d7a8b628-dshm\") pod \"d433b308-951f-4ea2-be77-ab55d7a8b628\" (UID: \"d433b308-951f-4ea2-be77-ab55d7a8b628\") "
Sep 08 02:20:28.687068 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:20:28.686909    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d433b308-951f-4ea2-be77-ab55d7a8b628-model-cache\") pod \"d433b308-951f-4ea2-be77-ab55d7a8b628\" (UID: \"d433b308-951f-4ea2-be77-ab55d7a8b628\") "
Sep 08 02:20:28.687068 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:20:28.686941    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d433b308-951f-4ea2-be77-ab55d7a8b628-tmp-dir\") pod \"d433b308-951f-4ea2-be77-ab55d7a8b628\" (UID: \"d433b308-951f-4ea2-be77-ab55d7a8b628\") "
Sep 08 02:20:28.687186 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:20:28.687082    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d433b308-951f-4ea2-be77-ab55d7a8b628-home" (OuterVolumeSpecName: "home") pod "d433b308-951f-4ea2-be77-ab55d7a8b628" (UID: "d433b308-951f-4ea2-be77-ab55d7a8b628"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 02:20:28.687246 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:20:28.687228    2547 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d433b308-951f-4ea2-be77-ab55d7a8b628-home\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:20:28.687300 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:20:28.687279    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d433b308-951f-4ea2-be77-ab55d7a8b628-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d433b308-951f-4ea2-be77-ab55d7a8b628" (UID: "d433b308-951f-4ea2-be77-ab55d7a8b628"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 02:20:28.687300 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:20:28.687275    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d433b308-951f-4ea2-be77-ab55d7a8b628-model-cache" (OuterVolumeSpecName: "model-cache") pod "d433b308-951f-4ea2-be77-ab55d7a8b628" (UID: "d433b308-951f-4ea2-be77-ab55d7a8b628"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 02:20:28.689115 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:20:28.689089    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d433b308-951f-4ea2-be77-ab55d7a8b628-dshm" (OuterVolumeSpecName: "dshm") pod "d433b308-951f-4ea2-be77-ab55d7a8b628" (UID: "d433b308-951f-4ea2-be77-ab55d7a8b628"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 02:20:28.689206 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:20:28.689167    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d433b308-951f-4ea2-be77-ab55d7a8b628-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d433b308-951f-4ea2-be77-ab55d7a8b628" (UID: "d433b308-951f-4ea2-be77-ab55d7a8b628"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 02:20:28.689267 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:20:28.689235    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d433b308-951f-4ea2-be77-ab55d7a8b628-kube-api-access-t66tj" (OuterVolumeSpecName: "kube-api-access-t66tj") pod "d433b308-951f-4ea2-be77-ab55d7a8b628" (UID: "d433b308-951f-4ea2-be77-ab55d7a8b628"). InnerVolumeSpecName "kube-api-access-t66tj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 02:20:28.787786 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:20:28.787743    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t66tj\" (UniqueName: \"kubernetes.io/projected/d433b308-951f-4ea2-be77-ab55d7a8b628-kube-api-access-t66tj\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:20:28.787786 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:20:28.787780    2547 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d433b308-951f-4ea2-be77-ab55d7a8b628-dshm\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:20:28.787786 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:20:28.787794    2547 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d433b308-951f-4ea2-be77-ab55d7a8b628-model-cache\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:20:28.788032 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:20:28.787803    2547 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d433b308-951f-4ea2-be77-ab55d7a8b628-tmp-dir\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:20:28.788032 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:20:28.787811    2547 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d433b308-951f-4ea2-be77-ab55d7a8b628-tls-certs\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:20:28.817753 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:20:28.817724    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-fcdd46/sub-test-model-fcdd46-kserve-5b968649fc-krgrv"
Sep 08 02:20:28.817855 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:20:28.817788    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-fcdd46/sub-test-model-fcdd46-kserve-5b968649fc-krgrv"
Sep 08 02:20:28.830432 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:20:28.830407    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-fcdd46/sub-test-model-fcdd46-kserve-5b968649fc-krgrv"
Sep 08 02:20:28.974268 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:20:28.974228    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-527ac8/auth-test-model-527ac8-kserve-64d86fff7f-pp528"]
Sep 08 02:20:28.975826 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:20:28.975800    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-527ac8/auth-test-model-527ac8-kserve-64d86fff7f-pp528"]
Sep 08 02:20:28.995490 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:20:28.995459    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d433b308-951f-4ea2-be77-ab55d7a8b628" path="/var/lib/kubelet/pods/d433b308-951f-4ea2-be77-ab55d7a8b628/volumes"
Sep 08 02:20:29.676375 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:20:29.676338    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-fcdd46/sub-test-model-fcdd46-kserve-5b968649fc-krgrv"
Sep 08 02:20:55.966938 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:20:55.966904    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c25e6e-69bdc9467-xcqws"]
Sep 08 02:20:55.967432 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:20:55.967291    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d433b308-951f-4ea2-be77-ab55d7a8b628" containerName="main"
Sep 08 02:20:55.967432 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:20:55.967303    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="d433b308-951f-4ea2-be77-ab55d7a8b628" containerName="main"
Sep 08 02:20:55.967432 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:20:55.967362    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="d433b308-951f-4ea2-be77-ab55d7a8b628" containerName="main"
Sep 08 02:20:55.971579 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:20:55.971553    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c25e6e-69bdc9467-xcqws"
Sep 08 02:20:55.973745 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:20:55.973725    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-c25e6e-dockercfg-xjn54\""
Sep 08 02:20:55.973855 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:20:55.973747    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-c25e6e\""
Sep 08 02:20:55.982641 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:20:55.982610    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c25e6e-69bdc9467-xcqws"]
Sep 08 02:20:56.022751 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:20:56.022707    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fe6fdf3-219d-4232-bb38-95072c2575da-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c25e6e-69bdc9467-xcqws\" (UID: \"9fe6fdf3-219d-4232-bb38-95072c2575da\") " pod="openshift-ingress/payload-processing-e2e-api-a-c25e6e-69bdc9467-xcqws"
Sep 08 02:20:56.022931 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:20:56.022817    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2v7h\" (UniqueName: \"kubernetes.io/projected/9fe6fdf3-219d-4232-bb38-95072c2575da-kube-api-access-r2v7h\") pod \"payload-processing-e2e-api-a-c25e6e-69bdc9467-xcqws\" (UID: \"9fe6fdf3-219d-4232-bb38-95072c2575da\") " pod="openshift-ingress/payload-processing-e2e-api-a-c25e6e-69bdc9467-xcqws"
Sep 08 02:20:56.123877 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:20:56.123838    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fe6fdf3-219d-4232-bb38-95072c2575da-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c25e6e-69bdc9467-xcqws\" (UID: \"9fe6fdf3-219d-4232-bb38-95072c2575da\") " pod="openshift-ingress/payload-processing-e2e-api-a-c25e6e-69bdc9467-xcqws"
Sep 08 02:20:56.124077 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:20:56.123981    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2v7h\" (UniqueName: \"kubernetes.io/projected/9fe6fdf3-219d-4232-bb38-95072c2575da-kube-api-access-r2v7h\") pod \"payload-processing-e2e-api-a-c25e6e-69bdc9467-xcqws\" (UID: \"9fe6fdf3-219d-4232-bb38-95072c2575da\") " pod="openshift-ingress/payload-processing-e2e-api-a-c25e6e-69bdc9467-xcqws"
Sep 08 02:20:56.124726 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:20:56.124704    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fe6fdf3-219d-4232-bb38-95072c2575da-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c25e6e-69bdc9467-xcqws\" (UID: \"9fe6fdf3-219d-4232-bb38-95072c2575da\") " pod="openshift-ingress/payload-processing-e2e-api-a-c25e6e-69bdc9467-xcqws"
Sep 08 02:20:56.132388 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:20:56.132357    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2v7h\" (UniqueName: \"kubernetes.io/projected/9fe6fdf3-219d-4232-bb38-95072c2575da-kube-api-access-r2v7h\") pod \"payload-processing-e2e-api-a-c25e6e-69bdc9467-xcqws\" (UID: \"9fe6fdf3-219d-4232-bb38-95072c2575da\") " pod="openshift-ingress/payload-processing-e2e-api-a-c25e6e-69bdc9467-xcqws"
Sep 08 02:20:56.283704 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:20:56.283613    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c25e6e-69bdc9467-xcqws"
Sep 08 02:20:56.554928 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:20:56.554902    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c25e6e-69bdc9467-xcqws"]
Sep 08 02:20:56.557464 ip-10-0-142-78 kubenswrapper[2547]: W0908 02:20:56.557432    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fe6fdf3_219d_4232_bb38_95072c2575da.slice/crio-c8dfaa6554ac7dfca2f4bf054dce11c5a2538df03c9c41d2315ade3ed5727745 WatchSource:0}: Error finding container c8dfaa6554ac7dfca2f4bf054dce11c5a2538df03c9c41d2315ade3ed5727745: Status 404 returned error can't find the container with id c8dfaa6554ac7dfca2f4bf054dce11c5a2538df03c9c41d2315ade3ed5727745
Sep 08 02:20:56.790775 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:20:56.790739    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c25e6e-69bdc9467-xcqws" event={"ID":"9fe6fdf3-219d-4232-bb38-95072c2575da","Type":"ContainerStarted","Data":"49bd3937d20e6ce2a7c22b02c66ddbc9b514e8d1d9ae38314efa2455078afc64"}
Sep 08 02:20:56.790969 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:20:56.790800    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c25e6e-69bdc9467-xcqws" event={"ID":"9fe6fdf3-219d-4232-bb38-95072c2575da","Type":"ContainerStarted","Data":"c8dfaa6554ac7dfca2f4bf054dce11c5a2538df03c9c41d2315ade3ed5727745"}
Sep 08 02:20:56.790969 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:20:56.790859    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-c25e6e-69bdc9467-xcqws"
Sep 08 02:20:56.806992 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:20:56.806895    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-c25e6e-69bdc9467-xcqws" podStartSLOduration=1.806880702 podStartE2EDuration="1.806880702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:20:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:20:56.805496103 +0000 UTC m=+1328.329271194" watchObservedRunningTime="2026-09-08 02:20:56.806880702 +0000 UTC m=+1328.330655781"
Sep 08 02:21:07.798254 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:21:07.798220    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-c25e6e-69bdc9467-xcqws"
Sep 08 02:21:39.503995 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:21:39.503958    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-fcdd46/sub-test-model-fcdd46-kserve-5b968649fc-krgrv"]
Sep 08 02:21:45.413742 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:21:45.413694    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-fcdd46/rate-test-model-fcdd46-kserve-6fcc98f97f-s5rbk"]
Sep 08 02:21:45.417396 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:21:45.417376    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-fcdd46/rate-test-model-fcdd46-kserve-6fcc98f97f-s5rbk"
Sep 08 02:21:45.419749 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:21:45.419720    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-fcdd46\"/\"rate-test-model-fcdd46-kserve-self-signed-certs\""
Sep 08 02:21:45.428556 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:21:45.428530    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-fcdd46/rate-test-model-fcdd46-kserve-6fcc98f97f-s5rbk"]
Sep 08 02:21:45.478144 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:21:45.478105    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gjf7\" (UniqueName: \"kubernetes.io/projected/98498211-e317-464f-8537-ab38bbe33904-kube-api-access-6gjf7\") pod \"rate-test-model-fcdd46-kserve-6fcc98f97f-s5rbk\" (UID: \"98498211-e317-464f-8537-ab38bbe33904\") " pod="ai-tenant-e2e-shared-a-w3-fcdd46/rate-test-model-fcdd46-kserve-6fcc98f97f-s5rbk"
Sep 08 02:21:45.478331 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:21:45.478197    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98498211-e317-464f-8537-ab38bbe33904-tmp-dir\") pod \"rate-test-model-fcdd46-kserve-6fcc98f97f-s5rbk\" (UID: \"98498211-e317-464f-8537-ab38bbe33904\") " pod="ai-tenant-e2e-shared-a-w3-fcdd46/rate-test-model-fcdd46-kserve-6fcc98f97f-s5rbk"
Sep 08 02:21:45.478331 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:21:45.478226    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98498211-e317-464f-8537-ab38bbe33904-home\") pod \"rate-test-model-fcdd46-kserve-6fcc98f97f-s5rbk\" (UID: \"98498211-e317-464f-8537-ab38bbe33904\") " pod="ai-tenant-e2e-shared-a-w3-fcdd46/rate-test-model-fcdd46-kserve-6fcc98f97f-s5rbk"
Sep 08 02:21:45.478331 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:21:45.478257    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98498211-e317-464f-8537-ab38bbe33904-model-cache\") pod \"rate-test-model-fcdd46-kserve-6fcc98f97f-s5rbk\" (UID: \"98498211-e317-464f-8537-ab38bbe33904\") " pod="ai-tenant-e2e-shared-a-w3-fcdd46/rate-test-model-fcdd46-kserve-6fcc98f97f-s5rbk"
Sep 08 02:21:45.478331 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:21:45.478302    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98498211-e317-464f-8537-ab38bbe33904-tls-certs\") pod \"rate-test-model-fcdd46-kserve-6fcc98f97f-s5rbk\" (UID: \"98498211-e317-464f-8537-ab38bbe33904\") " pod="ai-tenant-e2e-shared-a-w3-fcdd46/rate-test-model-fcdd46-kserve-6fcc98f97f-s5rbk"
Sep 08 02:21:45.478474 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:21:45.478372    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98498211-e317-464f-8537-ab38bbe33904-dshm\") pod \"rate-test-model-fcdd46-kserve-6fcc98f97f-s5rbk\" (UID: \"98498211-e317-464f-8537-ab38bbe33904\") " pod="ai-tenant-e2e-shared-a-w3-fcdd46/rate-test-model-fcdd46-kserve-6fcc98f97f-s5rbk"
Sep 08 02:21:45.579786 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:21:45.579745    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gjf7\" (UniqueName: \"kubernetes.io/projected/98498211-e317-464f-8537-ab38bbe33904-kube-api-access-6gjf7\") pod \"rate-test-model-fcdd46-kserve-6fcc98f97f-s5rbk\" (UID: \"98498211-e317-464f-8537-ab38bbe33904\") " pod="ai-tenant-e2e-shared-a-w3-fcdd46/rate-test-model-fcdd46-kserve-6fcc98f97f-s5rbk"
Sep 08 02:21:45.579958 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:21:45.579795    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98498211-e317-464f-8537-ab38bbe33904-tmp-dir\") pod \"rate-test-model-fcdd46-kserve-6fcc98f97f-s5rbk\" (UID: \"98498211-e317-464f-8537-ab38bbe33904\") " pod="ai-tenant-e2e-shared-a-w3-fcdd46/rate-test-model-fcdd46-kserve-6fcc98f97f-s5rbk"
Sep 08 02:21:45.579958 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:21:45.579836    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98498211-e317-464f-8537-ab38bbe33904-home\") pod \"rate-test-model-fcdd46-kserve-6fcc98f97f-s5rbk\" (UID: \"98498211-e317-464f-8537-ab38bbe33904\") " pod="ai-tenant-e2e-shared-a-w3-fcdd46/rate-test-model-fcdd46-kserve-6fcc98f97f-s5rbk"
Sep 08 02:21:45.579958 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:21:45.579864    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98498211-e317-464f-8537-ab38bbe33904-model-cache\") pod \"rate-test-model-fcdd46-kserve-6fcc98f97f-s5rbk\" (UID: \"98498211-e317-464f-8537-ab38bbe33904\") " pod="ai-tenant-e2e-shared-a-w3-fcdd46/rate-test-model-fcdd46-kserve-6fcc98f97f-s5rbk"
Sep 08 02:21:45.579958 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:21:45.579883    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98498211-e317-464f-8537-ab38bbe33904-tls-certs\") pod \"rate-test-model-fcdd46-kserve-6fcc98f97f-s5rbk\" (UID: \"98498211-e317-464f-8537-ab38bbe33904\") " pod="ai-tenant-e2e-shared-a-w3-fcdd46/rate-test-model-fcdd46-kserve-6fcc98f97f-s5rbk"
Sep 08 02:21:45.580233 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:21:45.580203    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98498211-e317-464f-8537-ab38bbe33904-dshm\") pod \"rate-test-model-fcdd46-kserve-6fcc98f97f-s5rbk\" (UID: \"98498211-e317-464f-8537-ab38bbe33904\") " pod="ai-tenant-e2e-shared-a-w3-fcdd46/rate-test-model-fcdd46-kserve-6fcc98f97f-s5rbk"
Sep 08 02:21:45.580319 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:21:45.580295    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98498211-e317-464f-8537-ab38bbe33904-home\") pod \"rate-test-model-fcdd46-kserve-6fcc98f97f-s5rbk\" (UID: \"98498211-e317-464f-8537-ab38bbe33904\") " pod="ai-tenant-e2e-shared-a-w3-fcdd46/rate-test-model-fcdd46-kserve-6fcc98f97f-s5rbk"
Sep 08 02:21:45.580365 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:21:45.580335    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98498211-e317-464f-8537-ab38bbe33904-model-cache\") pod \"rate-test-model-fcdd46-kserve-6fcc98f97f-s5rbk\" (UID: \"98498211-e317-464f-8537-ab38bbe33904\") " pod="ai-tenant-e2e-shared-a-w3-fcdd46/rate-test-model-fcdd46-kserve-6fcc98f97f-s5rbk"
Sep 08 02:21:45.580409 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:21:45.580361    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98498211-e317-464f-8537-ab38bbe33904-tmp-dir\") pod \"rate-test-model-fcdd46-kserve-6fcc98f97f-s5rbk\" (UID: \"98498211-e317-464f-8537-ab38bbe33904\") " pod="ai-tenant-e2e-shared-a-w3-fcdd46/rate-test-model-fcdd46-kserve-6fcc98f97f-s5rbk"
Sep 08 02:21:45.582392 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:21:45.582365    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98498211-e317-464f-8537-ab38bbe33904-dshm\") pod \"rate-test-model-fcdd46-kserve-6fcc98f97f-s5rbk\" (UID: \"98498211-e317-464f-8537-ab38bbe33904\") " pod="ai-tenant-e2e-shared-a-w3-fcdd46/rate-test-model-fcdd46-kserve-6fcc98f97f-s5rbk"
Sep 08 02:21:45.582704 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:21:45.582681    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98498211-e317-464f-8537-ab38bbe33904-tls-certs\") pod \"rate-test-model-fcdd46-kserve-6fcc98f97f-s5rbk\" (UID: \"98498211-e317-464f-8537-ab38bbe33904\") " pod="ai-tenant-e2e-shared-a-w3-fcdd46/rate-test-model-fcdd46-kserve-6fcc98f97f-s5rbk"
Sep 08 02:21:45.587635 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:21:45.587613    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gjf7\" (UniqueName: \"kubernetes.io/projected/98498211-e317-464f-8537-ab38bbe33904-kube-api-access-6gjf7\") pod \"rate-test-model-fcdd46-kserve-6fcc98f97f-s5rbk\" (UID: \"98498211-e317-464f-8537-ab38bbe33904\") " pod="ai-tenant-e2e-shared-a-w3-fcdd46/rate-test-model-fcdd46-kserve-6fcc98f97f-s5rbk"
Sep 08 02:21:45.727578 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:21:45.727475    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-fcdd46/rate-test-model-fcdd46-kserve-6fcc98f97f-s5rbk"
Sep 08 02:21:46.001053 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:21:46.001023    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-fcdd46/rate-test-model-fcdd46-kserve-6fcc98f97f-s5rbk"]
Sep 08 02:21:46.003620 ip-10-0-142-78 kubenswrapper[2547]: W0908 02:21:46.003592    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod98498211_e317_464f_8537_ab38bbe33904.slice/crio-95027d94937bc8def6b64f9ec4cb34f44db01c5217e500d73462fa670653ff49 WatchSource:0}: Error finding container 95027d94937bc8def6b64f9ec4cb34f44db01c5217e500d73462fa670653ff49: Status 404 returned error can't find the container with id 95027d94937bc8def6b64f9ec4cb34f44db01c5217e500d73462fa670653ff49
Sep 08 02:21:46.025006 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:21:46.024979    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-fcdd46/rate-test-model-fcdd46-kserve-6fcc98f97f-s5rbk" event={"ID":"98498211-e317-464f-8537-ab38bbe33904","Type":"ContainerStarted","Data":"95027d94937bc8def6b64f9ec4cb34f44db01c5217e500d73462fa670653ff49"}
Sep 08 02:21:47.030986 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:21:47.030952    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-fcdd46/rate-test-model-fcdd46-kserve-6fcc98f97f-s5rbk" event={"ID":"98498211-e317-464f-8537-ab38bbe33904","Type":"ContainerStarted","Data":"967c42e8dac2d1ebe9f1a286bc4ef8a0b378c7afbb6f2ffd5ee2753051487cbd"}
Sep 08 02:21:47.047933 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:21:47.047882    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-fcdd46/rate-test-model-fcdd46-kserve-6fcc98f97f-s5rbk" podStartSLOduration=2.047864878 podStartE2EDuration="2.047864878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:21:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:21:47.047225423 +0000 UTC m=+1378.571000519" watchObservedRunningTime="2026-09-08 02:21:47.047864878 +0000 UTC m=+1378.571639957"
Sep 08 02:21:49.077331 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:21:49.077299    2547 scope.go:117] "RemoveContainer" containerID="0a1610cde153aef0a52065673aef5fd18dd76ba5bb7847b46ff4d8e3d809bd01"
Sep 08 02:21:54.525464 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:21:54.525415    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-fcdd46/sub-test-model-fcdd46-kserve-5b968649fc-krgrv" podUID="9f31a2d6-de1c-4efa-a301-33622285044e" containerName="main" containerID="cri-o://6c3285bdb97f77213d5286b68b3d52ede5e5a17c06c8ace2f9b0b9d994e442e3" gracePeriod=45
Sep 08 02:21:54.780239 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:21:54.780176    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-fcdd46/sub-test-model-fcdd46-kserve-5b968649fc-krgrv"
Sep 08 02:21:54.869968 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:21:54.869923    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f31a2d6-de1c-4efa-a301-33622285044e-dshm\") pod \"9f31a2d6-de1c-4efa-a301-33622285044e\" (UID: \"9f31a2d6-de1c-4efa-a301-33622285044e\") "
Sep 08 02:21:54.870195 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:21:54.870023    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jvrs5\" (UniqueName: \"kubernetes.io/projected/9f31a2d6-de1c-4efa-a301-33622285044e-kube-api-access-jvrs5\") pod \"9f31a2d6-de1c-4efa-a301-33622285044e\" (UID: \"9f31a2d6-de1c-4efa-a301-33622285044e\") "
Sep 08 02:21:54.870195 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:21:54.870054    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f31a2d6-de1c-4efa-a301-33622285044e-tls-certs\") pod \"9f31a2d6-de1c-4efa-a301-33622285044e\" (UID: \"9f31a2d6-de1c-4efa-a301-33622285044e\") "
Sep 08 02:21:54.870195 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:21:54.870097    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f31a2d6-de1c-4efa-a301-33622285044e-tmp-dir\") pod \"9f31a2d6-de1c-4efa-a301-33622285044e\" (UID: \"9f31a2d6-de1c-4efa-a301-33622285044e\") "
Sep 08 02:21:54.870378 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:21:54.870247    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f31a2d6-de1c-4efa-a301-33622285044e-home\") pod \"9f31a2d6-de1c-4efa-a301-33622285044e\" (UID: \"9f31a2d6-de1c-4efa-a301-33622285044e\") "
Sep 08 02:21:54.870378 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:21:54.870304    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f31a2d6-de1c-4efa-a301-33622285044e-model-cache\") pod \"9f31a2d6-de1c-4efa-a301-33622285044e\" (UID: \"9f31a2d6-de1c-4efa-a301-33622285044e\") "
Sep 08 02:21:54.870482 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:21:54.870406    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f31a2d6-de1c-4efa-a301-33622285044e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9f31a2d6-de1c-4efa-a301-33622285044e" (UID: "9f31a2d6-de1c-4efa-a301-33622285044e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 02:21:54.870554 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:21:54.870514    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f31a2d6-de1c-4efa-a301-33622285044e-home" (OuterVolumeSpecName: "home") pod "9f31a2d6-de1c-4efa-a301-33622285044e" (UID: "9f31a2d6-de1c-4efa-a301-33622285044e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 02:21:54.870609 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:21:54.870546    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f31a2d6-de1c-4efa-a301-33622285044e-model-cache" (OuterVolumeSpecName: "model-cache") pod "9f31a2d6-de1c-4efa-a301-33622285044e" (UID: "9f31a2d6-de1c-4efa-a301-33622285044e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 02:21:54.870677 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:21:54.870650    2547 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f31a2d6-de1c-4efa-a301-33622285044e-tmp-dir\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:21:54.870733 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:21:54.870677    2547 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f31a2d6-de1c-4efa-a301-33622285044e-home\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:21:54.870733 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:21:54.870686    2547 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f31a2d6-de1c-4efa-a301-33622285044e-model-cache\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:21:54.872395 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:21:54.872371    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f31a2d6-de1c-4efa-a301-33622285044e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9f31a2d6-de1c-4efa-a301-33622285044e" (UID: "9f31a2d6-de1c-4efa-a301-33622285044e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 02:21:54.872395 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:21:54.872375    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f31a2d6-de1c-4efa-a301-33622285044e-dshm" (OuterVolumeSpecName: "dshm") pod "9f31a2d6-de1c-4efa-a301-33622285044e" (UID: "9f31a2d6-de1c-4efa-a301-33622285044e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 02:21:54.872526 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:21:54.872388    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f31a2d6-de1c-4efa-a301-33622285044e-kube-api-access-jvrs5" (OuterVolumeSpecName: "kube-api-access-jvrs5") pod "9f31a2d6-de1c-4efa-a301-33622285044e" (UID: "9f31a2d6-de1c-4efa-a301-33622285044e"). InnerVolumeSpecName "kube-api-access-jvrs5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 02:21:54.971328 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:21:54.971268    2547 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f31a2d6-de1c-4efa-a301-33622285044e-dshm\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:21:54.971328 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:21:54.971319    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvrs5\" (UniqueName: \"kubernetes.io/projected/9f31a2d6-de1c-4efa-a301-33622285044e-kube-api-access-jvrs5\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:21:54.971328 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:21:54.971330    2547 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f31a2d6-de1c-4efa-a301-33622285044e-tls-certs\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:21:55.069363 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:21:55.069270    2547 generic.go:356] "Generic (PLEG): container finished" podID="9f31a2d6-de1c-4efa-a301-33622285044e" containerID="6c3285bdb97f77213d5286b68b3d52ede5e5a17c06c8ace2f9b0b9d994e442e3" exitCode=0
Sep 08 02:21:55.069363 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:21:55.069324    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-fcdd46/sub-test-model-fcdd46-kserve-5b968649fc-krgrv"
Sep 08 02:21:55.069363 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:21:55.069343    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-fcdd46/sub-test-model-fcdd46-kserve-5b968649fc-krgrv" event={"ID":"9f31a2d6-de1c-4efa-a301-33622285044e","Type":"ContainerDied","Data":"6c3285bdb97f77213d5286b68b3d52ede5e5a17c06c8ace2f9b0b9d994e442e3"}
Sep 08 02:21:55.069632 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:21:55.069393    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-fcdd46/sub-test-model-fcdd46-kserve-5b968649fc-krgrv" event={"ID":"9f31a2d6-de1c-4efa-a301-33622285044e","Type":"ContainerDied","Data":"4c4e697c9b2dbd53996395a0119ab797319619351d59614d3a8d426958344c82"}
Sep 08 02:21:55.069632 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:21:55.069411    2547 scope.go:117] "RemoveContainer" containerID="6c3285bdb97f77213d5286b68b3d52ede5e5a17c06c8ace2f9b0b9d994e442e3"
Sep 08 02:21:55.077792 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:21:55.077771    2547 scope.go:117] "RemoveContainer" containerID="6c3285bdb97f77213d5286b68b3d52ede5e5a17c06c8ace2f9b0b9d994e442e3"
Sep 08 02:21:55.078111 ip-10-0-142-78 kubenswrapper[2547]: E0908 02:21:55.078092    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c3285bdb97f77213d5286b68b3d52ede5e5a17c06c8ace2f9b0b9d994e442e3\": container with ID starting with 6c3285bdb97f77213d5286b68b3d52ede5e5a17c06c8ace2f9b0b9d994e442e3 not found: ID does not exist" containerID="6c3285bdb97f77213d5286b68b3d52ede5e5a17c06c8ace2f9b0b9d994e442e3"
Sep 08 02:21:55.078186 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:21:55.078121    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c3285bdb97f77213d5286b68b3d52ede5e5a17c06c8ace2f9b0b9d994e442e3"} err="failed to get container status \"6c3285bdb97f77213d5286b68b3d52ede5e5a17c06c8ace2f9b0b9d994e442e3\": rpc error: code = NotFound desc = could not find container \"6c3285bdb97f77213d5286b68b3d52ede5e5a17c06c8ace2f9b0b9d994e442e3\": container with ID starting with 6c3285bdb97f77213d5286b68b3d52ede5e5a17c06c8ace2f9b0b9d994e442e3 not found: ID does not exist"
Sep 08 02:21:55.084905 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:21:55.084879    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-fcdd46/sub-test-model-fcdd46-kserve-5b968649fc-krgrv"]
Sep 08 02:21:55.086492 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:21:55.086470    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-fcdd46/sub-test-model-fcdd46-kserve-5b968649fc-krgrv"]
Sep 08 02:21:55.728566 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:21:55.728525    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-fcdd46/rate-test-model-fcdd46-kserve-6fcc98f97f-s5rbk"
Sep 08 02:21:55.729063 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:21:55.728580    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-fcdd46/rate-test-model-fcdd46-kserve-6fcc98f97f-s5rbk"
Sep 08 02:21:55.742035 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:21:55.742001    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-fcdd46/rate-test-model-fcdd46-kserve-6fcc98f97f-s5rbk"
Sep 08 02:21:56.086682 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:21:56.086592    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-fcdd46/rate-test-model-fcdd46-kserve-6fcc98f97f-s5rbk"
Sep 08 02:21:56.999803 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:21:56.996598    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f31a2d6-de1c-4efa-a301-33622285044e" path="/var/lib/kubelet/pods/9f31a2d6-de1c-4efa-a301-33622285044e/volumes"
Sep 08 02:22:06.487948 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:22:06.487863    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c25e6e-69bdc9467-xcqws"]
Sep 08 02:22:06.488773 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:22:06.488735    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-c25e6e-69bdc9467-xcqws" podUID="9fe6fdf3-219d-4232-bb38-95072c2575da" containerName="payload-processing" containerID="cri-o://49bd3937d20e6ce2a7c22b02c66ddbc9b514e8d1d9ae38314efa2455078afc64" gracePeriod=30
Sep 08 02:22:06.744891 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:22:06.744823    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c25e6e-69bdc9467-xcqws"
Sep 08 02:22:06.881860 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:22:06.881823    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r2v7h\" (UniqueName: \"kubernetes.io/projected/9fe6fdf3-219d-4232-bb38-95072c2575da-kube-api-access-r2v7h\") pod \"9fe6fdf3-219d-4232-bb38-95072c2575da\" (UID: \"9fe6fdf3-219d-4232-bb38-95072c2575da\") "
Sep 08 02:22:06.882035 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:22:06.881910    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fe6fdf3-219d-4232-bb38-95072c2575da-plugins-config-volume\") pod \"9fe6fdf3-219d-4232-bb38-95072c2575da\" (UID: \"9fe6fdf3-219d-4232-bb38-95072c2575da\") "
Sep 08 02:22:06.882324 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:22:06.882297    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fe6fdf3-219d-4232-bb38-95072c2575da-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9fe6fdf3-219d-4232-bb38-95072c2575da" (UID: "9fe6fdf3-219d-4232-bb38-95072c2575da"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 02:22:06.883944 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:22:06.883919    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fe6fdf3-219d-4232-bb38-95072c2575da-kube-api-access-r2v7h" (OuterVolumeSpecName: "kube-api-access-r2v7h") pod "9fe6fdf3-219d-4232-bb38-95072c2575da" (UID: "9fe6fdf3-219d-4232-bb38-95072c2575da"). InnerVolumeSpecName "kube-api-access-r2v7h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 02:22:06.982553 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:22:06.982511    2547 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fe6fdf3-219d-4232-bb38-95072c2575da-plugins-config-volume\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:22:06.982553 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:22:06.982548    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2v7h\" (UniqueName: \"kubernetes.io/projected/9fe6fdf3-219d-4232-bb38-95072c2575da-kube-api-access-r2v7h\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:22:07.127319 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:22:07.127282    2547 generic.go:356] "Generic (PLEG): container finished" podID="9fe6fdf3-219d-4232-bb38-95072c2575da" containerID="49bd3937d20e6ce2a7c22b02c66ddbc9b514e8d1d9ae38314efa2455078afc64" exitCode=0
Sep 08 02:22:07.127464 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:22:07.127331    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c25e6e-69bdc9467-xcqws"
Sep 08 02:22:07.127464 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:22:07.127364    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c25e6e-69bdc9467-xcqws" event={"ID":"9fe6fdf3-219d-4232-bb38-95072c2575da","Type":"ContainerDied","Data":"49bd3937d20e6ce2a7c22b02c66ddbc9b514e8d1d9ae38314efa2455078afc64"}
Sep 08 02:22:07.127464 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:22:07.127419    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c25e6e-69bdc9467-xcqws" event={"ID":"9fe6fdf3-219d-4232-bb38-95072c2575da","Type":"ContainerDied","Data":"c8dfaa6554ac7dfca2f4bf054dce11c5a2538df03c9c41d2315ade3ed5727745"}
Sep 08 02:22:07.127464 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:22:07.127442    2547 scope.go:117] "RemoveContainer" containerID="49bd3937d20e6ce2a7c22b02c66ddbc9b514e8d1d9ae38314efa2455078afc64"
Sep 08 02:22:07.136050 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:22:07.136028    2547 scope.go:117] "RemoveContainer" containerID="49bd3937d20e6ce2a7c22b02c66ddbc9b514e8d1d9ae38314efa2455078afc64"
Sep 08 02:22:07.136405 ip-10-0-142-78 kubenswrapper[2547]: E0908 02:22:07.136384    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49bd3937d20e6ce2a7c22b02c66ddbc9b514e8d1d9ae38314efa2455078afc64\": container with ID starting with 49bd3937d20e6ce2a7c22b02c66ddbc9b514e8d1d9ae38314efa2455078afc64 not found: ID does not exist" containerID="49bd3937d20e6ce2a7c22b02c66ddbc9b514e8d1d9ae38314efa2455078afc64"
Sep 08 02:22:07.136485 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:22:07.136415    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49bd3937d20e6ce2a7c22b02c66ddbc9b514e8d1d9ae38314efa2455078afc64"} err="failed to get container status \"49bd3937d20e6ce2a7c22b02c66ddbc9b514e8d1d9ae38314efa2455078afc64\": rpc error: code = NotFound desc = could not find container \"49bd3937d20e6ce2a7c22b02c66ddbc9b514e8d1d9ae38314efa2455078afc64\": container with ID starting with 49bd3937d20e6ce2a7c22b02c66ddbc9b514e8d1d9ae38314efa2455078afc64 not found: ID does not exist"
Sep 08 02:22:07.142340 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:22:07.142304    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c25e6e-69bdc9467-xcqws"]
Sep 08 02:22:07.145623 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:22:07.145594    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c25e6e-69bdc9467-xcqws"]
Sep 08 02:22:08.994685 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:22:08.994641    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9fe6fdf3-219d-4232-bb38-95072c2575da" path="/var/lib/kubelet/pods/9fe6fdf3-219d-4232-bb38-95072c2575da/volumes"
Sep 08 02:22:11.118116 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:22:11.118076    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-527ac8/rate-test-model-527ac8-kserve-65c877868c-bxfb5"]
Sep 08 02:22:11.118674 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:22:11.118639    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f31a2d6-de1c-4efa-a301-33622285044e" containerName="main"
Sep 08 02:22:11.118750 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:22:11.118678    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f31a2d6-de1c-4efa-a301-33622285044e" containerName="main"
Sep 08 02:22:11.118750 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:22:11.118692    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fe6fdf3-219d-4232-bb38-95072c2575da" containerName="payload-processing"
Sep 08 02:22:11.118750 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:22:11.118704    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fe6fdf3-219d-4232-bb38-95072c2575da" containerName="payload-processing"
Sep 08 02:22:11.118901 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:22:11.118797    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f31a2d6-de1c-4efa-a301-33622285044e" containerName="main"
Sep 08 02:22:11.118901 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:22:11.118811    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fe6fdf3-219d-4232-bb38-95072c2575da" containerName="payload-processing"
Sep 08 02:22:11.123293 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:22:11.123270    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-527ac8/rate-test-model-527ac8-kserve-65c877868c-bxfb5"
Sep 08 02:22:11.125639 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:22:11.125612    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-527ac8\"/\"rate-test-model-527ac8-kserve-self-signed-certs\""
Sep 08 02:22:11.125774 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:22:11.125617    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-527ac8\"/\"openshift-service-ca.crt\""
Sep 08 02:22:11.126482 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:22:11.126459    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-527ac8\"/\"default-dockercfg-zjlj4\""
Sep 08 02:22:11.126596 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:22:11.126489    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-527ac8\"/\"kube-root-ca.crt\""
Sep 08 02:22:11.131026 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:22:11.131003    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-527ac8/rate-test-model-527ac8-kserve-65c877868c-bxfb5"]
Sep 08 02:22:11.219084 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:22:11.219048    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db197e84-ccdb-4599-82c2-e1a6f6d5adc9-tls-certs\") pod \"rate-test-model-527ac8-kserve-65c877868c-bxfb5\" (UID: \"db197e84-ccdb-4599-82c2-e1a6f6d5adc9\") " pod="ai-tenant-e2e-shared-b-w3-527ac8/rate-test-model-527ac8-kserve-65c877868c-bxfb5"
Sep 08 02:22:11.219281 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:22:11.219090    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db197e84-ccdb-4599-82c2-e1a6f6d5adc9-dshm\") pod \"rate-test-model-527ac8-kserve-65c877868c-bxfb5\" (UID: \"db197e84-ccdb-4599-82c2-e1a6f6d5adc9\") " pod="ai-tenant-e2e-shared-b-w3-527ac8/rate-test-model-527ac8-kserve-65c877868c-bxfb5"
Sep 08 02:22:11.219281 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:22:11.219124    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db197e84-ccdb-4599-82c2-e1a6f6d5adc9-model-cache\") pod \"rate-test-model-527ac8-kserve-65c877868c-bxfb5\" (UID: \"db197e84-ccdb-4599-82c2-e1a6f6d5adc9\") " pod="ai-tenant-e2e-shared-b-w3-527ac8/rate-test-model-527ac8-kserve-65c877868c-bxfb5"
Sep 08 02:22:11.219281 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:22:11.219164    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db197e84-ccdb-4599-82c2-e1a6f6d5adc9-tmp-dir\") pod \"rate-test-model-527ac8-kserve-65c877868c-bxfb5\" (UID: \"db197e84-ccdb-4599-82c2-e1a6f6d5adc9\") " pod="ai-tenant-e2e-shared-b-w3-527ac8/rate-test-model-527ac8-kserve-65c877868c-bxfb5"
Sep 08 02:22:11.219281 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:22:11.219253    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db197e84-ccdb-4599-82c2-e1a6f6d5adc9-home\") pod \"rate-test-model-527ac8-kserve-65c877868c-bxfb5\" (UID: \"db197e84-ccdb-4599-82c2-e1a6f6d5adc9\") " pod="ai-tenant-e2e-shared-b-w3-527ac8/rate-test-model-527ac8-kserve-65c877868c-bxfb5"
Sep 08 02:22:11.219448 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:22:11.219311    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khpgk\" (UniqueName: \"kubernetes.io/projected/db197e84-ccdb-4599-82c2-e1a6f6d5adc9-kube-api-access-khpgk\") pod \"rate-test-model-527ac8-kserve-65c877868c-bxfb5\" (UID: \"db197e84-ccdb-4599-82c2-e1a6f6d5adc9\") " pod="ai-tenant-e2e-shared-b-w3-527ac8/rate-test-model-527ac8-kserve-65c877868c-bxfb5"
Sep 08 02:22:11.320357 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:22:11.320312    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khpgk\" (UniqueName: \"kubernetes.io/projected/db197e84-ccdb-4599-82c2-e1a6f6d5adc9-kube-api-access-khpgk\") pod \"rate-test-model-527ac8-kserve-65c877868c-bxfb5\" (UID: \"db197e84-ccdb-4599-82c2-e1a6f6d5adc9\") " pod="ai-tenant-e2e-shared-b-w3-527ac8/rate-test-model-527ac8-kserve-65c877868c-bxfb5"
Sep 08 02:22:11.320357 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:22:11.320358    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db197e84-ccdb-4599-82c2-e1a6f6d5adc9-tls-certs\") pod \"rate-test-model-527ac8-kserve-65c877868c-bxfb5\" (UID: \"db197e84-ccdb-4599-82c2-e1a6f6d5adc9\") " pod="ai-tenant-e2e-shared-b-w3-527ac8/rate-test-model-527ac8-kserve-65c877868c-bxfb5"
Sep 08 02:22:11.320579 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:22:11.320381    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db197e84-ccdb-4599-82c2-e1a6f6d5adc9-dshm\") pod \"rate-test-model-527ac8-kserve-65c877868c-bxfb5\" (UID: \"db197e84-ccdb-4599-82c2-e1a6f6d5adc9\") " pod="ai-tenant-e2e-shared-b-w3-527ac8/rate-test-model-527ac8-kserve-65c877868c-bxfb5"
Sep 08 02:22:11.320579 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:22:11.320548    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db197e84-ccdb-4599-82c2-e1a6f6d5adc9-model-cache\") pod \"rate-test-model-527ac8-kserve-65c877868c-bxfb5\" (UID: \"db197e84-ccdb-4599-82c2-e1a6f6d5adc9\") " pod="ai-tenant-e2e-shared-b-w3-527ac8/rate-test-model-527ac8-kserve-65c877868c-bxfb5"
Sep 08 02:22:11.320687 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:22:11.320578    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db197e84-ccdb-4599-82c2-e1a6f6d5adc9-tmp-dir\") pod \"rate-test-model-527ac8-kserve-65c877868c-bxfb5\" (UID: \"db197e84-ccdb-4599-82c2-e1a6f6d5adc9\") " pod="ai-tenant-e2e-shared-b-w3-527ac8/rate-test-model-527ac8-kserve-65c877868c-bxfb5"
Sep 08 02:22:11.320687 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:22:11.320647    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db197e84-ccdb-4599-82c2-e1a6f6d5adc9-home\") pod \"rate-test-model-527ac8-kserve-65c877868c-bxfb5\" (UID: \"db197e84-ccdb-4599-82c2-e1a6f6d5adc9\") " pod="ai-tenant-e2e-shared-b-w3-527ac8/rate-test-model-527ac8-kserve-65c877868c-bxfb5"
Sep 08 02:22:11.320987 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:22:11.320962    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db197e84-ccdb-4599-82c2-e1a6f6d5adc9-model-cache\") pod \"rate-test-model-527ac8-kserve-65c877868c-bxfb5\" (UID: \"db197e84-ccdb-4599-82c2-e1a6f6d5adc9\") " pod="ai-tenant-e2e-shared-b-w3-527ac8/rate-test-model-527ac8-kserve-65c877868c-bxfb5"
Sep 08 02:22:11.321116 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:22:11.320991    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db197e84-ccdb-4599-82c2-e1a6f6d5adc9-tmp-dir\") pod \"rate-test-model-527ac8-kserve-65c877868c-bxfb5\" (UID: \"db197e84-ccdb-4599-82c2-e1a6f6d5adc9\") " pod="ai-tenant-e2e-shared-b-w3-527ac8/rate-test-model-527ac8-kserve-65c877868c-bxfb5"
Sep 08 02:22:11.321116 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:22:11.321054    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db197e84-ccdb-4599-82c2-e1a6f6d5adc9-home\") pod \"rate-test-model-527ac8-kserve-65c877868c-bxfb5\" (UID: \"db197e84-ccdb-4599-82c2-e1a6f6d5adc9\") " pod="ai-tenant-e2e-shared-b-w3-527ac8/rate-test-model-527ac8-kserve-65c877868c-bxfb5"
Sep 08 02:22:11.322755 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:22:11.322726    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db197e84-ccdb-4599-82c2-e1a6f6d5adc9-dshm\") pod \"rate-test-model-527ac8-kserve-65c877868c-bxfb5\" (UID: \"db197e84-ccdb-4599-82c2-e1a6f6d5adc9\") " pod="ai-tenant-e2e-shared-b-w3-527ac8/rate-test-model-527ac8-kserve-65c877868c-bxfb5"
Sep 08 02:22:11.323020 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:22:11.322999    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db197e84-ccdb-4599-82c2-e1a6f6d5adc9-tls-certs\") pod \"rate-test-model-527ac8-kserve-65c877868c-bxfb5\" (UID: \"db197e84-ccdb-4599-82c2-e1a6f6d5adc9\") " pod="ai-tenant-e2e-shared-b-w3-527ac8/rate-test-model-527ac8-kserve-65c877868c-bxfb5"
Sep 08 02:22:11.328274 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:22:11.328253    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khpgk\" (UniqueName: \"kubernetes.io/projected/db197e84-ccdb-4599-82c2-e1a6f6d5adc9-kube-api-access-khpgk\") pod \"rate-test-model-527ac8-kserve-65c877868c-bxfb5\" (UID: \"db197e84-ccdb-4599-82c2-e1a6f6d5adc9\") " pod="ai-tenant-e2e-shared-b-w3-527ac8/rate-test-model-527ac8-kserve-65c877868c-bxfb5"
Sep 08 02:22:11.433332 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:22:11.433297    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-527ac8/rate-test-model-527ac8-kserve-65c877868c-bxfb5"
Sep 08 02:22:11.705115 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:22:11.705088    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-527ac8/rate-test-model-527ac8-kserve-65c877868c-bxfb5"]
Sep 08 02:22:11.707687 ip-10-0-142-78 kubenswrapper[2547]: W0908 02:22:11.707643    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddb197e84_ccdb_4599_82c2_e1a6f6d5adc9.slice/crio-19cfba137d9e9424ba9082d6df1cad6bc5e1fed9025c720cec5f8311d5bd1f52 WatchSource:0}: Error finding container 19cfba137d9e9424ba9082d6df1cad6bc5e1fed9025c720cec5f8311d5bd1f52: Status 404 returned error can't find the container with id 19cfba137d9e9424ba9082d6df1cad6bc5e1fed9025c720cec5f8311d5bd1f52
Sep 08 02:22:12.152828 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:22:12.152791    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-527ac8/rate-test-model-527ac8-kserve-65c877868c-bxfb5" event={"ID":"db197e84-ccdb-4599-82c2-e1a6f6d5adc9","Type":"ContainerStarted","Data":"cfdcef31bffa6edd0b21163ce9c2732448aa53df3f123fd1009dc6c5765d66b8"}
Sep 08 02:22:12.153253 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:22:12.152839    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-527ac8/rate-test-model-527ac8-kserve-65c877868c-bxfb5" event={"ID":"db197e84-ccdb-4599-82c2-e1a6f6d5adc9","Type":"ContainerStarted","Data":"19cfba137d9e9424ba9082d6df1cad6bc5e1fed9025c720cec5f8311d5bd1f52"}
Sep 08 02:22:12.171057 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:22:12.171000    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-527ac8/rate-test-model-527ac8-kserve-65c877868c-bxfb5" podStartSLOduration=1.170982769 podStartE2EDuration="1.170982769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:22:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:22:12.169290385 +0000 UTC m=+1403.693065465" watchObservedRunningTime="2026-09-08 02:22:12.170982769 +0000 UTC m=+1403.694757882"
Sep 08 02:22:21.433593 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:22:21.433553    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-527ac8/rate-test-model-527ac8-kserve-65c877868c-bxfb5"
Sep 08 02:22:21.433593 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:22:21.433604    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-527ac8/rate-test-model-527ac8-kserve-65c877868c-bxfb5"
Sep 08 02:22:21.446191 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:22:21.446159    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-527ac8/rate-test-model-527ac8-kserve-65c877868c-bxfb5"
Sep 08 02:22:22.214548 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:22:22.214514    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-527ac8/rate-test-model-527ac8-kserve-65c877868c-bxfb5"
Sep 08 02:22:46.267024 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:22:46.266938    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-670bc3-76644ff8c-jqf99"]
Sep 08 02:22:46.267575 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:22:46.267243    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-670bc3-76644ff8c-jqf99" podUID="60468f93-bb6e-470c-8c1a-cd6212d58c49" containerName="payload-processing" containerID="cri-o://9329c7ecafdfc2cb42b3936d2250a0190c166bc7f1f150c821cfb9be4d248f94" gracePeriod=30
Sep 08 02:22:46.275036 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:22:46.275001    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-670bc3-6fc959454b-jqpgb"]
Sep 08 02:22:46.275309 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:22:46.275282    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-670bc3-6fc959454b-jqpgb" podUID="f8dc52e3-ab0b-44cd-810f-bac422df9d46" containerName="payload-pre-processing" containerID="cri-o://f4e27b68961eb4f81c3aefcd2e79a0c34f1c95989e0cfc49f869005db446eafc" gracePeriod=30
Sep 08 02:22:46.552837 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:22:46.552809    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-670bc3-76644ff8c-jqf99"
Sep 08 02:22:46.555530 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:22:46.555508    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-670bc3-6fc959454b-jqpgb"
Sep 08 02:22:46.637478 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:22:46.637437    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rzt9h\" (UniqueName: \"kubernetes.io/projected/f8dc52e3-ab0b-44cd-810f-bac422df9d46-kube-api-access-rzt9h\") pod \"f8dc52e3-ab0b-44cd-810f-bac422df9d46\" (UID: \"f8dc52e3-ab0b-44cd-810f-bac422df9d46\") "
Sep 08 02:22:46.637654 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:22:46.637487    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8dc52e3-ab0b-44cd-810f-bac422df9d46-plugins-config-volume\") pod \"f8dc52e3-ab0b-44cd-810f-bac422df9d46\" (UID: \"f8dc52e3-ab0b-44cd-810f-bac422df9d46\") "
Sep 08 02:22:46.637654 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:22:46.637519    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jwbmj\" (UniqueName: \"kubernetes.io/projected/60468f93-bb6e-470c-8c1a-cd6212d58c49-kube-api-access-jwbmj\") pod \"60468f93-bb6e-470c-8c1a-cd6212d58c49\" (UID: \"60468f93-bb6e-470c-8c1a-cd6212d58c49\") "
Sep 08 02:22:46.637654 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:22:46.637570    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60468f93-bb6e-470c-8c1a-cd6212d58c49-plugins-config-volume\") pod \"60468f93-bb6e-470c-8c1a-cd6212d58c49\" (UID: \"60468f93-bb6e-470c-8c1a-cd6212d58c49\") "
Sep 08 02:22:46.637988 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:22:46.637951    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8dc52e3-ab0b-44cd-810f-bac422df9d46-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f8dc52e3-ab0b-44cd-810f-bac422df9d46" (UID: "f8dc52e3-ab0b-44cd-810f-bac422df9d46"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 02:22:46.638105 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:22:46.637962    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60468f93-bb6e-470c-8c1a-cd6212d58c49-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "60468f93-bb6e-470c-8c1a-cd6212d58c49" (UID: "60468f93-bb6e-470c-8c1a-cd6212d58c49"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 02:22:46.639760 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:22:46.639732    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60468f93-bb6e-470c-8c1a-cd6212d58c49-kube-api-access-jwbmj" (OuterVolumeSpecName: "kube-api-access-jwbmj") pod "60468f93-bb6e-470c-8c1a-cd6212d58c49" (UID: "60468f93-bb6e-470c-8c1a-cd6212d58c49"). InnerVolumeSpecName "kube-api-access-jwbmj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 02:22:46.639760 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:22:46.639752    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8dc52e3-ab0b-44cd-810f-bac422df9d46-kube-api-access-rzt9h" (OuterVolumeSpecName: "kube-api-access-rzt9h") pod "f8dc52e3-ab0b-44cd-810f-bac422df9d46" (UID: "f8dc52e3-ab0b-44cd-810f-bac422df9d46"). InnerVolumeSpecName "kube-api-access-rzt9h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 02:22:46.739114 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:22:46.739073    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzt9h\" (UniqueName: \"kubernetes.io/projected/f8dc52e3-ab0b-44cd-810f-bac422df9d46-kube-api-access-rzt9h\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:22:46.739114 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:22:46.739110    2547 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8dc52e3-ab0b-44cd-810f-bac422df9d46-plugins-config-volume\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:22:46.739114 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:22:46.739120    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwbmj\" (UniqueName: \"kubernetes.io/projected/60468f93-bb6e-470c-8c1a-cd6212d58c49-kube-api-access-jwbmj\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:22:46.739371 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:22:46.739128    2547 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60468f93-bb6e-470c-8c1a-cd6212d58c49-plugins-config-volume\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:22:47.318769 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:22:47.318736    2547 generic.go:356] "Generic (PLEG): container finished" podID="f8dc52e3-ab0b-44cd-810f-bac422df9d46" containerID="f4e27b68961eb4f81c3aefcd2e79a0c34f1c95989e0cfc49f869005db446eafc" exitCode=0
Sep 08 02:22:47.319292 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:22:47.318792    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-670bc3-6fc959454b-jqpgb"
Sep 08 02:22:47.319292 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:22:47.318814    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-670bc3-6fc959454b-jqpgb" event={"ID":"f8dc52e3-ab0b-44cd-810f-bac422df9d46","Type":"ContainerDied","Data":"f4e27b68961eb4f81c3aefcd2e79a0c34f1c95989e0cfc49f869005db446eafc"}
Sep 08 02:22:47.319292 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:22:47.318870    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-670bc3-6fc959454b-jqpgb" event={"ID":"f8dc52e3-ab0b-44cd-810f-bac422df9d46","Type":"ContainerDied","Data":"bcecbc046ca9c514c9acfd486006c5958f1027f5afd7141aa794c22f93f09c29"}
Sep 08 02:22:47.319292 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:22:47.318894    2547 scope.go:117] "RemoveContainer" containerID="f4e27b68961eb4f81c3aefcd2e79a0c34f1c95989e0cfc49f869005db446eafc"
Sep 08 02:22:47.320280 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:22:47.320118    2547 generic.go:356] "Generic (PLEG): container finished" podID="60468f93-bb6e-470c-8c1a-cd6212d58c49" containerID="9329c7ecafdfc2cb42b3936d2250a0190c166bc7f1f150c821cfb9be4d248f94" exitCode=0
Sep 08 02:22:47.320280 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:22:47.320164    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-670bc3-76644ff8c-jqf99" event={"ID":"60468f93-bb6e-470c-8c1a-cd6212d58c49","Type":"ContainerDied","Data":"9329c7ecafdfc2cb42b3936d2250a0190c166bc7f1f150c821cfb9be4d248f94"}
Sep 08 02:22:47.320280 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:22:47.320181    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-670bc3-76644ff8c-jqf99"
Sep 08 02:22:47.320280 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:22:47.320212    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-670bc3-76644ff8c-jqf99" event={"ID":"60468f93-bb6e-470c-8c1a-cd6212d58c49","Type":"ContainerDied","Data":"dcc20f76b839470bf2a6427324c56d0335f5f4efa41860b1539cca66e31e594a"}
Sep 08 02:22:47.327694 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:22:47.327673    2547 scope.go:117] "RemoveContainer" containerID="f4e27b68961eb4f81c3aefcd2e79a0c34f1c95989e0cfc49f869005db446eafc"
Sep 08 02:22:47.327990 ip-10-0-142-78 kubenswrapper[2547]: E0908 02:22:47.327970    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4e27b68961eb4f81c3aefcd2e79a0c34f1c95989e0cfc49f869005db446eafc\": container with ID starting with f4e27b68961eb4f81c3aefcd2e79a0c34f1c95989e0cfc49f869005db446eafc not found: ID does not exist" containerID="f4e27b68961eb4f81c3aefcd2e79a0c34f1c95989e0cfc49f869005db446eafc"
Sep 08 02:22:47.328055 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:22:47.327999    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4e27b68961eb4f81c3aefcd2e79a0c34f1c95989e0cfc49f869005db446eafc"} err="failed to get container status \"f4e27b68961eb4f81c3aefcd2e79a0c34f1c95989e0cfc49f869005db446eafc\": rpc error: code = NotFound desc = could not find container \"f4e27b68961eb4f81c3aefcd2e79a0c34f1c95989e0cfc49f869005db446eafc\": container with ID starting with f4e27b68961eb4f81c3aefcd2e79a0c34f1c95989e0cfc49f869005db446eafc not found: ID does not exist"
Sep 08 02:22:47.328055 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:22:47.328016    2547 scope.go:117] "RemoveContainer" containerID="9329c7ecafdfc2cb42b3936d2250a0190c166bc7f1f150c821cfb9be4d248f94"
Sep 08 02:22:47.335863 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:22:47.335843    2547 scope.go:117] "RemoveContainer" containerID="9329c7ecafdfc2cb42b3936d2250a0190c166bc7f1f150c821cfb9be4d248f94"
Sep 08 02:22:47.336298 ip-10-0-142-78 kubenswrapper[2547]: E0908 02:22:47.336215    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9329c7ecafdfc2cb42b3936d2250a0190c166bc7f1f150c821cfb9be4d248f94\": container with ID starting with 9329c7ecafdfc2cb42b3936d2250a0190c166bc7f1f150c821cfb9be4d248f94 not found: ID does not exist" containerID="9329c7ecafdfc2cb42b3936d2250a0190c166bc7f1f150c821cfb9be4d248f94"
Sep 08 02:22:47.336298 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:22:47.336276    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9329c7ecafdfc2cb42b3936d2250a0190c166bc7f1f150c821cfb9be4d248f94"} err="failed to get container status \"9329c7ecafdfc2cb42b3936d2250a0190c166bc7f1f150c821cfb9be4d248f94\": rpc error: code = NotFound desc = could not find container \"9329c7ecafdfc2cb42b3936d2250a0190c166bc7f1f150c821cfb9be4d248f94\": container with ID starting with 9329c7ecafdfc2cb42b3936d2250a0190c166bc7f1f150c821cfb9be4d248f94 not found: ID does not exist"
Sep 08 02:22:47.338601 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:22:47.338577    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-670bc3-6fc959454b-jqpgb"]
Sep 08 02:22:47.342719 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:22:47.342693    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-670bc3-6fc959454b-jqpgb"]
Sep 08 02:22:47.351405 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:22:47.351378    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-670bc3-76644ff8c-jqf99"]
Sep 08 02:22:47.355565 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:22:47.355529    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-670bc3-76644ff8c-jqf99"]
Sep 08 02:22:48.996543 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:22:48.996508    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60468f93-bb6e-470c-8c1a-cd6212d58c49" path="/var/lib/kubelet/pods/60468f93-bb6e-470c-8c1a-cd6212d58c49/volumes"
Sep 08 02:22:48.996915 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:22:48.996866    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8dc52e3-ab0b-44cd-810f-bac422df9d46" path="/var/lib/kubelet/pods/f8dc52e3-ab0b-44cd-810f-bac422df9d46/volumes"
Sep 08 02:22:49.096410 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:22:49.096378    2547 scope.go:117] "RemoveContainer" containerID="962d9a8e3f65f08273877d357d19a2e24594610dafa4f7134b9977d3bced9bf8"
Sep 08 02:22:57.378099 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:22:57.378065    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-fcdd46/rate-test-model-fcdd46-kserve-6fcc98f97f-s5rbk"]
Sep 08 02:22:58.333011 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:22:58.332972    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-527ac8/rate-test-model-527ac8-kserve-65c877868c-bxfb5"]
Sep 08 02:23:03.943417 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:03.943375    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-1d1413-6998765b7f-m47zs"]
Sep 08 02:23:03.943872 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:03.943638    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-1d1413-6998765b7f-m47zs" podUID="98cdb5bb-741c-4f68-b21a-9b75ec91c30e" containerName="payload-processing" containerID="cri-o://ef13f08e5262d75ff831a9367fb158c18ee374efd8e82a5e86a3d5efa09d0dbd" gracePeriod=30
Sep 08 02:23:03.952372 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:03.952340    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1d1413-d9c688cd-mcjkg"]
Sep 08 02:23:03.952586 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:03.952562    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1d1413-d9c688cd-mcjkg" podUID="e0a235cc-d789-472a-94b1-0d62c74aa161" containerName="payload-pre-processing" containerID="cri-o://5ea7252d98629bb392f43f1e846bdc25d70d56d02f57ea817f3fd6eec0ddad22" gracePeriod=30
Sep 08 02:23:04.233757 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:04.233721    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-1d1413-6998765b7f-m47zs"
Sep 08 02:23:04.236513 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:04.236489    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1d1413-d9c688cd-mcjkg"
Sep 08 02:23:04.300617 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:04.300582    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-znxbc\" (UniqueName: \"kubernetes.io/projected/98cdb5bb-741c-4f68-b21a-9b75ec91c30e-kube-api-access-znxbc\") pod \"98cdb5bb-741c-4f68-b21a-9b75ec91c30e\" (UID: \"98cdb5bb-741c-4f68-b21a-9b75ec91c30e\") "
Sep 08 02:23:04.300792 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:04.300653    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0a235cc-d789-472a-94b1-0d62c74aa161-plugins-config-volume\") pod \"e0a235cc-d789-472a-94b1-0d62c74aa161\" (UID: \"e0a235cc-d789-472a-94b1-0d62c74aa161\") "
Sep 08 02:23:04.300792 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:04.300688    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vrfx6\" (UniqueName: \"kubernetes.io/projected/e0a235cc-d789-472a-94b1-0d62c74aa161-kube-api-access-vrfx6\") pod \"e0a235cc-d789-472a-94b1-0d62c74aa161\" (UID: \"e0a235cc-d789-472a-94b1-0d62c74aa161\") "
Sep 08 02:23:04.300792 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:04.300703    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98cdb5bb-741c-4f68-b21a-9b75ec91c30e-plugins-config-volume\") pod \"98cdb5bb-741c-4f68-b21a-9b75ec91c30e\" (UID: \"98cdb5bb-741c-4f68-b21a-9b75ec91c30e\") "
Sep 08 02:23:04.301090 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:04.301062    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0a235cc-d789-472a-94b1-0d62c74aa161-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e0a235cc-d789-472a-94b1-0d62c74aa161" (UID: "e0a235cc-d789-472a-94b1-0d62c74aa161"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 02:23:04.301249 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:04.301119    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98cdb5bb-741c-4f68-b21a-9b75ec91c30e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "98cdb5bb-741c-4f68-b21a-9b75ec91c30e" (UID: "98cdb5bb-741c-4f68-b21a-9b75ec91c30e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 02:23:04.302652 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:04.302619    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98cdb5bb-741c-4f68-b21a-9b75ec91c30e-kube-api-access-znxbc" (OuterVolumeSpecName: "kube-api-access-znxbc") pod "98cdb5bb-741c-4f68-b21a-9b75ec91c30e" (UID: "98cdb5bb-741c-4f68-b21a-9b75ec91c30e"). InnerVolumeSpecName "kube-api-access-znxbc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 02:23:04.302785 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:04.302768    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0a235cc-d789-472a-94b1-0d62c74aa161-kube-api-access-vrfx6" (OuterVolumeSpecName: "kube-api-access-vrfx6") pod "e0a235cc-d789-472a-94b1-0d62c74aa161" (UID: "e0a235cc-d789-472a-94b1-0d62c74aa161"). InnerVolumeSpecName "kube-api-access-vrfx6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 02:23:04.401283 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:04.401250    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znxbc\" (UniqueName: \"kubernetes.io/projected/98cdb5bb-741c-4f68-b21a-9b75ec91c30e-kube-api-access-znxbc\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:23:04.401283 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:04.401279    2547 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0a235cc-d789-472a-94b1-0d62c74aa161-plugins-config-volume\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:23:04.401533 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:04.401293    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrfx6\" (UniqueName: \"kubernetes.io/projected/e0a235cc-d789-472a-94b1-0d62c74aa161-kube-api-access-vrfx6\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:23:04.401533 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:04.401306    2547 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98cdb5bb-741c-4f68-b21a-9b75ec91c30e-plugins-config-volume\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:23:04.401602 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:04.401552    2547 generic.go:356] "Generic (PLEG): container finished" podID="e0a235cc-d789-472a-94b1-0d62c74aa161" containerID="5ea7252d98629bb392f43f1e846bdc25d70d56d02f57ea817f3fd6eec0ddad22" exitCode=0
Sep 08 02:23:04.401602 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:04.401594    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1d1413-d9c688cd-mcjkg"
Sep 08 02:23:04.401666 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:04.401645    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1d1413-d9c688cd-mcjkg" event={"ID":"e0a235cc-d789-472a-94b1-0d62c74aa161","Type":"ContainerDied","Data":"5ea7252d98629bb392f43f1e846bdc25d70d56d02f57ea817f3fd6eec0ddad22"}
Sep 08 02:23:04.401715 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:04.401702    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1d1413-d9c688cd-mcjkg" event={"ID":"e0a235cc-d789-472a-94b1-0d62c74aa161","Type":"ContainerDied","Data":"b3004e4a538a66b469eb419d5f5f7b010645ae9ae612fa12871bb2e5c852d9a7"}
Sep 08 02:23:04.401761 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:04.401729    2547 scope.go:117] "RemoveContainer" containerID="5ea7252d98629bb392f43f1e846bdc25d70d56d02f57ea817f3fd6eec0ddad22"
Sep 08 02:23:04.403021 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:04.403001    2547 generic.go:356] "Generic (PLEG): container finished" podID="98cdb5bb-741c-4f68-b21a-9b75ec91c30e" containerID="ef13f08e5262d75ff831a9367fb158c18ee374efd8e82a5e86a3d5efa09d0dbd" exitCode=0
Sep 08 02:23:04.403119 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:04.403040    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-1d1413-6998765b7f-m47zs" event={"ID":"98cdb5bb-741c-4f68-b21a-9b75ec91c30e","Type":"ContainerDied","Data":"ef13f08e5262d75ff831a9367fb158c18ee374efd8e82a5e86a3d5efa09d0dbd"}
Sep 08 02:23:04.403119 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:04.403045    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-1d1413-6998765b7f-m47zs"
Sep 08 02:23:04.403119 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:04.403078    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-1d1413-6998765b7f-m47zs" event={"ID":"98cdb5bb-741c-4f68-b21a-9b75ec91c30e","Type":"ContainerDied","Data":"6e84dcb60a64e2ec42fc8536a89999c412a81325a6bc579e01c7006c83fa9de4"}
Sep 08 02:23:04.411689 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:04.411665    2547 scope.go:117] "RemoveContainer" containerID="5ea7252d98629bb392f43f1e846bdc25d70d56d02f57ea817f3fd6eec0ddad22"
Sep 08 02:23:04.412004 ip-10-0-142-78 kubenswrapper[2547]: E0908 02:23:04.411985    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ea7252d98629bb392f43f1e846bdc25d70d56d02f57ea817f3fd6eec0ddad22\": container with ID starting with 5ea7252d98629bb392f43f1e846bdc25d70d56d02f57ea817f3fd6eec0ddad22 not found: ID does not exist" containerID="5ea7252d98629bb392f43f1e846bdc25d70d56d02f57ea817f3fd6eec0ddad22"
Sep 08 02:23:04.412084 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:04.412013    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ea7252d98629bb392f43f1e846bdc25d70d56d02f57ea817f3fd6eec0ddad22"} err="failed to get container status \"5ea7252d98629bb392f43f1e846bdc25d70d56d02f57ea817f3fd6eec0ddad22\": rpc error: code = NotFound desc = could not find container \"5ea7252d98629bb392f43f1e846bdc25d70d56d02f57ea817f3fd6eec0ddad22\": container with ID starting with 5ea7252d98629bb392f43f1e846bdc25d70d56d02f57ea817f3fd6eec0ddad22 not found: ID does not exist"
Sep 08 02:23:04.412084 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:04.412030    2547 scope.go:117] "RemoveContainer" containerID="ef13f08e5262d75ff831a9367fb158c18ee374efd8e82a5e86a3d5efa09d0dbd"
Sep 08 02:23:04.420081 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:04.420060    2547 scope.go:117] "RemoveContainer" containerID="ef13f08e5262d75ff831a9367fb158c18ee374efd8e82a5e86a3d5efa09d0dbd"
Sep 08 02:23:04.420399 ip-10-0-142-78 kubenswrapper[2547]: E0908 02:23:04.420381    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef13f08e5262d75ff831a9367fb158c18ee374efd8e82a5e86a3d5efa09d0dbd\": container with ID starting with ef13f08e5262d75ff831a9367fb158c18ee374efd8e82a5e86a3d5efa09d0dbd not found: ID does not exist" containerID="ef13f08e5262d75ff831a9367fb158c18ee374efd8e82a5e86a3d5efa09d0dbd"
Sep 08 02:23:04.420453 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:04.420406    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef13f08e5262d75ff831a9367fb158c18ee374efd8e82a5e86a3d5efa09d0dbd"} err="failed to get container status \"ef13f08e5262d75ff831a9367fb158c18ee374efd8e82a5e86a3d5efa09d0dbd\": rpc error: code = NotFound desc = could not find container \"ef13f08e5262d75ff831a9367fb158c18ee374efd8e82a5e86a3d5efa09d0dbd\": container with ID starting with ef13f08e5262d75ff831a9367fb158c18ee374efd8e82a5e86a3d5efa09d0dbd not found: ID does not exist"
Sep 08 02:23:04.435668 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:04.435635    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-1d1413-6998765b7f-m47zs"]
Sep 08 02:23:04.439336 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:04.439306    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-1d1413-6998765b7f-m47zs"]
Sep 08 02:23:04.478847 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:04.478758    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1d1413-d9c688cd-mcjkg"]
Sep 08 02:23:04.484057 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:04.484018    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1d1413-d9c688cd-mcjkg"]
Sep 08 02:23:04.994536 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:04.994498    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98cdb5bb-741c-4f68-b21a-9b75ec91c30e" path="/var/lib/kubelet/pods/98cdb5bb-741c-4f68-b21a-9b75ec91c30e/volumes"
Sep 08 02:23:04.994930 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:04.994864    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0a235cc-d789-472a-94b1-0d62c74aa161" path="/var/lib/kubelet/pods/e0a235cc-d789-472a-94b1-0d62c74aa161/volumes"
Sep 08 02:23:12.397579 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:12.397536    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-fcdd46/rate-test-model-fcdd46-kserve-6fcc98f97f-s5rbk" podUID="98498211-e317-464f-8537-ab38bbe33904" containerName="main" containerID="cri-o://967c42e8dac2d1ebe9f1a286bc4ef8a0b378c7afbb6f2ffd5ee2753051487cbd" gracePeriod=45
Sep 08 02:23:12.651012 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:12.650945    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-fcdd46/rate-test-model-fcdd46-kserve-6fcc98f97f-s5rbk"
Sep 08 02:23:12.771216 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:12.771164    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98498211-e317-464f-8537-ab38bbe33904-model-cache\") pod \"98498211-e317-464f-8537-ab38bbe33904\" (UID: \"98498211-e317-464f-8537-ab38bbe33904\") "
Sep 08 02:23:12.771394 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:12.771254    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98498211-e317-464f-8537-ab38bbe33904-tmp-dir\") pod \"98498211-e317-464f-8537-ab38bbe33904\" (UID: \"98498211-e317-464f-8537-ab38bbe33904\") "
Sep 08 02:23:12.771394 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:12.771286    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98498211-e317-464f-8537-ab38bbe33904-home\") pod \"98498211-e317-464f-8537-ab38bbe33904\" (UID: \"98498211-e317-464f-8537-ab38bbe33904\") "
Sep 08 02:23:12.771394 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:12.771316    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6gjf7\" (UniqueName: \"kubernetes.io/projected/98498211-e317-464f-8537-ab38bbe33904-kube-api-access-6gjf7\") pod \"98498211-e317-464f-8537-ab38bbe33904\" (UID: \"98498211-e317-464f-8537-ab38bbe33904\") "
Sep 08 02:23:12.771394 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:12.771332    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98498211-e317-464f-8537-ab38bbe33904-dshm\") pod \"98498211-e317-464f-8537-ab38bbe33904\" (UID: \"98498211-e317-464f-8537-ab38bbe33904\") "
Sep 08 02:23:12.771394 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:12.771359    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98498211-e317-464f-8537-ab38bbe33904-tls-certs\") pod \"98498211-e317-464f-8537-ab38bbe33904\" (UID: \"98498211-e317-464f-8537-ab38bbe33904\") "
Sep 08 02:23:12.771693 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:12.771499    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98498211-e317-464f-8537-ab38bbe33904-model-cache" (OuterVolumeSpecName: "model-cache") pod "98498211-e317-464f-8537-ab38bbe33904" (UID: "98498211-e317-464f-8537-ab38bbe33904"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 02:23:12.771693 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:12.771516    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98498211-e317-464f-8537-ab38bbe33904-home" (OuterVolumeSpecName: "home") pod "98498211-e317-464f-8537-ab38bbe33904" (UID: "98498211-e317-464f-8537-ab38bbe33904"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 02:23:12.771693 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:12.771582    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98498211-e317-464f-8537-ab38bbe33904-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "98498211-e317-464f-8537-ab38bbe33904" (UID: "98498211-e317-464f-8537-ab38bbe33904"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 02:23:12.771693 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:12.771629    2547 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98498211-e317-464f-8537-ab38bbe33904-model-cache\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:23:12.771693 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:12.771646    2547 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98498211-e317-464f-8537-ab38bbe33904-home\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:23:12.773593 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:12.773562    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98498211-e317-464f-8537-ab38bbe33904-dshm" (OuterVolumeSpecName: "dshm") pod "98498211-e317-464f-8537-ab38bbe33904" (UID: "98498211-e317-464f-8537-ab38bbe33904"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 02:23:12.773593 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:12.773575    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98498211-e317-464f-8537-ab38bbe33904-kube-api-access-6gjf7" (OuterVolumeSpecName: "kube-api-access-6gjf7") pod "98498211-e317-464f-8537-ab38bbe33904" (UID: "98498211-e317-464f-8537-ab38bbe33904"). InnerVolumeSpecName "kube-api-access-6gjf7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 02:23:12.773764 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:12.773565    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98498211-e317-464f-8537-ab38bbe33904-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "98498211-e317-464f-8537-ab38bbe33904" (UID: "98498211-e317-464f-8537-ab38bbe33904"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 02:23:12.872451 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:12.872417    2547 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98498211-e317-464f-8537-ab38bbe33904-tmp-dir\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:23:12.872451 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:12.872448    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gjf7\" (UniqueName: \"kubernetes.io/projected/98498211-e317-464f-8537-ab38bbe33904-kube-api-access-6gjf7\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:23:12.872451 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:12.872459    2547 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98498211-e317-464f-8537-ab38bbe33904-dshm\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:23:12.872789 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:12.872469    2547 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98498211-e317-464f-8537-ab38bbe33904-tls-certs\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:23:13.355005 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:13.354889    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-527ac8/rate-test-model-527ac8-kserve-65c877868c-bxfb5" podUID="db197e84-ccdb-4599-82c2-e1a6f6d5adc9" containerName="main" containerID="cri-o://cfdcef31bffa6edd0b21163ce9c2732448aa53df3f123fd1009dc6c5765d66b8" gracePeriod=45
Sep 08 02:23:13.445599 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:13.445563    2547 generic.go:356] "Generic (PLEG): container finished" podID="98498211-e317-464f-8537-ab38bbe33904" containerID="967c42e8dac2d1ebe9f1a286bc4ef8a0b378c7afbb6f2ffd5ee2753051487cbd" exitCode=0
Sep 08 02:23:13.446049 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:13.445608    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-fcdd46/rate-test-model-fcdd46-kserve-6fcc98f97f-s5rbk" event={"ID":"98498211-e317-464f-8537-ab38bbe33904","Type":"ContainerDied","Data":"967c42e8dac2d1ebe9f1a286bc4ef8a0b378c7afbb6f2ffd5ee2753051487cbd"}
Sep 08 02:23:13.446049 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:13.445617    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-fcdd46/rate-test-model-fcdd46-kserve-6fcc98f97f-s5rbk"
Sep 08 02:23:13.446049 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:13.445636    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-fcdd46/rate-test-model-fcdd46-kserve-6fcc98f97f-s5rbk" event={"ID":"98498211-e317-464f-8537-ab38bbe33904","Type":"ContainerDied","Data":"95027d94937bc8def6b64f9ec4cb34f44db01c5217e500d73462fa670653ff49"}
Sep 08 02:23:13.446049 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:13.445653    2547 scope.go:117] "RemoveContainer" containerID="967c42e8dac2d1ebe9f1a286bc4ef8a0b378c7afbb6f2ffd5ee2753051487cbd"
Sep 08 02:23:13.461320 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:13.461290    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-fcdd46/rate-test-model-fcdd46-kserve-6fcc98f97f-s5rbk"]
Sep 08 02:23:13.465854 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:13.465737    2547 scope.go:117] "RemoveContainer" containerID="967c42e8dac2d1ebe9f1a286bc4ef8a0b378c7afbb6f2ffd5ee2753051487cbd"
Sep 08 02:23:13.466244 ip-10-0-142-78 kubenswrapper[2547]: E0908 02:23:13.466216    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"967c42e8dac2d1ebe9f1a286bc4ef8a0b378c7afbb6f2ffd5ee2753051487cbd\": container with ID starting with 967c42e8dac2d1ebe9f1a286bc4ef8a0b378c7afbb6f2ffd5ee2753051487cbd not found: ID does not exist" containerID="967c42e8dac2d1ebe9f1a286bc4ef8a0b378c7afbb6f2ffd5ee2753051487cbd"
Sep 08 02:23:13.466332 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:13.466259    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"967c42e8dac2d1ebe9f1a286bc4ef8a0b378c7afbb6f2ffd5ee2753051487cbd"} err="failed to get container status \"967c42e8dac2d1ebe9f1a286bc4ef8a0b378c7afbb6f2ffd5ee2753051487cbd\": rpc error: code = NotFound desc = could not find container \"967c42e8dac2d1ebe9f1a286bc4ef8a0b378c7afbb6f2ffd5ee2753051487cbd\": container with ID starting with 967c42e8dac2d1ebe9f1a286bc4ef8a0b378c7afbb6f2ffd5ee2753051487cbd not found: ID does not exist"
Sep 08 02:23:13.467126 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:13.467103    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-fcdd46/rate-test-model-fcdd46-kserve-6fcc98f97f-s5rbk"]
Sep 08 02:23:13.617621 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:13.617551    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-527ac8/rate-test-model-527ac8-kserve-65c877868c-bxfb5"
Sep 08 02:23:13.679432 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:13.679389    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-khpgk\" (UniqueName: \"kubernetes.io/projected/db197e84-ccdb-4599-82c2-e1a6f6d5adc9-kube-api-access-khpgk\") pod \"db197e84-ccdb-4599-82c2-e1a6f6d5adc9\" (UID: \"db197e84-ccdb-4599-82c2-e1a6f6d5adc9\") "
Sep 08 02:23:13.679638 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:13.679458    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db197e84-ccdb-4599-82c2-e1a6f6d5adc9-tmp-dir\") pod \"db197e84-ccdb-4599-82c2-e1a6f6d5adc9\" (UID: \"db197e84-ccdb-4599-82c2-e1a6f6d5adc9\") "
Sep 08 02:23:13.679638 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:13.679484    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db197e84-ccdb-4599-82c2-e1a6f6d5adc9-tls-certs\") pod \"db197e84-ccdb-4599-82c2-e1a6f6d5adc9\" (UID: \"db197e84-ccdb-4599-82c2-e1a6f6d5adc9\") "
Sep 08 02:23:13.679638 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:13.679504    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db197e84-ccdb-4599-82c2-e1a6f6d5adc9-model-cache\") pod \"db197e84-ccdb-4599-82c2-e1a6f6d5adc9\" (UID: \"db197e84-ccdb-4599-82c2-e1a6f6d5adc9\") "
Sep 08 02:23:13.679638 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:13.679578    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db197e84-ccdb-4599-82c2-e1a6f6d5adc9-dshm\") pod \"db197e84-ccdb-4599-82c2-e1a6f6d5adc9\" (UID: \"db197e84-ccdb-4599-82c2-e1a6f6d5adc9\") "
Sep 08 02:23:13.679638 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:13.679633    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db197e84-ccdb-4599-82c2-e1a6f6d5adc9-home\") pod \"db197e84-ccdb-4599-82c2-e1a6f6d5adc9\" (UID: \"db197e84-ccdb-4599-82c2-e1a6f6d5adc9\") "
Sep 08 02:23:13.679868 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:13.679743    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db197e84-ccdb-4599-82c2-e1a6f6d5adc9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "db197e84-ccdb-4599-82c2-e1a6f6d5adc9" (UID: "db197e84-ccdb-4599-82c2-e1a6f6d5adc9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 02:23:13.679925 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:13.679861    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db197e84-ccdb-4599-82c2-e1a6f6d5adc9-model-cache" (OuterVolumeSpecName: "model-cache") pod "db197e84-ccdb-4599-82c2-e1a6f6d5adc9" (UID: "db197e84-ccdb-4599-82c2-e1a6f6d5adc9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 02:23:13.679999 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:13.679959    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db197e84-ccdb-4599-82c2-e1a6f6d5adc9-home" (OuterVolumeSpecName: "home") pod "db197e84-ccdb-4599-82c2-e1a6f6d5adc9" (UID: "db197e84-ccdb-4599-82c2-e1a6f6d5adc9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 02:23:13.679999 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:13.679986    2547 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db197e84-ccdb-4599-82c2-e1a6f6d5adc9-tmp-dir\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:23:13.680101 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:13.680003    2547 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db197e84-ccdb-4599-82c2-e1a6f6d5adc9-model-cache\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:23:13.681758 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:13.681728    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db197e84-ccdb-4599-82c2-e1a6f6d5adc9-dshm" (OuterVolumeSpecName: "dshm") pod "db197e84-ccdb-4599-82c2-e1a6f6d5adc9" (UID: "db197e84-ccdb-4599-82c2-e1a6f6d5adc9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 02:23:13.681870 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:13.681784    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db197e84-ccdb-4599-82c2-e1a6f6d5adc9-kube-api-access-khpgk" (OuterVolumeSpecName: "kube-api-access-khpgk") pod "db197e84-ccdb-4599-82c2-e1a6f6d5adc9" (UID: "db197e84-ccdb-4599-82c2-e1a6f6d5adc9"). InnerVolumeSpecName "kube-api-access-khpgk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 02:23:13.681870 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:13.681795    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db197e84-ccdb-4599-82c2-e1a6f6d5adc9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "db197e84-ccdb-4599-82c2-e1a6f6d5adc9" (UID: "db197e84-ccdb-4599-82c2-e1a6f6d5adc9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 02:23:13.780906 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:13.780861    2547 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db197e84-ccdb-4599-82c2-e1a6f6d5adc9-dshm\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:23:13.780906 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:13.780897    2547 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db197e84-ccdb-4599-82c2-e1a6f6d5adc9-home\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:23:13.780906 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:13.780906    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khpgk\" (UniqueName: \"kubernetes.io/projected/db197e84-ccdb-4599-82c2-e1a6f6d5adc9-kube-api-access-khpgk\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:23:13.780906 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:13.780917    2547 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db197e84-ccdb-4599-82c2-e1a6f6d5adc9-tls-certs\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:23:14.450892 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:14.450855    2547 generic.go:356] "Generic (PLEG): container finished" podID="db197e84-ccdb-4599-82c2-e1a6f6d5adc9" containerID="cfdcef31bffa6edd0b21163ce9c2732448aa53df3f123fd1009dc6c5765d66b8" exitCode=0
Sep 08 02:23:14.451330 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:14.450903    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-527ac8/rate-test-model-527ac8-kserve-65c877868c-bxfb5"
Sep 08 02:23:14.451330 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:14.450941    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-527ac8/rate-test-model-527ac8-kserve-65c877868c-bxfb5" event={"ID":"db197e84-ccdb-4599-82c2-e1a6f6d5adc9","Type":"ContainerDied","Data":"cfdcef31bffa6edd0b21163ce9c2732448aa53df3f123fd1009dc6c5765d66b8"}
Sep 08 02:23:14.451330 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:14.450982    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-527ac8/rate-test-model-527ac8-kserve-65c877868c-bxfb5" event={"ID":"db197e84-ccdb-4599-82c2-e1a6f6d5adc9","Type":"ContainerDied","Data":"19cfba137d9e9424ba9082d6df1cad6bc5e1fed9025c720cec5f8311d5bd1f52"}
Sep 08 02:23:14.451330 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:14.451000    2547 scope.go:117] "RemoveContainer" containerID="cfdcef31bffa6edd0b21163ce9c2732448aa53df3f123fd1009dc6c5765d66b8"
Sep 08 02:23:14.459491 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:14.459472    2547 scope.go:117] "RemoveContainer" containerID="cfdcef31bffa6edd0b21163ce9c2732448aa53df3f123fd1009dc6c5765d66b8"
Sep 08 02:23:14.459768 ip-10-0-142-78 kubenswrapper[2547]: E0908 02:23:14.459747    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfdcef31bffa6edd0b21163ce9c2732448aa53df3f123fd1009dc6c5765d66b8\": container with ID starting with cfdcef31bffa6edd0b21163ce9c2732448aa53df3f123fd1009dc6c5765d66b8 not found: ID does not exist" containerID="cfdcef31bffa6edd0b21163ce9c2732448aa53df3f123fd1009dc6c5765d66b8"
Sep 08 02:23:14.459830 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:14.459776    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfdcef31bffa6edd0b21163ce9c2732448aa53df3f123fd1009dc6c5765d66b8"} err="failed to get container status \"cfdcef31bffa6edd0b21163ce9c2732448aa53df3f123fd1009dc6c5765d66b8\": rpc error: code = NotFound desc = could not find container \"cfdcef31bffa6edd0b21163ce9c2732448aa53df3f123fd1009dc6c5765d66b8\": container with ID starting with cfdcef31bffa6edd0b21163ce9c2732448aa53df3f123fd1009dc6c5765d66b8 not found: ID does not exist"
Sep 08 02:23:14.470433 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:14.470411    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-527ac8/rate-test-model-527ac8-kserve-65c877868c-bxfb5"]
Sep 08 02:23:14.474260 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:14.474237    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-527ac8/rate-test-model-527ac8-kserve-65c877868c-bxfb5"]
Sep 08 02:23:14.993782 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:14.993751    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98498211-e317-464f-8537-ab38bbe33904" path="/var/lib/kubelet/pods/98498211-e317-464f-8537-ab38bbe33904/volumes"
Sep 08 02:23:14.994121 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:14.994108    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db197e84-ccdb-4599-82c2-e1a6f6d5adc9" path="/var/lib/kubelet/pods/db197e84-ccdb-4599-82c2-e1a6f6d5adc9/volumes"
Sep 08 02:23:32.986582 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:32.986546    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d4845d-79f8f95f4b-2kjbp"]
Sep 08 02:23:32.987048 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:32.986880    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98cdb5bb-741c-4f68-b21a-9b75ec91c30e" containerName="payload-processing"
Sep 08 02:23:32.987048 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:32.986892    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="98cdb5bb-741c-4f68-b21a-9b75ec91c30e" containerName="payload-processing"
Sep 08 02:23:32.987048 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:32.986904    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98498211-e317-464f-8537-ab38bbe33904" containerName="main"
Sep 08 02:23:32.987048 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:32.986910    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="98498211-e317-464f-8537-ab38bbe33904" containerName="main"
Sep 08 02:23:32.987048 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:32.986919    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60468f93-bb6e-470c-8c1a-cd6212d58c49" containerName="payload-processing"
Sep 08 02:23:32.987048 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:32.986927    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="60468f93-bb6e-470c-8c1a-cd6212d58c49" containerName="payload-processing"
Sep 08 02:23:32.987048 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:32.986940    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8dc52e3-ab0b-44cd-810f-bac422df9d46" containerName="payload-pre-processing"
Sep 08 02:23:32.987048 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:32.986945    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8dc52e3-ab0b-44cd-810f-bac422df9d46" containerName="payload-pre-processing"
Sep 08 02:23:32.987048 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:32.986952    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db197e84-ccdb-4599-82c2-e1a6f6d5adc9" containerName="main"
Sep 08 02:23:32.987048 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:32.986957    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="db197e84-ccdb-4599-82c2-e1a6f6d5adc9" containerName="main"
Sep 08 02:23:32.987048 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:32.986972    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0a235cc-d789-472a-94b1-0d62c74aa161" containerName="payload-pre-processing"
Sep 08 02:23:32.987048 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:32.986977    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0a235cc-d789-472a-94b1-0d62c74aa161" containerName="payload-pre-processing"
Sep 08 02:23:32.987048 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:32.987026    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8dc52e3-ab0b-44cd-810f-bac422df9d46" containerName="payload-pre-processing"
Sep 08 02:23:32.987048 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:32.987034    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="98cdb5bb-741c-4f68-b21a-9b75ec91c30e" containerName="payload-processing"
Sep 08 02:23:32.987048 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:32.987041    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="60468f93-bb6e-470c-8c1a-cd6212d58c49" containerName="payload-processing"
Sep 08 02:23:32.987048 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:32.987046    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0a235cc-d789-472a-94b1-0d62c74aa161" containerName="payload-pre-processing"
Sep 08 02:23:32.987048 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:32.987054    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="98498211-e317-464f-8537-ab38bbe33904" containerName="main"
Sep 08 02:23:32.987591 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:32.987059    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="db197e84-ccdb-4599-82c2-e1a6f6d5adc9" containerName="main"
Sep 08 02:23:32.990120 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:32.990090    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d4845d-79f8f95f4b-2kjbp"
Sep 08 02:23:32.992402 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:32.992378    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-d4845d-dockercfg-4wjk8\""
Sep 08 02:23:32.992534 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:32.992513    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-d4845d\""
Sep 08 02:23:33.000333 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:33.000304    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d4845d-79f8f95f4b-2kjbp"]
Sep 08 02:23:33.016255 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:33.016219    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d4845d-74bfb58d9c-h6b8p"]
Sep 08 02:23:33.027833 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:33.027789    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d4845d-74bfb58d9c-h6b8p"
Sep 08 02:23:33.028276 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:33.028249    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d4845d-74bfb58d9c-h6b8p"]
Sep 08 02:23:33.054363 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:33.054327    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28e3bb6b-3ea0-4b98-b4fc-979fad49107c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d4845d-79f8f95f4b-2kjbp\" (UID: \"28e3bb6b-3ea0-4b98-b4fc-979fad49107c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d4845d-79f8f95f4b-2kjbp"
Sep 08 02:23:33.054550 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:33.054382    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sq455\" (UniqueName: \"kubernetes.io/projected/28e3bb6b-3ea0-4b98-b4fc-979fad49107c-kube-api-access-sq455\") pod \"payload-pre-processing-e2e-ipp-b-d4845d-79f8f95f4b-2kjbp\" (UID: \"28e3bb6b-3ea0-4b98-b4fc-979fad49107c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d4845d-79f8f95f4b-2kjbp"
Sep 08 02:23:33.154880 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:33.154832    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sq455\" (UniqueName: \"kubernetes.io/projected/28e3bb6b-3ea0-4b98-b4fc-979fad49107c-kube-api-access-sq455\") pod \"payload-pre-processing-e2e-ipp-b-d4845d-79f8f95f4b-2kjbp\" (UID: \"28e3bb6b-3ea0-4b98-b4fc-979fad49107c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d4845d-79f8f95f4b-2kjbp"
Sep 08 02:23:33.154880 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:33.154884    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6qbf\" (UniqueName: \"kubernetes.io/projected/e9ee4413-722d-4b83-a34c-1f37bf956fa5-kube-api-access-d6qbf\") pod \"payload-processing-e2e-ipp-b-d4845d-74bfb58d9c-h6b8p\" (UID: \"e9ee4413-722d-4b83-a34c-1f37bf956fa5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d4845d-74bfb58d9c-h6b8p"
Sep 08 02:23:33.155158 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:33.154913    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9ee4413-722d-4b83-a34c-1f37bf956fa5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d4845d-74bfb58d9c-h6b8p\" (UID: \"e9ee4413-722d-4b83-a34c-1f37bf956fa5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d4845d-74bfb58d9c-h6b8p"
Sep 08 02:23:33.155158 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:33.155055    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28e3bb6b-3ea0-4b98-b4fc-979fad49107c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d4845d-79f8f95f4b-2kjbp\" (UID: \"28e3bb6b-3ea0-4b98-b4fc-979fad49107c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d4845d-79f8f95f4b-2kjbp"
Sep 08 02:23:33.155784 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:33.155763    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28e3bb6b-3ea0-4b98-b4fc-979fad49107c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d4845d-79f8f95f4b-2kjbp\" (UID: \"28e3bb6b-3ea0-4b98-b4fc-979fad49107c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d4845d-79f8f95f4b-2kjbp"
Sep 08 02:23:33.162578 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:33.162550    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sq455\" (UniqueName: \"kubernetes.io/projected/28e3bb6b-3ea0-4b98-b4fc-979fad49107c-kube-api-access-sq455\") pod \"payload-pre-processing-e2e-ipp-b-d4845d-79f8f95f4b-2kjbp\" (UID: \"28e3bb6b-3ea0-4b98-b4fc-979fad49107c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d4845d-79f8f95f4b-2kjbp"
Sep 08 02:23:33.255970 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:33.255867    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6qbf\" (UniqueName: \"kubernetes.io/projected/e9ee4413-722d-4b83-a34c-1f37bf956fa5-kube-api-access-d6qbf\") pod \"payload-processing-e2e-ipp-b-d4845d-74bfb58d9c-h6b8p\" (UID: \"e9ee4413-722d-4b83-a34c-1f37bf956fa5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d4845d-74bfb58d9c-h6b8p"
Sep 08 02:23:33.255970 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:33.255920    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9ee4413-722d-4b83-a34c-1f37bf956fa5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d4845d-74bfb58d9c-h6b8p\" (UID: \"e9ee4413-722d-4b83-a34c-1f37bf956fa5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d4845d-74bfb58d9c-h6b8p"
Sep 08 02:23:33.256626 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:33.256607    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9ee4413-722d-4b83-a34c-1f37bf956fa5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d4845d-74bfb58d9c-h6b8p\" (UID: \"e9ee4413-722d-4b83-a34c-1f37bf956fa5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d4845d-74bfb58d9c-h6b8p"
Sep 08 02:23:33.263667 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:33.263647    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6qbf\" (UniqueName: \"kubernetes.io/projected/e9ee4413-722d-4b83-a34c-1f37bf956fa5-kube-api-access-d6qbf\") pod \"payload-processing-e2e-ipp-b-d4845d-74bfb58d9c-h6b8p\" (UID: \"e9ee4413-722d-4b83-a34c-1f37bf956fa5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d4845d-74bfb58d9c-h6b8p"
Sep 08 02:23:33.301302 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:33.301256    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d4845d-79f8f95f4b-2kjbp"
Sep 08 02:23:33.339007 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:33.338651    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d4845d-74bfb58d9c-h6b8p"
Sep 08 02:23:33.582406 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:33.582380    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d4845d-79f8f95f4b-2kjbp"]
Sep 08 02:23:33.584600 ip-10-0-142-78 kubenswrapper[2547]: W0908 02:23:33.584576    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28e3bb6b_3ea0_4b98_b4fc_979fad49107c.slice/crio-cf7ce758dc85734da74e53d91e14764dece916ab60f35eda46a62925f4db9f7f WatchSource:0}: Error finding container cf7ce758dc85734da74e53d91e14764dece916ab60f35eda46a62925f4db9f7f: Status 404 returned error can't find the container with id cf7ce758dc85734da74e53d91e14764dece916ab60f35eda46a62925f4db9f7f
Sep 08 02:23:33.615701 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:33.615550    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d4845d-74bfb58d9c-h6b8p"]
Sep 08 02:23:33.618344 ip-10-0-142-78 kubenswrapper[2547]: W0908 02:23:33.618317    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9ee4413_722d_4b83_a34c_1f37bf956fa5.slice/crio-0b896f5be2bd6a68a6fca512f7e8e546ed43bd453a3a31c0c3415c3981f6f5a4 WatchSource:0}: Error finding container 0b896f5be2bd6a68a6fca512f7e8e546ed43bd453a3a31c0c3415c3981f6f5a4: Status 404 returned error can't find the container with id 0b896f5be2bd6a68a6fca512f7e8e546ed43bd453a3a31c0c3415c3981f6f5a4
Sep 08 02:23:34.548404 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:34.548369    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d4845d-74bfb58d9c-h6b8p" event={"ID":"e9ee4413-722d-4b83-a34c-1f37bf956fa5","Type":"ContainerStarted","Data":"408b60950ff12fe0185f2f85c5672e31fb08738712de3243e6a448be57210e73"}
Sep 08 02:23:34.548837 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:34.548435    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d4845d-74bfb58d9c-h6b8p" event={"ID":"e9ee4413-722d-4b83-a34c-1f37bf956fa5","Type":"ContainerStarted","Data":"0b896f5be2bd6a68a6fca512f7e8e546ed43bd453a3a31c0c3415c3981f6f5a4"}
Sep 08 02:23:34.548837 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:34.548588    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-d4845d-74bfb58d9c-h6b8p"
Sep 08 02:23:34.549806 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:34.549785    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d4845d-79f8f95f4b-2kjbp" event={"ID":"28e3bb6b-3ea0-4b98-b4fc-979fad49107c","Type":"ContainerStarted","Data":"53506b17c89e5efa94fb159bef0aeddf2d4c89c00cdc11a1097feba7b678b865"}
Sep 08 02:23:34.549868 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:34.549824    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d4845d-79f8f95f4b-2kjbp" event={"ID":"28e3bb6b-3ea0-4b98-b4fc-979fad49107c","Type":"ContainerStarted","Data":"cf7ce758dc85734da74e53d91e14764dece916ab60f35eda46a62925f4db9f7f"}
Sep 08 02:23:34.549868 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:34.549860    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d4845d-79f8f95f4b-2kjbp"
Sep 08 02:23:34.565188 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:34.565119    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-d4845d-74bfb58d9c-h6b8p" podStartSLOduration=2.565103444 podStartE2EDuration="2.565103444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:23:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:23:34.563705915 +0000 UTC m=+1486.087481005" watchObservedRunningTime="2026-09-08 02:23:34.565103444 +0000 UTC m=+1486.088878584"
Sep 08 02:23:34.581452 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:34.581389    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d4845d-79f8f95f4b-2kjbp" podStartSLOduration=2.581371291 podStartE2EDuration="2.581371291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:23:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:23:34.579594479 +0000 UTC m=+1486.103369559" watchObservedRunningTime="2026-09-08 02:23:34.581371291 +0000 UTC m=+1486.105146371"
Sep 08 02:23:45.556992 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:45.556955    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-d4845d-74bfb58d9c-h6b8p"
Sep 08 02:23:45.557484 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:23:45.557041    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d4845d-79f8f95f4b-2kjbp"
Sep 08 02:24:01.470353 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:24:01.470312    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-d4845d-74bfb58d9c-h6b8p_e9ee4413-722d-4b83-a34c-1f37bf956fa5/payload-processing/0.log"
Sep 08 02:24:14.212482 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:24:14.212397    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-b23439/ipp-route-b23439-kserve-59c6cdf559-zlmmv"]
Sep 08 02:24:14.216029 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:24:14.216006    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-b23439/ipp-route-b23439-kserve-59c6cdf559-zlmmv"
Sep 08 02:24:14.219404 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:24:14.219367    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-b23439\"/\"openshift-service-ca.crt\""
Sep 08 02:24:14.219522 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:24:14.219407    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-b23439\"/\"kube-root-ca.crt\""
Sep 08 02:24:14.219522 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:24:14.219428    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-b23439\"/\"default-dockercfg-zx9v5\""
Sep 08 02:24:14.219522 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:24:14.219450    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-b23439\"/\"ipp-route-b23439-kserve-self-signed-certs\""
Sep 08 02:24:14.225759 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:24:14.225735    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-b23439/ipp-route-b23439-kserve-59c6cdf559-zlmmv"]
Sep 08 02:24:14.316284 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:24:14.316251    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94501423-47ec-4a6f-957d-8097786b4bc3-tls-certs\") pod \"ipp-route-b23439-kserve-59c6cdf559-zlmmv\" (UID: \"94501423-47ec-4a6f-957d-8097786b4bc3\") " pod="ai-tenant-e2e-ipp-a-b23439/ipp-route-b23439-kserve-59c6cdf559-zlmmv"
Sep 08 02:24:14.316284 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:24:14.316286    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rhzq\" (UniqueName: \"kubernetes.io/projected/94501423-47ec-4a6f-957d-8097786b4bc3-kube-api-access-8rhzq\") pod \"ipp-route-b23439-kserve-59c6cdf559-zlmmv\" (UID: \"94501423-47ec-4a6f-957d-8097786b4bc3\") " pod="ai-tenant-e2e-ipp-a-b23439/ipp-route-b23439-kserve-59c6cdf559-zlmmv"
Sep 08 02:24:14.316500 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:24:14.316330    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94501423-47ec-4a6f-957d-8097786b4bc3-home\") pod \"ipp-route-b23439-kserve-59c6cdf559-zlmmv\" (UID: \"94501423-47ec-4a6f-957d-8097786b4bc3\") " pod="ai-tenant-e2e-ipp-a-b23439/ipp-route-b23439-kserve-59c6cdf559-zlmmv"
Sep 08 02:24:14.316500 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:24:14.316350    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94501423-47ec-4a6f-957d-8097786b4bc3-tmp-dir\") pod \"ipp-route-b23439-kserve-59c6cdf559-zlmmv\" (UID: \"94501423-47ec-4a6f-957d-8097786b4bc3\") " pod="ai-tenant-e2e-ipp-a-b23439/ipp-route-b23439-kserve-59c6cdf559-zlmmv"
Sep 08 02:24:14.316500 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:24:14.316385    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94501423-47ec-4a6f-957d-8097786b4bc3-dshm\") pod \"ipp-route-b23439-kserve-59c6cdf559-zlmmv\" (UID: \"94501423-47ec-4a6f-957d-8097786b4bc3\") " pod="ai-tenant-e2e-ipp-a-b23439/ipp-route-b23439-kserve-59c6cdf559-zlmmv"
Sep 08 02:24:14.316500 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:24:14.316431    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94501423-47ec-4a6f-957d-8097786b4bc3-model-cache\") pod \"ipp-route-b23439-kserve-59c6cdf559-zlmmv\" (UID: \"94501423-47ec-4a6f-957d-8097786b4bc3\") " pod="ai-tenant-e2e-ipp-a-b23439/ipp-route-b23439-kserve-59c6cdf559-zlmmv"
Sep 08 02:24:14.417785 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:24:14.417747    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94501423-47ec-4a6f-957d-8097786b4bc3-dshm\") pod \"ipp-route-b23439-kserve-59c6cdf559-zlmmv\" (UID: \"94501423-47ec-4a6f-957d-8097786b4bc3\") " pod="ai-tenant-e2e-ipp-a-b23439/ipp-route-b23439-kserve-59c6cdf559-zlmmv"
Sep 08 02:24:14.417785 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:24:14.417795    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94501423-47ec-4a6f-957d-8097786b4bc3-model-cache\") pod \"ipp-route-b23439-kserve-59c6cdf559-zlmmv\" (UID: \"94501423-47ec-4a6f-957d-8097786b4bc3\") " pod="ai-tenant-e2e-ipp-a-b23439/ipp-route-b23439-kserve-59c6cdf559-zlmmv"
Sep 08 02:24:14.418052 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:24:14.417833    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94501423-47ec-4a6f-957d-8097786b4bc3-tls-certs\") pod \"ipp-route-b23439-kserve-59c6cdf559-zlmmv\" (UID: \"94501423-47ec-4a6f-957d-8097786b4bc3\") " pod="ai-tenant-e2e-ipp-a-b23439/ipp-route-b23439-kserve-59c6cdf559-zlmmv"
Sep 08 02:24:14.418052 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:24:14.417856    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rhzq\" (UniqueName: \"kubernetes.io/projected/94501423-47ec-4a6f-957d-8097786b4bc3-kube-api-access-8rhzq\") pod \"ipp-route-b23439-kserve-59c6cdf559-zlmmv\" (UID: \"94501423-47ec-4a6f-957d-8097786b4bc3\") " pod="ai-tenant-e2e-ipp-a-b23439/ipp-route-b23439-kserve-59c6cdf559-zlmmv"
Sep 08 02:24:14.418052 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:24:14.417910    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94501423-47ec-4a6f-957d-8097786b4bc3-home\") pod \"ipp-route-b23439-kserve-59c6cdf559-zlmmv\" (UID: \"94501423-47ec-4a6f-957d-8097786b4bc3\") " pod="ai-tenant-e2e-ipp-a-b23439/ipp-route-b23439-kserve-59c6cdf559-zlmmv"
Sep 08 02:24:14.418052 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:24:14.417931    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94501423-47ec-4a6f-957d-8097786b4bc3-tmp-dir\") pod \"ipp-route-b23439-kserve-59c6cdf559-zlmmv\" (UID: \"94501423-47ec-4a6f-957d-8097786b4bc3\") " pod="ai-tenant-e2e-ipp-a-b23439/ipp-route-b23439-kserve-59c6cdf559-zlmmv"
Sep 08 02:24:14.418359 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:24:14.418332    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94501423-47ec-4a6f-957d-8097786b4bc3-model-cache\") pod \"ipp-route-b23439-kserve-59c6cdf559-zlmmv\" (UID: \"94501423-47ec-4a6f-957d-8097786b4bc3\") " pod="ai-tenant-e2e-ipp-a-b23439/ipp-route-b23439-kserve-59c6cdf559-zlmmv"
Sep 08 02:24:14.418359 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:24:14.418352    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94501423-47ec-4a6f-957d-8097786b4bc3-tmp-dir\") pod \"ipp-route-b23439-kserve-59c6cdf559-zlmmv\" (UID: \"94501423-47ec-4a6f-957d-8097786b4bc3\") " pod="ai-tenant-e2e-ipp-a-b23439/ipp-route-b23439-kserve-59c6cdf559-zlmmv"
Sep 08 02:24:14.418531 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:24:14.418395    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94501423-47ec-4a6f-957d-8097786b4bc3-home\") pod \"ipp-route-b23439-kserve-59c6cdf559-zlmmv\" (UID: \"94501423-47ec-4a6f-957d-8097786b4bc3\") " pod="ai-tenant-e2e-ipp-a-b23439/ipp-route-b23439-kserve-59c6cdf559-zlmmv"
Sep 08 02:24:14.420059 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:24:14.420042    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94501423-47ec-4a6f-957d-8097786b4bc3-dshm\") pod \"ipp-route-b23439-kserve-59c6cdf559-zlmmv\" (UID: \"94501423-47ec-4a6f-957d-8097786b4bc3\") " pod="ai-tenant-e2e-ipp-a-b23439/ipp-route-b23439-kserve-59c6cdf559-zlmmv"
Sep 08 02:24:14.420411 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:24:14.420394    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94501423-47ec-4a6f-957d-8097786b4bc3-tls-certs\") pod \"ipp-route-b23439-kserve-59c6cdf559-zlmmv\" (UID: \"94501423-47ec-4a6f-957d-8097786b4bc3\") " pod="ai-tenant-e2e-ipp-a-b23439/ipp-route-b23439-kserve-59c6cdf559-zlmmv"
Sep 08 02:24:14.425924 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:24:14.425895    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rhzq\" (UniqueName: \"kubernetes.io/projected/94501423-47ec-4a6f-957d-8097786b4bc3-kube-api-access-8rhzq\") pod \"ipp-route-b23439-kserve-59c6cdf559-zlmmv\" (UID: \"94501423-47ec-4a6f-957d-8097786b4bc3\") " pod="ai-tenant-e2e-ipp-a-b23439/ipp-route-b23439-kserve-59c6cdf559-zlmmv"
Sep 08 02:24:14.526987 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:24:14.526895    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-b23439/ipp-route-b23439-kserve-59c6cdf559-zlmmv"
Sep 08 02:24:14.795670 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:24:14.795590    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-b23439/ipp-route-b23439-kserve-59c6cdf559-zlmmv"]
Sep 08 02:24:14.798697 ip-10-0-142-78 kubenswrapper[2547]: W0908 02:24:14.798668    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod94501423_47ec_4a6f_957d_8097786b4bc3.slice/crio-7c540ab6cb6292929d1a301fb9306cae831694617c5411d4ded3025c7a49febd WatchSource:0}: Error finding container 7c540ab6cb6292929d1a301fb9306cae831694617c5411d4ded3025c7a49febd: Status 404 returned error can't find the container with id 7c540ab6cb6292929d1a301fb9306cae831694617c5411d4ded3025c7a49febd
Sep 08 02:24:15.736806 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:24:15.736769    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-b23439/ipp-route-b23439-kserve-59c6cdf559-zlmmv" event={"ID":"94501423-47ec-4a6f-957d-8097786b4bc3","Type":"ContainerStarted","Data":"408b7e1a69996cdba3bbd4e50e5c190271089f7878fd6c009ccf72f4440513fc"}
Sep 08 02:24:15.737216 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:24:15.736814    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-b23439/ipp-route-b23439-kserve-59c6cdf559-zlmmv" event={"ID":"94501423-47ec-4a6f-957d-8097786b4bc3","Type":"ContainerStarted","Data":"7c540ab6cb6292929d1a301fb9306cae831694617c5411d4ded3025c7a49febd"}
Sep 08 02:24:15.754775 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:24:15.754712    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-b23439/ipp-route-b23439-kserve-59c6cdf559-zlmmv" podStartSLOduration=1.7546946879999998 podStartE2EDuration="1.754694688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:24:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:24:15.754430698 +0000 UTC m=+1527.278205788" watchObservedRunningTime="2026-09-08 02:24:15.754694688 +0000 UTC m=+1527.278469780"
Sep 08 02:24:24.527801 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:24:24.527766    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-b23439/ipp-route-b23439-kserve-59c6cdf559-zlmmv"
Sep 08 02:24:24.528210 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:24:24.527814    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-b23439/ipp-route-b23439-kserve-59c6cdf559-zlmmv"
Sep 08 02:24:24.539893 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:24:24.539867    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-b23439/ipp-route-b23439-kserve-59c6cdf559-zlmmv"
Sep 08 02:24:24.794538 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:24:24.794454    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-b23439/ipp-route-b23439-kserve-59c6cdf559-zlmmv"
Sep 08 02:24:41.350338 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:24:41.350306    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-d4845d-74bfb58d9c-h6b8p_e9ee4413-722d-4b83-a34c-1f37bf956fa5/payload-processing/0.log"
Sep 08 02:24:46.487146 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:24:46.487097    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ae715d-57ccc84694-c2s9s"]
Sep 08 02:24:46.490650 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:24:46.490631    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ae715d-57ccc84694-c2s9s"
Sep 08 02:24:46.493006 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:24:46.492979    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-ae715d\""
Sep 08 02:24:46.493131 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:24:46.493059    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-ae715d-dockercfg-b6f2k\""
Sep 08 02:24:46.500736 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:24:46.500713    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ae715d-57ccc84694-c2s9s"]
Sep 08 02:24:46.604625 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:24:46.604585    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ded89ca-283d-4848-900b-9c93239b0fe6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-ae715d-57ccc84694-c2s9s\" (UID: \"2ded89ca-283d-4848-900b-9c93239b0fe6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ae715d-57ccc84694-c2s9s"
Sep 08 02:24:46.604826 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:24:46.604659    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rdll\" (UniqueName: \"kubernetes.io/projected/2ded89ca-283d-4848-900b-9c93239b0fe6-kube-api-access-2rdll\") pod \"payload-pre-processing-e2e-ipp-cleanup-ae715d-57ccc84694-c2s9s\" (UID: \"2ded89ca-283d-4848-900b-9c93239b0fe6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ae715d-57ccc84694-c2s9s"
Sep 08 02:24:46.705358 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:24:46.705321    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rdll\" (UniqueName: \"kubernetes.io/projected/2ded89ca-283d-4848-900b-9c93239b0fe6-kube-api-access-2rdll\") pod \"payload-pre-processing-e2e-ipp-cleanup-ae715d-57ccc84694-c2s9s\" (UID: \"2ded89ca-283d-4848-900b-9c93239b0fe6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ae715d-57ccc84694-c2s9s"
Sep 08 02:24:46.705545 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:24:46.705405    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ded89ca-283d-4848-900b-9c93239b0fe6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-ae715d-57ccc84694-c2s9s\" (UID: \"2ded89ca-283d-4848-900b-9c93239b0fe6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ae715d-57ccc84694-c2s9s"
Sep 08 02:24:46.706159 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:24:46.706117    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ded89ca-283d-4848-900b-9c93239b0fe6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-ae715d-57ccc84694-c2s9s\" (UID: \"2ded89ca-283d-4848-900b-9c93239b0fe6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ae715d-57ccc84694-c2s9s"
Sep 08 02:24:46.713421 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:24:46.713401    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rdll\" (UniqueName: \"kubernetes.io/projected/2ded89ca-283d-4848-900b-9c93239b0fe6-kube-api-access-2rdll\") pod \"payload-pre-processing-e2e-ipp-cleanup-ae715d-57ccc84694-c2s9s\" (UID: \"2ded89ca-283d-4848-900b-9c93239b0fe6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ae715d-57ccc84694-c2s9s"
Sep 08 02:24:46.802660 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:24:46.802563    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ae715d-57ccc84694-c2s9s"
Sep 08 02:24:47.074007 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:24:47.073977    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ae715d-57ccc84694-c2s9s"]
Sep 08 02:24:47.075818 ip-10-0-142-78 kubenswrapper[2547]: W0908 02:24:47.075792    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ded89ca_283d_4848_900b_9c93239b0fe6.slice/crio-872e7afe4f6efd8be663e3265141e57e205574dbe7355c3d04d60c6cccd0fc93 WatchSource:0}: Error finding container 872e7afe4f6efd8be663e3265141e57e205574dbe7355c3d04d60c6cccd0fc93: Status 404 returned error can't find the container with id 872e7afe4f6efd8be663e3265141e57e205574dbe7355c3d04d60c6cccd0fc93
Sep 08 02:24:47.889046 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:24:47.889011    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ae715d-57ccc84694-c2s9s" event={"ID":"2ded89ca-283d-4848-900b-9c93239b0fe6","Type":"ContainerStarted","Data":"f2d325165512cb05162be1e7b5c08dea8b6b223ad918f5e1e914709d7b5aa94b"}
Sep 08 02:24:47.889547 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:24:47.889066    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ae715d-57ccc84694-c2s9s" event={"ID":"2ded89ca-283d-4848-900b-9c93239b0fe6","Type":"ContainerStarted","Data":"872e7afe4f6efd8be663e3265141e57e205574dbe7355c3d04d60c6cccd0fc93"}
Sep 08 02:24:47.889547 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:24:47.889116    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ae715d-57ccc84694-c2s9s"
Sep 08 02:24:47.910068 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:24:47.910014    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ae715d-57ccc84694-c2s9s" podStartSLOduration=1.909999679 podStartE2EDuration="1.909999679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:24:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:24:47.908213777 +0000 UTC m=+1559.431988857" watchObservedRunningTime="2026-09-08 02:24:47.909999679 +0000 UTC m=+1559.433774759"
Sep 08 02:24:58.896216 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:24:58.896183    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ae715d-57ccc84694-c2s9s"
Sep 08 02:25:16.438710 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:25:16.438649    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ae715d-57ccc84694-c2s9s"]
Sep 08 02:25:16.439647 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:25:16.439590    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ae715d-57ccc84694-c2s9s" podUID="2ded89ca-283d-4848-900b-9c93239b0fe6" containerName="payload-pre-processing" containerID="cri-o://f2d325165512cb05162be1e7b5c08dea8b6b223ad918f5e1e914709d7b5aa94b" gracePeriod=30
Sep 08 02:25:16.709785 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:25:16.709758    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ae715d-57ccc84694-c2s9s"
Sep 08 02:25:16.868621 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:25:16.868576    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2rdll\" (UniqueName: \"kubernetes.io/projected/2ded89ca-283d-4848-900b-9c93239b0fe6-kube-api-access-2rdll\") pod \"2ded89ca-283d-4848-900b-9c93239b0fe6\" (UID: \"2ded89ca-283d-4848-900b-9c93239b0fe6\") "
Sep 08 02:25:16.868791 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:25:16.868670    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ded89ca-283d-4848-900b-9c93239b0fe6-plugins-config-volume\") pod \"2ded89ca-283d-4848-900b-9c93239b0fe6\" (UID: \"2ded89ca-283d-4848-900b-9c93239b0fe6\") "
Sep 08 02:25:16.869036 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:25:16.869010    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ded89ca-283d-4848-900b-9c93239b0fe6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2ded89ca-283d-4848-900b-9c93239b0fe6" (UID: "2ded89ca-283d-4848-900b-9c93239b0fe6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 02:25:16.870690 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:25:16.870664    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ded89ca-283d-4848-900b-9c93239b0fe6-kube-api-access-2rdll" (OuterVolumeSpecName: "kube-api-access-2rdll") pod "2ded89ca-283d-4848-900b-9c93239b0fe6" (UID: "2ded89ca-283d-4848-900b-9c93239b0fe6"). InnerVolumeSpecName "kube-api-access-2rdll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 02:25:16.969843 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:25:16.969740    2547 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ded89ca-283d-4848-900b-9c93239b0fe6-plugins-config-volume\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:25:16.969843 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:25:16.969776    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rdll\" (UniqueName: \"kubernetes.io/projected/2ded89ca-283d-4848-900b-9c93239b0fe6-kube-api-access-2rdll\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:25:17.024761 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:25:17.024721    2547 generic.go:356] "Generic (PLEG): container finished" podID="2ded89ca-283d-4848-900b-9c93239b0fe6" containerID="f2d325165512cb05162be1e7b5c08dea8b6b223ad918f5e1e914709d7b5aa94b" exitCode=0
Sep 08 02:25:17.024761 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:25:17.024763    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ae715d-57ccc84694-c2s9s"
Sep 08 02:25:17.025033 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:25:17.024809    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ae715d-57ccc84694-c2s9s" event={"ID":"2ded89ca-283d-4848-900b-9c93239b0fe6","Type":"ContainerDied","Data":"f2d325165512cb05162be1e7b5c08dea8b6b223ad918f5e1e914709d7b5aa94b"}
Sep 08 02:25:17.025033 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:25:17.024868    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ae715d-57ccc84694-c2s9s" event={"ID":"2ded89ca-283d-4848-900b-9c93239b0fe6","Type":"ContainerDied","Data":"872e7afe4f6efd8be663e3265141e57e205574dbe7355c3d04d60c6cccd0fc93"}
Sep 08 02:25:17.025033 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:25:17.024899    2547 scope.go:117] "RemoveContainer" containerID="f2d325165512cb05162be1e7b5c08dea8b6b223ad918f5e1e914709d7b5aa94b"
Sep 08 02:25:17.033420 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:25:17.033398    2547 scope.go:117] "RemoveContainer" containerID="f2d325165512cb05162be1e7b5c08dea8b6b223ad918f5e1e914709d7b5aa94b"
Sep 08 02:25:17.033750 ip-10-0-142-78 kubenswrapper[2547]: E0908 02:25:17.033729    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2d325165512cb05162be1e7b5c08dea8b6b223ad918f5e1e914709d7b5aa94b\": container with ID starting with f2d325165512cb05162be1e7b5c08dea8b6b223ad918f5e1e914709d7b5aa94b not found: ID does not exist" containerID="f2d325165512cb05162be1e7b5c08dea8b6b223ad918f5e1e914709d7b5aa94b"
Sep 08 02:25:17.033795 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:25:17.033760    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2d325165512cb05162be1e7b5c08dea8b6b223ad918f5e1e914709d7b5aa94b"} err="failed to get container status \"f2d325165512cb05162be1e7b5c08dea8b6b223ad918f5e1e914709d7b5aa94b\": rpc error: code = NotFound desc = could not find container \"f2d325165512cb05162be1e7b5c08dea8b6b223ad918f5e1e914709d7b5aa94b\": container with ID starting with f2d325165512cb05162be1e7b5c08dea8b6b223ad918f5e1e914709d7b5aa94b not found: ID does not exist"
Sep 08 02:25:17.040958 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:25:17.040926    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ae715d-57ccc84694-c2s9s"]
Sep 08 02:25:17.044619 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:25:17.044593    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ae715d-57ccc84694-c2s9s"]
Sep 08 02:25:18.994466 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:25:18.994433    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ded89ca-283d-4848-900b-9c93239b0fe6" path="/var/lib/kubelet/pods/2ded89ca-283d-4848-900b-9c93239b0fe6/volumes"
Sep 08 02:25:47.345043 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:25:47.344958    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-b23439/ipp-route-b23439-kserve-59c6cdf559-zlmmv"]
Sep 08 02:26:02.364805 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:26:02.364742    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-b23439/ipp-route-b23439-kserve-59c6cdf559-zlmmv" podUID="94501423-47ec-4a6f-957d-8097786b4bc3" containerName="main" containerID="cri-o://408b7e1a69996cdba3bbd4e50e5c190271089f7878fd6c009ccf72f4440513fc" gracePeriod=45
Sep 08 02:26:02.619380 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:26:02.619299    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-b23439/ipp-route-b23439-kserve-59c6cdf559-zlmmv"
Sep 08 02:26:02.774032 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:26:02.773994    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94501423-47ec-4a6f-957d-8097786b4bc3-model-cache\") pod \"94501423-47ec-4a6f-957d-8097786b4bc3\" (UID: \"94501423-47ec-4a6f-957d-8097786b4bc3\") "
Sep 08 02:26:02.774032 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:26:02.774039    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94501423-47ec-4a6f-957d-8097786b4bc3-tmp-dir\") pod \"94501423-47ec-4a6f-957d-8097786b4bc3\" (UID: \"94501423-47ec-4a6f-957d-8097786b4bc3\") "
Sep 08 02:26:02.774311 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:26:02.774062    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8rhzq\" (UniqueName: \"kubernetes.io/projected/94501423-47ec-4a6f-957d-8097786b4bc3-kube-api-access-8rhzq\") pod \"94501423-47ec-4a6f-957d-8097786b4bc3\" (UID: \"94501423-47ec-4a6f-957d-8097786b4bc3\") "
Sep 08 02:26:02.774311 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:26:02.774100    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94501423-47ec-4a6f-957d-8097786b4bc3-tls-certs\") pod \"94501423-47ec-4a6f-957d-8097786b4bc3\" (UID: \"94501423-47ec-4a6f-957d-8097786b4bc3\") "
Sep 08 02:26:02.774311 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:26:02.774131    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94501423-47ec-4a6f-957d-8097786b4bc3-home\") pod \"94501423-47ec-4a6f-957d-8097786b4bc3\" (UID: \"94501423-47ec-4a6f-957d-8097786b4bc3\") "
Sep 08 02:26:02.774311 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:26:02.774225    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94501423-47ec-4a6f-957d-8097786b4bc3-dshm\") pod \"94501423-47ec-4a6f-957d-8097786b4bc3\" (UID: \"94501423-47ec-4a6f-957d-8097786b4bc3\") "
Sep 08 02:26:02.774520 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:26:02.774411    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94501423-47ec-4a6f-957d-8097786b4bc3-model-cache" (OuterVolumeSpecName: "model-cache") pod "94501423-47ec-4a6f-957d-8097786b4bc3" (UID: "94501423-47ec-4a6f-957d-8097786b4bc3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 02:26:02.774520 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:26:02.774405    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94501423-47ec-4a6f-957d-8097786b4bc3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "94501423-47ec-4a6f-957d-8097786b4bc3" (UID: "94501423-47ec-4a6f-957d-8097786b4bc3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 02:26:02.774611 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:26:02.774511    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94501423-47ec-4a6f-957d-8097786b4bc3-home" (OuterVolumeSpecName: "home") pod "94501423-47ec-4a6f-957d-8097786b4bc3" (UID: "94501423-47ec-4a6f-957d-8097786b4bc3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 02:26:02.774611 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:26:02.774592    2547 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94501423-47ec-4a6f-957d-8097786b4bc3-model-cache\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:26:02.774705 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:26:02.774612    2547 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94501423-47ec-4a6f-957d-8097786b4bc3-tmp-dir\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:26:02.776478 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:26:02.776448    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94501423-47ec-4a6f-957d-8097786b4bc3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "94501423-47ec-4a6f-957d-8097786b4bc3" (UID: "94501423-47ec-4a6f-957d-8097786b4bc3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 02:26:02.776478 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:26:02.776460    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94501423-47ec-4a6f-957d-8097786b4bc3-dshm" (OuterVolumeSpecName: "dshm") pod "94501423-47ec-4a6f-957d-8097786b4bc3" (UID: "94501423-47ec-4a6f-957d-8097786b4bc3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 02:26:02.776478 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:26:02.776463    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94501423-47ec-4a6f-957d-8097786b4bc3-kube-api-access-8rhzq" (OuterVolumeSpecName: "kube-api-access-8rhzq") pod "94501423-47ec-4a6f-957d-8097786b4bc3" (UID: "94501423-47ec-4a6f-957d-8097786b4bc3"). InnerVolumeSpecName "kube-api-access-8rhzq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 02:26:02.875232 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:26:02.875115    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rhzq\" (UniqueName: \"kubernetes.io/projected/94501423-47ec-4a6f-957d-8097786b4bc3-kube-api-access-8rhzq\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:26:02.875232 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:26:02.875177    2547 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94501423-47ec-4a6f-957d-8097786b4bc3-tls-certs\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:26:02.875232 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:26:02.875188    2547 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94501423-47ec-4a6f-957d-8097786b4bc3-home\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:26:02.875232 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:26:02.875197    2547 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94501423-47ec-4a6f-957d-8097786b4bc3-dshm\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:26:03.234337 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:26:03.234298    2547 generic.go:356] "Generic (PLEG): container finished" podID="94501423-47ec-4a6f-957d-8097786b4bc3" containerID="408b7e1a69996cdba3bbd4e50e5c190271089f7878fd6c009ccf72f4440513fc" exitCode=0
Sep 08 02:26:03.234521 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:26:03.234365    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-b23439/ipp-route-b23439-kserve-59c6cdf559-zlmmv" event={"ID":"94501423-47ec-4a6f-957d-8097786b4bc3","Type":"ContainerDied","Data":"408b7e1a69996cdba3bbd4e50e5c190271089f7878fd6c009ccf72f4440513fc"}
Sep 08 02:26:03.234521 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:26:03.234374    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-b23439/ipp-route-b23439-kserve-59c6cdf559-zlmmv"
Sep 08 02:26:03.234521 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:26:03.234405    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-b23439/ipp-route-b23439-kserve-59c6cdf559-zlmmv" event={"ID":"94501423-47ec-4a6f-957d-8097786b4bc3","Type":"ContainerDied","Data":"7c540ab6cb6292929d1a301fb9306cae831694617c5411d4ded3025c7a49febd"}
Sep 08 02:26:03.234521 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:26:03.234428    2547 scope.go:117] "RemoveContainer" containerID="408b7e1a69996cdba3bbd4e50e5c190271089f7878fd6c009ccf72f4440513fc"
Sep 08 02:26:03.243147 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:26:03.243113    2547 scope.go:117] "RemoveContainer" containerID="408b7e1a69996cdba3bbd4e50e5c190271089f7878fd6c009ccf72f4440513fc"
Sep 08 02:26:03.243464 ip-10-0-142-78 kubenswrapper[2547]: E0908 02:26:03.243440    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"408b7e1a69996cdba3bbd4e50e5c190271089f7878fd6c009ccf72f4440513fc\": container with ID starting with 408b7e1a69996cdba3bbd4e50e5c190271089f7878fd6c009ccf72f4440513fc not found: ID does not exist" containerID="408b7e1a69996cdba3bbd4e50e5c190271089f7878fd6c009ccf72f4440513fc"
Sep 08 02:26:03.243513 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:26:03.243475    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"408b7e1a69996cdba3bbd4e50e5c190271089f7878fd6c009ccf72f4440513fc"} err="failed to get container status \"408b7e1a69996cdba3bbd4e50e5c190271089f7878fd6c009ccf72f4440513fc\": rpc error: code = NotFound desc = could not find container \"408b7e1a69996cdba3bbd4e50e5c190271089f7878fd6c009ccf72f4440513fc\": container with ID starting with 408b7e1a69996cdba3bbd4e50e5c190271089f7878fd6c009ccf72f4440513fc not found: ID does not exist"
Sep 08 02:26:03.249764 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:26:03.249733    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-b23439/ipp-route-b23439-kserve-59c6cdf559-zlmmv"]
Sep 08 02:26:03.253370 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:26:03.253342    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-b23439/ipp-route-b23439-kserve-59c6cdf559-zlmmv"]
Sep 08 02:26:04.993862 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:26:04.993828    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94501423-47ec-4a6f-957d-8097786b4bc3" path="/var/lib/kubelet/pods/94501423-47ec-4a6f-957d-8097786b4bc3/volumes"
Sep 08 02:26:46.729909 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:26:46.729863    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d4845d-74bfb58d9c-h6b8p"]
Sep 08 02:26:46.730452 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:26:46.730185    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-d4845d-74bfb58d9c-h6b8p" podUID="e9ee4413-722d-4b83-a34c-1f37bf956fa5" containerName="payload-processing" containerID="cri-o://408b60950ff12fe0185f2f85c5672e31fb08738712de3243e6a448be57210e73" gracePeriod=30
Sep 08 02:26:46.734686 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:26:46.734655    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d4845d-79f8f95f4b-2kjbp"]
Sep 08 02:26:46.734947 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:26:46.734901    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d4845d-79f8f95f4b-2kjbp" podUID="28e3bb6b-3ea0-4b98-b4fc-979fad49107c" containerName="payload-pre-processing" containerID="cri-o://53506b17c89e5efa94fb159bef0aeddf2d4c89c00cdc11a1097feba7b678b865" gracePeriod=30
Sep 08 02:26:47.014959 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:26:47.014930    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d4845d-74bfb58d9c-h6b8p"
Sep 08 02:26:47.017844 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:26:47.017818    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d4845d-79f8f95f4b-2kjbp"
Sep 08 02:26:47.160312 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:26:47.160271    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sq455\" (UniqueName: \"kubernetes.io/projected/28e3bb6b-3ea0-4b98-b4fc-979fad49107c-kube-api-access-sq455\") pod \"28e3bb6b-3ea0-4b98-b4fc-979fad49107c\" (UID: \"28e3bb6b-3ea0-4b98-b4fc-979fad49107c\") "
Sep 08 02:26:47.160312 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:26:47.160312    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9ee4413-722d-4b83-a34c-1f37bf956fa5-plugins-config-volume\") pod \"e9ee4413-722d-4b83-a34c-1f37bf956fa5\" (UID: \"e9ee4413-722d-4b83-a34c-1f37bf956fa5\") "
Sep 08 02:26:47.160573 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:26:47.160347    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28e3bb6b-3ea0-4b98-b4fc-979fad49107c-plugins-config-volume\") pod \"28e3bb6b-3ea0-4b98-b4fc-979fad49107c\" (UID: \"28e3bb6b-3ea0-4b98-b4fc-979fad49107c\") "
Sep 08 02:26:47.160573 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:26:47.160417    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d6qbf\" (UniqueName: \"kubernetes.io/projected/e9ee4413-722d-4b83-a34c-1f37bf956fa5-kube-api-access-d6qbf\") pod \"e9ee4413-722d-4b83-a34c-1f37bf956fa5\" (UID: \"e9ee4413-722d-4b83-a34c-1f37bf956fa5\") "
Sep 08 02:26:47.160745 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:26:47.160715    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28e3bb6b-3ea0-4b98-b4fc-979fad49107c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "28e3bb6b-3ea0-4b98-b4fc-979fad49107c" (UID: "28e3bb6b-3ea0-4b98-b4fc-979fad49107c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 02:26:47.160798 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:26:47.160716    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9ee4413-722d-4b83-a34c-1f37bf956fa5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e9ee4413-722d-4b83-a34c-1f37bf956fa5" (UID: "e9ee4413-722d-4b83-a34c-1f37bf956fa5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 02:26:47.162606 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:26:47.162570    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28e3bb6b-3ea0-4b98-b4fc-979fad49107c-kube-api-access-sq455" (OuterVolumeSpecName: "kube-api-access-sq455") pod "28e3bb6b-3ea0-4b98-b4fc-979fad49107c" (UID: "28e3bb6b-3ea0-4b98-b4fc-979fad49107c"). InnerVolumeSpecName "kube-api-access-sq455". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 02:26:47.162724 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:26:47.162604    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9ee4413-722d-4b83-a34c-1f37bf956fa5-kube-api-access-d6qbf" (OuterVolumeSpecName: "kube-api-access-d6qbf") pod "e9ee4413-722d-4b83-a34c-1f37bf956fa5" (UID: "e9ee4413-722d-4b83-a34c-1f37bf956fa5"). InnerVolumeSpecName "kube-api-access-d6qbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 02:26:47.261557 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:26:47.261472    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sq455\" (UniqueName: \"kubernetes.io/projected/28e3bb6b-3ea0-4b98-b4fc-979fad49107c-kube-api-access-sq455\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:26:47.261557 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:26:47.261505    2547 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9ee4413-722d-4b83-a34c-1f37bf956fa5-plugins-config-volume\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:26:47.261557 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:26:47.261514    2547 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28e3bb6b-3ea0-4b98-b4fc-979fad49107c-plugins-config-volume\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:26:47.261557 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:26:47.261525    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6qbf\" (UniqueName: \"kubernetes.io/projected/e9ee4413-722d-4b83-a34c-1f37bf956fa5-kube-api-access-d6qbf\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:26:47.440432 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:26:47.440398    2547 generic.go:356] "Generic (PLEG): container finished" podID="28e3bb6b-3ea0-4b98-b4fc-979fad49107c" containerID="53506b17c89e5efa94fb159bef0aeddf2d4c89c00cdc11a1097feba7b678b865" exitCode=0
Sep 08 02:26:47.440594 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:26:47.440446    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d4845d-79f8f95f4b-2kjbp"
Sep 08 02:26:47.440594 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:26:47.440480    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d4845d-79f8f95f4b-2kjbp" event={"ID":"28e3bb6b-3ea0-4b98-b4fc-979fad49107c","Type":"ContainerDied","Data":"53506b17c89e5efa94fb159bef0aeddf2d4c89c00cdc11a1097feba7b678b865"}
Sep 08 02:26:47.440594 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:26:47.440527    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d4845d-79f8f95f4b-2kjbp" event={"ID":"28e3bb6b-3ea0-4b98-b4fc-979fad49107c","Type":"ContainerDied","Data":"cf7ce758dc85734da74e53d91e14764dece916ab60f35eda46a62925f4db9f7f"}
Sep 08 02:26:47.440594 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:26:47.440548    2547 scope.go:117] "RemoveContainer" containerID="53506b17c89e5efa94fb159bef0aeddf2d4c89c00cdc11a1097feba7b678b865"
Sep 08 02:26:47.441869 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:26:47.441842    2547 generic.go:356] "Generic (PLEG): container finished" podID="e9ee4413-722d-4b83-a34c-1f37bf956fa5" containerID="408b60950ff12fe0185f2f85c5672e31fb08738712de3243e6a448be57210e73" exitCode=0
Sep 08 02:26:47.442018 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:26:47.441870    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d4845d-74bfb58d9c-h6b8p" event={"ID":"e9ee4413-722d-4b83-a34c-1f37bf956fa5","Type":"ContainerDied","Data":"408b60950ff12fe0185f2f85c5672e31fb08738712de3243e6a448be57210e73"}
Sep 08 02:26:47.442018 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:26:47.441903    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d4845d-74bfb58d9c-h6b8p"
Sep 08 02:26:47.442018 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:26:47.441909    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d4845d-74bfb58d9c-h6b8p" event={"ID":"e9ee4413-722d-4b83-a34c-1f37bf956fa5","Type":"ContainerDied","Data":"0b896f5be2bd6a68a6fca512f7e8e546ed43bd453a3a31c0c3415c3981f6f5a4"}
Sep 08 02:26:47.449635 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:26:47.449521    2547 scope.go:117] "RemoveContainer" containerID="53506b17c89e5efa94fb159bef0aeddf2d4c89c00cdc11a1097feba7b678b865"
Sep 08 02:26:47.449902 ip-10-0-142-78 kubenswrapper[2547]: E0908 02:26:47.449879    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53506b17c89e5efa94fb159bef0aeddf2d4c89c00cdc11a1097feba7b678b865\": container with ID starting with 53506b17c89e5efa94fb159bef0aeddf2d4c89c00cdc11a1097feba7b678b865 not found: ID does not exist" containerID="53506b17c89e5efa94fb159bef0aeddf2d4c89c00cdc11a1097feba7b678b865"
Sep 08 02:26:47.449984 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:26:47.449914    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53506b17c89e5efa94fb159bef0aeddf2d4c89c00cdc11a1097feba7b678b865"} err="failed to get container status \"53506b17c89e5efa94fb159bef0aeddf2d4c89c00cdc11a1097feba7b678b865\": rpc error: code = NotFound desc = could not find container \"53506b17c89e5efa94fb159bef0aeddf2d4c89c00cdc11a1097feba7b678b865\": container with ID starting with 53506b17c89e5efa94fb159bef0aeddf2d4c89c00cdc11a1097feba7b678b865 not found: ID does not exist"
Sep 08 02:26:47.449984 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:26:47.449937    2547 scope.go:117] "RemoveContainer" containerID="408b60950ff12fe0185f2f85c5672e31fb08738712de3243e6a448be57210e73"
Sep 08 02:26:47.457982 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:26:47.457959    2547 scope.go:117] "RemoveContainer" containerID="408b60950ff12fe0185f2f85c5672e31fb08738712de3243e6a448be57210e73"
Sep 08 02:26:47.458328 ip-10-0-142-78 kubenswrapper[2547]: E0908 02:26:47.458308    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"408b60950ff12fe0185f2f85c5672e31fb08738712de3243e6a448be57210e73\": container with ID starting with 408b60950ff12fe0185f2f85c5672e31fb08738712de3243e6a448be57210e73 not found: ID does not exist" containerID="408b60950ff12fe0185f2f85c5672e31fb08738712de3243e6a448be57210e73"
Sep 08 02:26:47.458403 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:26:47.458337    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"408b60950ff12fe0185f2f85c5672e31fb08738712de3243e6a448be57210e73"} err="failed to get container status \"408b60950ff12fe0185f2f85c5672e31fb08738712de3243e6a448be57210e73\": rpc error: code = NotFound desc = could not find container \"408b60950ff12fe0185f2f85c5672e31fb08738712de3243e6a448be57210e73\": container with ID starting with 408b60950ff12fe0185f2f85c5672e31fb08738712de3243e6a448be57210e73 not found: ID does not exist"
Sep 08 02:26:47.464356 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:26:47.464334    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d4845d-74bfb58d9c-h6b8p"]
Sep 08 02:26:47.471292 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:26:47.471265    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d4845d-74bfb58d9c-h6b8p"]
Sep 08 02:26:47.481330 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:26:47.481302    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d4845d-79f8f95f4b-2kjbp"]
Sep 08 02:26:47.485295 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:26:47.485269    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d4845d-79f8f95f4b-2kjbp"]
Sep 08 02:26:48.993931 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:26:48.993893    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28e3bb6b-3ea0-4b98-b4fc-979fad49107c" path="/var/lib/kubelet/pods/28e3bb6b-3ea0-4b98-b4fc-979fad49107c/volumes"
Sep 08 02:26:48.994423 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:26:48.994406    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9ee4413-722d-4b83-a34c-1f37bf956fa5" path="/var/lib/kubelet/pods/e9ee4413-722d-4b83-a34c-1f37bf956fa5/volumes"
Sep 08 02:27:04.253537 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:04.253499    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-fcdd46-5488fbfb5-rjwkz"]
Sep 08 02:27:04.254037 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:04.253698    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-fcdd46-5488fbfb5-rjwkz" podUID="cb5bb8a7-37b7-4581-be4c-49892f01db9e" containerName="payload-processing" containerID="cri-o://aedf16d8279776cd25072d0245eefa74907c3978081770ec144648e53d0b005b" gracePeriod=30
Sep 08 02:27:04.264739 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:04.264708    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fcdd46-64c469749d-jc7xm"]
Sep 08 02:27:04.264988 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:04.264957    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fcdd46-64c469749d-jc7xm" podUID="a7b93571-26fc-4e6f-89b5-48926f61dbf8" containerName="payload-pre-processing" containerID="cri-o://f135d722b948270d3184216daaeedf567683ce712639dbc674eea0c06908e2ee" gracePeriod=30
Sep 08 02:27:04.522546 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:04.522510    2547 generic.go:356] "Generic (PLEG): container finished" podID="a7b93571-26fc-4e6f-89b5-48926f61dbf8" containerID="f135d722b948270d3184216daaeedf567683ce712639dbc674eea0c06908e2ee" exitCode=0
Sep 08 02:27:04.522674 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:04.522584    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fcdd46-64c469749d-jc7xm" event={"ID":"a7b93571-26fc-4e6f-89b5-48926f61dbf8","Type":"ContainerDied","Data":"f135d722b948270d3184216daaeedf567683ce712639dbc674eea0c06908e2ee"}
Sep 08 02:27:04.524177 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:04.524154    2547 generic.go:356] "Generic (PLEG): container finished" podID="cb5bb8a7-37b7-4581-be4c-49892f01db9e" containerID="aedf16d8279776cd25072d0245eefa74907c3978081770ec144648e53d0b005b" exitCode=0
Sep 08 02:27:04.524296 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:04.524185    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-fcdd46-5488fbfb5-rjwkz" event={"ID":"cb5bb8a7-37b7-4581-be4c-49892f01db9e","Type":"ContainerDied","Data":"aedf16d8279776cd25072d0245eefa74907c3978081770ec144648e53d0b005b"}
Sep 08 02:27:04.534834 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:04.534813    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-fcdd46-5488fbfb5-rjwkz"
Sep 08 02:27:04.546820 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:04.546796    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fcdd46-64c469749d-jc7xm"
Sep 08 02:27:04.613755 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:04.613719    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z7f7w\" (UniqueName: \"kubernetes.io/projected/cb5bb8a7-37b7-4581-be4c-49892f01db9e-kube-api-access-z7f7w\") pod \"cb5bb8a7-37b7-4581-be4c-49892f01db9e\" (UID: \"cb5bb8a7-37b7-4581-be4c-49892f01db9e\") "
Sep 08 02:27:04.613934 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:04.613773    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb5bb8a7-37b7-4581-be4c-49892f01db9e-plugins-config-volume\") pod \"cb5bb8a7-37b7-4581-be4c-49892f01db9e\" (UID: \"cb5bb8a7-37b7-4581-be4c-49892f01db9e\") "
Sep 08 02:27:04.613934 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:04.613791    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lnvf5\" (UniqueName: \"kubernetes.io/projected/a7b93571-26fc-4e6f-89b5-48926f61dbf8-kube-api-access-lnvf5\") pod \"a7b93571-26fc-4e6f-89b5-48926f61dbf8\" (UID: \"a7b93571-26fc-4e6f-89b5-48926f61dbf8\") "
Sep 08 02:27:04.613934 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:04.613857    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7b93571-26fc-4e6f-89b5-48926f61dbf8-plugins-config-volume\") pod \"a7b93571-26fc-4e6f-89b5-48926f61dbf8\" (UID: \"a7b93571-26fc-4e6f-89b5-48926f61dbf8\") "
Sep 08 02:27:04.614269 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:04.614237    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb5bb8a7-37b7-4581-be4c-49892f01db9e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cb5bb8a7-37b7-4581-be4c-49892f01db9e" (UID: "cb5bb8a7-37b7-4581-be4c-49892f01db9e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 02:27:04.614393 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:04.614305    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7b93571-26fc-4e6f-89b5-48926f61dbf8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a7b93571-26fc-4e6f-89b5-48926f61dbf8" (UID: "a7b93571-26fc-4e6f-89b5-48926f61dbf8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 02:27:04.615929 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:04.615903    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7b93571-26fc-4e6f-89b5-48926f61dbf8-kube-api-access-lnvf5" (OuterVolumeSpecName: "kube-api-access-lnvf5") pod "a7b93571-26fc-4e6f-89b5-48926f61dbf8" (UID: "a7b93571-26fc-4e6f-89b5-48926f61dbf8"). InnerVolumeSpecName "kube-api-access-lnvf5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 02:27:04.616023 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:04.615945    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb5bb8a7-37b7-4581-be4c-49892f01db9e-kube-api-access-z7f7w" (OuterVolumeSpecName: "kube-api-access-z7f7w") pod "cb5bb8a7-37b7-4581-be4c-49892f01db9e" (UID: "cb5bb8a7-37b7-4581-be4c-49892f01db9e"). InnerVolumeSpecName "kube-api-access-z7f7w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 02:27:04.715046 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:04.715004    2547 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7b93571-26fc-4e6f-89b5-48926f61dbf8-plugins-config-volume\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:27:04.715046 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:04.715041    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7f7w\" (UniqueName: \"kubernetes.io/projected/cb5bb8a7-37b7-4581-be4c-49892f01db9e-kube-api-access-z7f7w\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:27:04.715288 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:04.715057    2547 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb5bb8a7-37b7-4581-be4c-49892f01db9e-plugins-config-volume\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:27:04.715288 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:04.715071    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnvf5\" (UniqueName: \"kubernetes.io/projected/a7b93571-26fc-4e6f-89b5-48926f61dbf8-kube-api-access-lnvf5\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:27:05.530148 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:05.530051    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fcdd46-64c469749d-jc7xm"
Sep 08 02:27:05.530593 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:05.530051    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fcdd46-64c469749d-jc7xm" event={"ID":"a7b93571-26fc-4e6f-89b5-48926f61dbf8","Type":"ContainerDied","Data":"cd3ec2ee04edaa97ac94d77f49a9c9e9fb9e088ada4602ccdb5cac1f3d6774e6"}
Sep 08 02:27:05.530593 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:05.530205    2547 scope.go:117] "RemoveContainer" containerID="f135d722b948270d3184216daaeedf567683ce712639dbc674eea0c06908e2ee"
Sep 08 02:27:05.531471 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:05.531446    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-fcdd46-5488fbfb5-rjwkz" event={"ID":"cb5bb8a7-37b7-4581-be4c-49892f01db9e","Type":"ContainerDied","Data":"405f4360748ab43ac1c997a766b92b3e8e7980a31fbc9a717ad01a6bdf8063a6"}
Sep 08 02:27:05.531535 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:05.531488    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-fcdd46-5488fbfb5-rjwkz"
Sep 08 02:27:05.538410 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:05.538385    2547 scope.go:117] "RemoveContainer" containerID="aedf16d8279776cd25072d0245eefa74907c3978081770ec144648e53d0b005b"
Sep 08 02:27:05.551979 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:05.551953    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fcdd46-64c469749d-jc7xm"]
Sep 08 02:27:05.560400 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:05.560370    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fcdd46-64c469749d-jc7xm"]
Sep 08 02:27:05.589762 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:05.589728    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-fcdd46-5488fbfb5-rjwkz"]
Sep 08 02:27:05.591953 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:05.591924    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-fcdd46-5488fbfb5-rjwkz"]
Sep 08 02:27:06.993933 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:06.993894    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7b93571-26fc-4e6f-89b5-48926f61dbf8" path="/var/lib/kubelet/pods/a7b93571-26fc-4e6f-89b5-48926f61dbf8/volumes"
Sep 08 02:27:06.994345 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:06.994273    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb5bb8a7-37b7-4581-be4c-49892f01db9e" path="/var/lib/kubelet/pods/cb5bb8a7-37b7-4581-be4c-49892f01db9e/volumes"
Sep 08 02:27:11.423451 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:11.423367    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-527ac8-6ctt9"]
Sep 08 02:27:11.423822 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:11.423696    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb5bb8a7-37b7-4581-be4c-49892f01db9e" containerName="payload-processing"
Sep 08 02:27:11.423822 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:11.423708    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb5bb8a7-37b7-4581-be4c-49892f01db9e" containerName="payload-processing"
Sep 08 02:27:11.423822 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:11.423723    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9ee4413-722d-4b83-a34c-1f37bf956fa5" containerName="payload-processing"
Sep 08 02:27:11.423822 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:11.423729    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9ee4413-722d-4b83-a34c-1f37bf956fa5" containerName="payload-processing"
Sep 08 02:27:11.423822 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:11.423738    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28e3bb6b-3ea0-4b98-b4fc-979fad49107c" containerName="payload-pre-processing"
Sep 08 02:27:11.423822 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:11.423744    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="28e3bb6b-3ea0-4b98-b4fc-979fad49107c" containerName="payload-pre-processing"
Sep 08 02:27:11.423822 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:11.423751    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ded89ca-283d-4848-900b-9c93239b0fe6" containerName="payload-pre-processing"
Sep 08 02:27:11.423822 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:11.423756    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ded89ca-283d-4848-900b-9c93239b0fe6" containerName="payload-pre-processing"
Sep 08 02:27:11.423822 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:11.423763    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94501423-47ec-4a6f-957d-8097786b4bc3" containerName="main"
Sep 08 02:27:11.423822 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:11.423768    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="94501423-47ec-4a6f-957d-8097786b4bc3" containerName="main"
Sep 08 02:27:11.423822 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:11.423776    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7b93571-26fc-4e6f-89b5-48926f61dbf8" containerName="payload-pre-processing"
Sep 08 02:27:11.423822 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:11.423781    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7b93571-26fc-4e6f-89b5-48926f61dbf8" containerName="payload-pre-processing"
Sep 08 02:27:11.424214 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:11.423832    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="28e3bb6b-3ea0-4b98-b4fc-979fad49107c" containerName="payload-pre-processing"
Sep 08 02:27:11.424214 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:11.423841    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ded89ca-283d-4848-900b-9c93239b0fe6" containerName="payload-pre-processing"
Sep 08 02:27:11.424214 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:11.423847    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9ee4413-722d-4b83-a34c-1f37bf956fa5" containerName="payload-processing"
Sep 08 02:27:11.424214 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:11.423852    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb5bb8a7-37b7-4581-be4c-49892f01db9e" containerName="payload-processing"
Sep 08 02:27:11.424214 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:11.423858    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="94501423-47ec-4a6f-957d-8097786b4bc3" containerName="main"
Sep 08 02:27:11.424214 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:11.423863    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7b93571-26fc-4e6f-89b5-48926f61dbf8" containerName="payload-pre-processing"
Sep 08 02:27:11.428516 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:11.428496    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-527ac8-6ctt9"
Sep 08 02:27:11.431267 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:11.431241    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 02:27:11.431411 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:11.431341    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8mbwc\""
Sep 08 02:27:11.434548 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:11.434520    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-527ac8-6ctt9"]
Sep 08 02:27:11.462367 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:11.462334    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79982c52-0f40-42bd-9467-36b7c7358dfa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-527ac8-6ctt9\" (UID: \"79982c52-0f40-42bd-9467-36b7c7358dfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-527ac8-6ctt9"
Sep 08 02:27:11.563169 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:11.563116    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79982c52-0f40-42bd-9467-36b7c7358dfa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-527ac8-6ctt9\" (UID: \"79982c52-0f40-42bd-9467-36b7c7358dfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-527ac8-6ctt9"
Sep 08 02:27:11.563765 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:11.563745    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79982c52-0f40-42bd-9467-36b7c7358dfa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-527ac8-6ctt9\" (UID: \"79982c52-0f40-42bd-9467-36b7c7358dfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-527ac8-6ctt9"
Sep 08 02:27:11.739672 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:11.739582    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-527ac8-6ctt9"
Sep 08 02:27:12.022985 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:12.022957    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-527ac8-6ctt9"]
Sep 08 02:27:12.025661 ip-10-0-142-78 kubenswrapper[2547]: W0908 02:27:12.025628    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79982c52_0f40_42bd_9467_36b7c7358dfa.slice/crio-364ee6849fb42b59119d32d1ec61d312cbff432fb0e195fde02567713dd6000f WatchSource:0}: Error finding container 364ee6849fb42b59119d32d1ec61d312cbff432fb0e195fde02567713dd6000f: Status 404 returned error can't find the container with id 364ee6849fb42b59119d32d1ec61d312cbff432fb0e195fde02567713dd6000f
Sep 08 02:27:12.566728 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:12.566692    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-527ac8-6ctt9" event={"ID":"79982c52-0f40-42bd-9467-36b7c7358dfa","Type":"ContainerStarted","Data":"1a2bf2071e5feb23445de5b99cf3ab7ec0547641a2625f02930f86f044fabe9d"}
Sep 08 02:27:12.567102 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:12.566747    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-527ac8-6ctt9" event={"ID":"79982c52-0f40-42bd-9467-36b7c7358dfa","Type":"ContainerStarted","Data":"364ee6849fb42b59119d32d1ec61d312cbff432fb0e195fde02567713dd6000f"}
Sep 08 02:27:12.586063 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:12.586003    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-527ac8-6ctt9" podStartSLOduration=1.5859846069999999 podStartE2EDuration="1.585984607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:27:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:27:12.585036236 +0000 UTC m=+1704.108811315" watchObservedRunningTime="2026-09-08 02:27:12.585984607 +0000 UTC m=+1704.109759687"
Sep 08 02:27:13.572046 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:13.572006    2547 generic.go:356] "Generic (PLEG): container finished" podID="79982c52-0f40-42bd-9467-36b7c7358dfa" containerID="1a2bf2071e5feb23445de5b99cf3ab7ec0547641a2625f02930f86f044fabe9d" exitCode=0
Sep 08 02:27:13.572515 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:13.572063    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-527ac8-6ctt9" event={"ID":"79982c52-0f40-42bd-9467-36b7c7358dfa","Type":"ContainerDied","Data":"1a2bf2071e5feb23445de5b99cf3ab7ec0547641a2625f02930f86f044fabe9d"}
Sep 08 02:27:14.709514 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:14.709486    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-527ac8-6ctt9"
Sep 08 02:27:14.784054 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:14.784012    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79982c52-0f40-42bd-9467-36b7c7358dfa-maas-api-service-ca\") pod \"79982c52-0f40-42bd-9467-36b7c7358dfa\" (UID: \"79982c52-0f40-42bd-9467-36b7c7358dfa\") "
Sep 08 02:27:14.784435 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:14.784411    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79982c52-0f40-42bd-9467-36b7c7358dfa-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "79982c52-0f40-42bd-9467-36b7c7358dfa" (UID: "79982c52-0f40-42bd-9467-36b7c7358dfa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 02:27:14.885185 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:14.885065    2547 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79982c52-0f40-42bd-9467-36b7c7358dfa-maas-api-service-ca\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:27:15.582271 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:15.582233    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-527ac8-6ctt9" event={"ID":"79982c52-0f40-42bd-9467-36b7c7358dfa","Type":"ContainerDied","Data":"364ee6849fb42b59119d32d1ec61d312cbff432fb0e195fde02567713dd6000f"}
Sep 08 02:27:15.582271 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:15.582269    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-527ac8-6ctt9"
Sep 08 02:27:15.582491 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:15.582311    2547 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="364ee6849fb42b59119d32d1ec61d312cbff432fb0e195fde02567713dd6000f"
Sep 08 02:27:22.010568 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:22.010520    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-527ac8-7cc5bfb4d9-hzk4w"]
Sep 08 02:27:22.011008 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:22.010796    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-527ac8-7cc5bfb4d9-hzk4w" podUID="5ef60051-508f-41e4-9293-d4d04b7f47a5" containerName="payload-processing" containerID="cri-o://c04363fbbb9a358ba8295e7bb0518aca221d4c3a7fd1c62a445a42b776044331" gracePeriod=30
Sep 08 02:27:22.015995 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:22.015941    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-527ac8-5fb9794fb6-rhqvt"]
Sep 08 02:27:22.016248 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:22.016218    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-527ac8-5fb9794fb6-rhqvt" podUID="8bedf1b4-79ee-40a3-aeac-28908af7f972" containerName="payload-pre-processing" containerID="cri-o://77c575680def701a662b6547c4175667568fb1f14238fa72c105618c8f7d888d" gracePeriod=30
Sep 08 02:27:22.192216 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:22.192182    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-527ac8-6ctt9"]
Sep 08 02:27:22.195128 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:22.195101    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-527ac8-6ctt9"]
Sep 08 02:27:22.289006 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:22.288975    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-527ac8-7cc5bfb4d9-hzk4w"
Sep 08 02:27:22.291573 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:22.291551    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-527ac8-5fb9794fb6-rhqvt"
Sep 08 02:27:22.343009 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:22.342978    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6b57m\" (UniqueName: \"kubernetes.io/projected/8bedf1b4-79ee-40a3-aeac-28908af7f972-kube-api-access-6b57m\") pod \"8bedf1b4-79ee-40a3-aeac-28908af7f972\" (UID: \"8bedf1b4-79ee-40a3-aeac-28908af7f972\") "
Sep 08 02:27:22.343204 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:22.343029    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bedf1b4-79ee-40a3-aeac-28908af7f972-plugins-config-volume\") pod \"8bedf1b4-79ee-40a3-aeac-28908af7f972\" (UID: \"8bedf1b4-79ee-40a3-aeac-28908af7f972\") "
Sep 08 02:27:22.343204 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:22.343050    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ef60051-508f-41e4-9293-d4d04b7f47a5-plugins-config-volume\") pod \"5ef60051-508f-41e4-9293-d4d04b7f47a5\" (UID: \"5ef60051-508f-41e4-9293-d4d04b7f47a5\") "
Sep 08 02:27:22.343204 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:22.343109    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wl9lb\" (UniqueName: \"kubernetes.io/projected/5ef60051-508f-41e4-9293-d4d04b7f47a5-kube-api-access-wl9lb\") pod \"5ef60051-508f-41e4-9293-d4d04b7f47a5\" (UID: \"5ef60051-508f-41e4-9293-d4d04b7f47a5\") "
Sep 08 02:27:22.343464 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:22.343430    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bedf1b4-79ee-40a3-aeac-28908af7f972-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8bedf1b4-79ee-40a3-aeac-28908af7f972" (UID: "8bedf1b4-79ee-40a3-aeac-28908af7f972"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 02:27:22.343544 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:22.343516    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ef60051-508f-41e4-9293-d4d04b7f47a5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5ef60051-508f-41e4-9293-d4d04b7f47a5" (UID: "5ef60051-508f-41e4-9293-d4d04b7f47a5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 02:27:22.345349 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:22.345324    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ef60051-508f-41e4-9293-d4d04b7f47a5-kube-api-access-wl9lb" (OuterVolumeSpecName: "kube-api-access-wl9lb") pod "5ef60051-508f-41e4-9293-d4d04b7f47a5" (UID: "5ef60051-508f-41e4-9293-d4d04b7f47a5"). InnerVolumeSpecName "kube-api-access-wl9lb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 02:27:22.345430 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:22.345350    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bedf1b4-79ee-40a3-aeac-28908af7f972-kube-api-access-6b57m" (OuterVolumeSpecName: "kube-api-access-6b57m") pod "8bedf1b4-79ee-40a3-aeac-28908af7f972" (UID: "8bedf1b4-79ee-40a3-aeac-28908af7f972"). InnerVolumeSpecName "kube-api-access-6b57m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 02:27:22.443865 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:22.443822    2547 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bedf1b4-79ee-40a3-aeac-28908af7f972-plugins-config-volume\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:27:22.443865 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:22.443855    2547 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ef60051-508f-41e4-9293-d4d04b7f47a5-plugins-config-volume\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:27:22.443865 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:22.443871    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wl9lb\" (UniqueName: \"kubernetes.io/projected/5ef60051-508f-41e4-9293-d4d04b7f47a5-kube-api-access-wl9lb\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:27:22.444091 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:22.443880    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6b57m\" (UniqueName: \"kubernetes.io/projected/8bedf1b4-79ee-40a3-aeac-28908af7f972-kube-api-access-6b57m\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:27:22.620098 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:22.620012    2547 generic.go:356] "Generic (PLEG): container finished" podID="5ef60051-508f-41e4-9293-d4d04b7f47a5" containerID="c04363fbbb9a358ba8295e7bb0518aca221d4c3a7fd1c62a445a42b776044331" exitCode=0
Sep 08 02:27:22.620098 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:22.620063    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-527ac8-7cc5bfb4d9-hzk4w"
Sep 08 02:27:22.620321 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:22.620094    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-527ac8-7cc5bfb4d9-hzk4w" event={"ID":"5ef60051-508f-41e4-9293-d4d04b7f47a5","Type":"ContainerDied","Data":"c04363fbbb9a358ba8295e7bb0518aca221d4c3a7fd1c62a445a42b776044331"}
Sep 08 02:27:22.620321 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:22.620201    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-527ac8-7cc5bfb4d9-hzk4w" event={"ID":"5ef60051-508f-41e4-9293-d4d04b7f47a5","Type":"ContainerDied","Data":"4a647b31afc1f0d7b2ccd125eedab9fb3011fb76580835bdcebc4e964108f772"}
Sep 08 02:27:22.620321 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:22.620229    2547 scope.go:117] "RemoveContainer" containerID="c04363fbbb9a358ba8295e7bb0518aca221d4c3a7fd1c62a445a42b776044331"
Sep 08 02:27:22.621500 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:22.621476    2547 generic.go:356] "Generic (PLEG): container finished" podID="8bedf1b4-79ee-40a3-aeac-28908af7f972" containerID="77c575680def701a662b6547c4175667568fb1f14238fa72c105618c8f7d888d" exitCode=0
Sep 08 02:27:22.621588 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:22.621528    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-527ac8-5fb9794fb6-rhqvt" event={"ID":"8bedf1b4-79ee-40a3-aeac-28908af7f972","Type":"ContainerDied","Data":"77c575680def701a662b6547c4175667568fb1f14238fa72c105618c8f7d888d"}
Sep 08 02:27:22.621588 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:22.621563    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-527ac8-5fb9794fb6-rhqvt" event={"ID":"8bedf1b4-79ee-40a3-aeac-28908af7f972","Type":"ContainerDied","Data":"275f3a00898a11647f40d1e17747c26c52654e5e1e826d97b9a0befbd3c62e69"}
Sep 08 02:27:22.621588 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:22.621530    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-527ac8-5fb9794fb6-rhqvt"
Sep 08 02:27:22.629754 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:22.629725    2547 scope.go:117] "RemoveContainer" containerID="c04363fbbb9a358ba8295e7bb0518aca221d4c3a7fd1c62a445a42b776044331"
Sep 08 02:27:22.630210 ip-10-0-142-78 kubenswrapper[2547]: E0908 02:27:22.630174    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c04363fbbb9a358ba8295e7bb0518aca221d4c3a7fd1c62a445a42b776044331\": container with ID starting with c04363fbbb9a358ba8295e7bb0518aca221d4c3a7fd1c62a445a42b776044331 not found: ID does not exist" containerID="c04363fbbb9a358ba8295e7bb0518aca221d4c3a7fd1c62a445a42b776044331"
Sep 08 02:27:22.630273 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:22.630212    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c04363fbbb9a358ba8295e7bb0518aca221d4c3a7fd1c62a445a42b776044331"} err="failed to get container status \"c04363fbbb9a358ba8295e7bb0518aca221d4c3a7fd1c62a445a42b776044331\": rpc error: code = NotFound desc = could not find container \"c04363fbbb9a358ba8295e7bb0518aca221d4c3a7fd1c62a445a42b776044331\": container with ID starting with c04363fbbb9a358ba8295e7bb0518aca221d4c3a7fd1c62a445a42b776044331 not found: ID does not exist"
Sep 08 02:27:22.630273 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:22.630238    2547 scope.go:117] "RemoveContainer" containerID="77c575680def701a662b6547c4175667568fb1f14238fa72c105618c8f7d888d"
Sep 08 02:27:22.638039 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:22.638013    2547 scope.go:117] "RemoveContainer" containerID="77c575680def701a662b6547c4175667568fb1f14238fa72c105618c8f7d888d"
Sep 08 02:27:22.638352 ip-10-0-142-78 kubenswrapper[2547]: E0908 02:27:22.638333    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77c575680def701a662b6547c4175667568fb1f14238fa72c105618c8f7d888d\": container with ID starting with 77c575680def701a662b6547c4175667568fb1f14238fa72c105618c8f7d888d not found: ID does not exist" containerID="77c575680def701a662b6547c4175667568fb1f14238fa72c105618c8f7d888d"
Sep 08 02:27:22.638418 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:22.638358    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77c575680def701a662b6547c4175667568fb1f14238fa72c105618c8f7d888d"} err="failed to get container status \"77c575680def701a662b6547c4175667568fb1f14238fa72c105618c8f7d888d\": rpc error: code = NotFound desc = could not find container \"77c575680def701a662b6547c4175667568fb1f14238fa72c105618c8f7d888d\": container with ID starting with 77c575680def701a662b6547c4175667568fb1f14238fa72c105618c8f7d888d not found: ID does not exist"
Sep 08 02:27:22.648500 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:22.648472    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-527ac8-5fb9794fb6-rhqvt"]
Sep 08 02:27:22.651851 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:22.651821    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-527ac8-5fb9794fb6-rhqvt"]
Sep 08 02:27:22.675039 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:22.674989    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-527ac8-7cc5bfb4d9-hzk4w"]
Sep 08 02:27:22.677851 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:22.677820    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-527ac8-7cc5bfb4d9-hzk4w"]
Sep 08 02:27:22.994240 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:22.994198    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ef60051-508f-41e4-9293-d4d04b7f47a5" path="/var/lib/kubelet/pods/5ef60051-508f-41e4-9293-d4d04b7f47a5/volumes"
Sep 08 02:27:22.994594 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:22.994576    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79982c52-0f40-42bd-9467-36b7c7358dfa" path="/var/lib/kubelet/pods/79982c52-0f40-42bd-9467-36b7c7358dfa/volumes"
Sep 08 02:27:22.994961 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:27:22.994943    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8bedf1b4-79ee-40a3-aeac-28908af7f972" path="/var/lib/kubelet/pods/8bedf1b4-79ee-40a3-aeac-28908af7f972/volumes"
Sep 08 02:31:51.423842 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:31:51.423752    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-a5f19aab-kserve-c4d6fdb79-dltgw"]
Sep 08 02:31:51.424701 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:31:51.424034    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ef60051-508f-41e4-9293-d4d04b7f47a5" containerName="payload-processing"
Sep 08 02:31:51.424701 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:31:51.424045    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ef60051-508f-41e4-9293-d4d04b7f47a5" containerName="payload-processing"
Sep 08 02:31:51.424701 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:31:51.424062    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79982c52-0f40-42bd-9467-36b7c7358dfa" containerName="revoke-keys"
Sep 08 02:31:51.424701 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:31:51.424067    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="79982c52-0f40-42bd-9467-36b7c7358dfa" containerName="revoke-keys"
Sep 08 02:31:51.424701 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:31:51.424079    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bedf1b4-79ee-40a3-aeac-28908af7f972" containerName="payload-pre-processing"
Sep 08 02:31:51.424701 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:31:51.424085    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bedf1b4-79ee-40a3-aeac-28908af7f972" containerName="payload-pre-processing"
Sep 08 02:31:51.424701 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:31:51.424172    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bedf1b4-79ee-40a3-aeac-28908af7f972" containerName="payload-pre-processing"
Sep 08 02:31:51.424701 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:31:51.424182    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ef60051-508f-41e4-9293-d4d04b7f47a5" containerName="payload-processing"
Sep 08 02:31:51.424701 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:31:51.424187    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="79982c52-0f40-42bd-9467-36b7c7358dfa" containerName="revoke-keys"
Sep 08 02:31:51.427034 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:31:51.427013    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-a5f19aab-kserve-c4d6fdb79-dltgw"
Sep 08 02:31:51.429330 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:31:51.429309    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-a5f19aab-kserve-self-signed-certs\""
Sep 08 02:31:51.436615 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:31:51.436587    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-a5f19aab-kserve-c4d6fdb79-dltgw"]
Sep 08 02:31:51.510925 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:31:51.510881    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ae6a9c3-b15c-439f-9364-c9f38c3fb360-dshm\") pod \"e2e-same-id-a-a5f19aab-kserve-c4d6fdb79-dltgw\" (UID: \"0ae6a9c3-b15c-439f-9364-c9f38c3fb360\") " pod="llm/e2e-same-id-a-a5f19aab-kserve-c4d6fdb79-dltgw"
Sep 08 02:31:51.510925 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:31:51.510924    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ae6a9c3-b15c-439f-9364-c9f38c3fb360-home\") pod \"e2e-same-id-a-a5f19aab-kserve-c4d6fdb79-dltgw\" (UID: \"0ae6a9c3-b15c-439f-9364-c9f38c3fb360\") " pod="llm/e2e-same-id-a-a5f19aab-kserve-c4d6fdb79-dltgw"
Sep 08 02:31:51.511180 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:31:51.510951    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ae6a9c3-b15c-439f-9364-c9f38c3fb360-tmp-dir\") pod \"e2e-same-id-a-a5f19aab-kserve-c4d6fdb79-dltgw\" (UID: \"0ae6a9c3-b15c-439f-9364-c9f38c3fb360\") " pod="llm/e2e-same-id-a-a5f19aab-kserve-c4d6fdb79-dltgw"
Sep 08 02:31:51.511180 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:31:51.511004    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ae6a9c3-b15c-439f-9364-c9f38c3fb360-model-cache\") pod \"e2e-same-id-a-a5f19aab-kserve-c4d6fdb79-dltgw\" (UID: \"0ae6a9c3-b15c-439f-9364-c9f38c3fb360\") " pod="llm/e2e-same-id-a-a5f19aab-kserve-c4d6fdb79-dltgw"
Sep 08 02:31:51.511180 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:31:51.511027    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ntgk\" (UniqueName: \"kubernetes.io/projected/0ae6a9c3-b15c-439f-9364-c9f38c3fb360-kube-api-access-8ntgk\") pod \"e2e-same-id-a-a5f19aab-kserve-c4d6fdb79-dltgw\" (UID: \"0ae6a9c3-b15c-439f-9364-c9f38c3fb360\") " pod="llm/e2e-same-id-a-a5f19aab-kserve-c4d6fdb79-dltgw"
Sep 08 02:31:51.511180 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:31:51.511053    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ae6a9c3-b15c-439f-9364-c9f38c3fb360-tls-certs\") pod \"e2e-same-id-a-a5f19aab-kserve-c4d6fdb79-dltgw\" (UID: \"0ae6a9c3-b15c-439f-9364-c9f38c3fb360\") " pod="llm/e2e-same-id-a-a5f19aab-kserve-c4d6fdb79-dltgw"
Sep 08 02:31:51.611557 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:31:51.611520    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ntgk\" (UniqueName: \"kubernetes.io/projected/0ae6a9c3-b15c-439f-9364-c9f38c3fb360-kube-api-access-8ntgk\") pod \"e2e-same-id-a-a5f19aab-kserve-c4d6fdb79-dltgw\" (UID: \"0ae6a9c3-b15c-439f-9364-c9f38c3fb360\") " pod="llm/e2e-same-id-a-a5f19aab-kserve-c4d6fdb79-dltgw"
Sep 08 02:31:51.611557 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:31:51.611564    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ae6a9c3-b15c-439f-9364-c9f38c3fb360-tls-certs\") pod \"e2e-same-id-a-a5f19aab-kserve-c4d6fdb79-dltgw\" (UID: \"0ae6a9c3-b15c-439f-9364-c9f38c3fb360\") " pod="llm/e2e-same-id-a-a5f19aab-kserve-c4d6fdb79-dltgw"
Sep 08 02:31:51.611790 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:31:51.611602    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ae6a9c3-b15c-439f-9364-c9f38c3fb360-dshm\") pod \"e2e-same-id-a-a5f19aab-kserve-c4d6fdb79-dltgw\" (UID: \"0ae6a9c3-b15c-439f-9364-c9f38c3fb360\") " pod="llm/e2e-same-id-a-a5f19aab-kserve-c4d6fdb79-dltgw"
Sep 08 02:31:51.611790 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:31:51.611618    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ae6a9c3-b15c-439f-9364-c9f38c3fb360-home\") pod \"e2e-same-id-a-a5f19aab-kserve-c4d6fdb79-dltgw\" (UID: \"0ae6a9c3-b15c-439f-9364-c9f38c3fb360\") " pod="llm/e2e-same-id-a-a5f19aab-kserve-c4d6fdb79-dltgw"
Sep 08 02:31:51.611790 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:31:51.611637    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ae6a9c3-b15c-439f-9364-c9f38c3fb360-tmp-dir\") pod \"e2e-same-id-a-a5f19aab-kserve-c4d6fdb79-dltgw\" (UID: \"0ae6a9c3-b15c-439f-9364-c9f38c3fb360\") " pod="llm/e2e-same-id-a-a5f19aab-kserve-c4d6fdb79-dltgw"
Sep 08 02:31:51.611790 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:31:51.611750    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ae6a9c3-b15c-439f-9364-c9f38c3fb360-model-cache\") pod \"e2e-same-id-a-a5f19aab-kserve-c4d6fdb79-dltgw\" (UID: \"0ae6a9c3-b15c-439f-9364-c9f38c3fb360\") " pod="llm/e2e-same-id-a-a5f19aab-kserve-c4d6fdb79-dltgw"
Sep 08 02:31:51.612071 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:31:51.612049    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ae6a9c3-b15c-439f-9364-c9f38c3fb360-home\") pod \"e2e-same-id-a-a5f19aab-kserve-c4d6fdb79-dltgw\" (UID: \"0ae6a9c3-b15c-439f-9364-c9f38c3fb360\") " pod="llm/e2e-same-id-a-a5f19aab-kserve-c4d6fdb79-dltgw"
Sep 08 02:31:51.612071 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:31:51.612067    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ae6a9c3-b15c-439f-9364-c9f38c3fb360-tmp-dir\") pod \"e2e-same-id-a-a5f19aab-kserve-c4d6fdb79-dltgw\" (UID: \"0ae6a9c3-b15c-439f-9364-c9f38c3fb360\") " pod="llm/e2e-same-id-a-a5f19aab-kserve-c4d6fdb79-dltgw"
Sep 08 02:31:51.612213 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:31:51.612195    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ae6a9c3-b15c-439f-9364-c9f38c3fb360-model-cache\") pod \"e2e-same-id-a-a5f19aab-kserve-c4d6fdb79-dltgw\" (UID: \"0ae6a9c3-b15c-439f-9364-c9f38c3fb360\") " pod="llm/e2e-same-id-a-a5f19aab-kserve-c4d6fdb79-dltgw"
Sep 08 02:31:51.613946 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:31:51.613923    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ae6a9c3-b15c-439f-9364-c9f38c3fb360-dshm\") pod \"e2e-same-id-a-a5f19aab-kserve-c4d6fdb79-dltgw\" (UID: \"0ae6a9c3-b15c-439f-9364-c9f38c3fb360\") " pod="llm/e2e-same-id-a-a5f19aab-kserve-c4d6fdb79-dltgw"
Sep 08 02:31:51.614223 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:31:51.614207    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ae6a9c3-b15c-439f-9364-c9f38c3fb360-tls-certs\") pod \"e2e-same-id-a-a5f19aab-kserve-c4d6fdb79-dltgw\" (UID: \"0ae6a9c3-b15c-439f-9364-c9f38c3fb360\") " pod="llm/e2e-same-id-a-a5f19aab-kserve-c4d6fdb79-dltgw"
Sep 08 02:31:51.618524 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:31:51.618500    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ntgk\" (UniqueName: \"kubernetes.io/projected/0ae6a9c3-b15c-439f-9364-c9f38c3fb360-kube-api-access-8ntgk\") pod \"e2e-same-id-a-a5f19aab-kserve-c4d6fdb79-dltgw\" (UID: \"0ae6a9c3-b15c-439f-9364-c9f38c3fb360\") " pod="llm/e2e-same-id-a-a5f19aab-kserve-c4d6fdb79-dltgw"
Sep 08 02:31:51.736303 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:31:51.736201    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-a5f19aab-kserve-c4d6fdb79-dltgw"
Sep 08 02:31:52.007246 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:31:52.007218    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-a5f19aab-kserve-c4d6fdb79-dltgw"]
Sep 08 02:31:52.009560 ip-10-0-142-78 kubenswrapper[2547]: W0908 02:31:52.009530    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0ae6a9c3_b15c_439f_9364_c9f38c3fb360.slice/crio-237d9060e173815c35522e5d2bed36606072b3733eeaf58f67a883c7483262cf WatchSource:0}: Error finding container 237d9060e173815c35522e5d2bed36606072b3733eeaf58f67a883c7483262cf: Status 404 returned error can't find the container with id 237d9060e173815c35522e5d2bed36606072b3733eeaf58f67a883c7483262cf
Sep 08 02:31:52.757590 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:31:52.757547    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-a5f19aab-kserve-c4d6fdb79-dltgw" event={"ID":"0ae6a9c3-b15c-439f-9364-c9f38c3fb360","Type":"ContainerStarted","Data":"ae8cf20d61501b0eed45847e29da49e9c894a7da82c3d705abefeedd7e3d8cba"}
Sep 08 02:31:52.757590 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:31:52.757597    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-a5f19aab-kserve-c4d6fdb79-dltgw" event={"ID":"0ae6a9c3-b15c-439f-9364-c9f38c3fb360","Type":"ContainerStarted","Data":"237d9060e173815c35522e5d2bed36606072b3733eeaf58f67a883c7483262cf"}
Sep 08 02:31:52.773746 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:31:52.773682    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-a5f19aab-kserve-c4d6fdb79-dltgw" podStartSLOduration=1.773665184 podStartE2EDuration="1.773665184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:31:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:31:52.772869766 +0000 UTC m=+1984.296644847" watchObservedRunningTime="2026-09-08 02:31:52.773665184 +0000 UTC m=+1984.297440266"
Sep 08 02:31:59.915191 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:31:59.915156    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-a5f19aab-kserve-8bc96b7d7-2h6fm"]
Sep 08 02:31:59.918538 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:31:59.918515    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-a5f19aab-kserve-8bc96b7d7-2h6fm"
Sep 08 02:31:59.920976 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:31:59.920957    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-a5f19aab-kserve-self-signed-certs\""
Sep 08 02:31:59.928534 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:31:59.928506    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-a5f19aab-kserve-8bc96b7d7-2h6fm"]
Sep 08 02:31:59.973019 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:31:59.972983    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eba28cd5-8b79-4601-be5b-287072ca4d2f-home\") pod \"e2e-same-id-b-a5f19aab-kserve-8bc96b7d7-2h6fm\" (UID: \"eba28cd5-8b79-4601-be5b-287072ca4d2f\") " pod="llm/e2e-same-id-b-a5f19aab-kserve-8bc96b7d7-2h6fm"
Sep 08 02:31:59.973019 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:31:59.973020    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfqn8\" (UniqueName: \"kubernetes.io/projected/eba28cd5-8b79-4601-be5b-287072ca4d2f-kube-api-access-jfqn8\") pod \"e2e-same-id-b-a5f19aab-kserve-8bc96b7d7-2h6fm\" (UID: \"eba28cd5-8b79-4601-be5b-287072ca4d2f\") " pod="llm/e2e-same-id-b-a5f19aab-kserve-8bc96b7d7-2h6fm"
Sep 08 02:31:59.973243 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:31:59.973094    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eba28cd5-8b79-4601-be5b-287072ca4d2f-model-cache\") pod \"e2e-same-id-b-a5f19aab-kserve-8bc96b7d7-2h6fm\" (UID: \"eba28cd5-8b79-4601-be5b-287072ca4d2f\") " pod="llm/e2e-same-id-b-a5f19aab-kserve-8bc96b7d7-2h6fm"
Sep 08 02:31:59.973243 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:31:59.973156    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eba28cd5-8b79-4601-be5b-287072ca4d2f-tmp-dir\") pod \"e2e-same-id-b-a5f19aab-kserve-8bc96b7d7-2h6fm\" (UID: \"eba28cd5-8b79-4601-be5b-287072ca4d2f\") " pod="llm/e2e-same-id-b-a5f19aab-kserve-8bc96b7d7-2h6fm"
Sep 08 02:31:59.973243 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:31:59.973178    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eba28cd5-8b79-4601-be5b-287072ca4d2f-tls-certs\") pod \"e2e-same-id-b-a5f19aab-kserve-8bc96b7d7-2h6fm\" (UID: \"eba28cd5-8b79-4601-be5b-287072ca4d2f\") " pod="llm/e2e-same-id-b-a5f19aab-kserve-8bc96b7d7-2h6fm"
Sep 08 02:31:59.973243 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:31:59.973217    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eba28cd5-8b79-4601-be5b-287072ca4d2f-dshm\") pod \"e2e-same-id-b-a5f19aab-kserve-8bc96b7d7-2h6fm\" (UID: \"eba28cd5-8b79-4601-be5b-287072ca4d2f\") " pod="llm/e2e-same-id-b-a5f19aab-kserve-8bc96b7d7-2h6fm"
Sep 08 02:32:00.073955 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:00.073911    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eba28cd5-8b79-4601-be5b-287072ca4d2f-tmp-dir\") pod \"e2e-same-id-b-a5f19aab-kserve-8bc96b7d7-2h6fm\" (UID: \"eba28cd5-8b79-4601-be5b-287072ca4d2f\") " pod="llm/e2e-same-id-b-a5f19aab-kserve-8bc96b7d7-2h6fm"
Sep 08 02:32:00.073955 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:00.073953    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eba28cd5-8b79-4601-be5b-287072ca4d2f-tls-certs\") pod \"e2e-same-id-b-a5f19aab-kserve-8bc96b7d7-2h6fm\" (UID: \"eba28cd5-8b79-4601-be5b-287072ca4d2f\") " pod="llm/e2e-same-id-b-a5f19aab-kserve-8bc96b7d7-2h6fm"
Sep 08 02:32:00.074244 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:00.073983    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eba28cd5-8b79-4601-be5b-287072ca4d2f-dshm\") pod \"e2e-same-id-b-a5f19aab-kserve-8bc96b7d7-2h6fm\" (UID: \"eba28cd5-8b79-4601-be5b-287072ca4d2f\") " pod="llm/e2e-same-id-b-a5f19aab-kserve-8bc96b7d7-2h6fm"
Sep 08 02:32:00.074244 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:00.074029    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eba28cd5-8b79-4601-be5b-287072ca4d2f-home\") pod \"e2e-same-id-b-a5f19aab-kserve-8bc96b7d7-2h6fm\" (UID: \"eba28cd5-8b79-4601-be5b-287072ca4d2f\") " pod="llm/e2e-same-id-b-a5f19aab-kserve-8bc96b7d7-2h6fm"
Sep 08 02:32:00.074244 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:00.074047    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfqn8\" (UniqueName: \"kubernetes.io/projected/eba28cd5-8b79-4601-be5b-287072ca4d2f-kube-api-access-jfqn8\") pod \"e2e-same-id-b-a5f19aab-kserve-8bc96b7d7-2h6fm\" (UID: \"eba28cd5-8b79-4601-be5b-287072ca4d2f\") " pod="llm/e2e-same-id-b-a5f19aab-kserve-8bc96b7d7-2h6fm"
Sep 08 02:32:00.074244 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:00.074103    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eba28cd5-8b79-4601-be5b-287072ca4d2f-model-cache\") pod \"e2e-same-id-b-a5f19aab-kserve-8bc96b7d7-2h6fm\" (UID: \"eba28cd5-8b79-4601-be5b-287072ca4d2f\") " pod="llm/e2e-same-id-b-a5f19aab-kserve-8bc96b7d7-2h6fm"
Sep 08 02:32:00.074488 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:00.074457    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eba28cd5-8b79-4601-be5b-287072ca4d2f-home\") pod \"e2e-same-id-b-a5f19aab-kserve-8bc96b7d7-2h6fm\" (UID: \"eba28cd5-8b79-4601-be5b-287072ca4d2f\") " pod="llm/e2e-same-id-b-a5f19aab-kserve-8bc96b7d7-2h6fm"
Sep 08 02:32:00.074488 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:00.074482    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eba28cd5-8b79-4601-be5b-287072ca4d2f-tmp-dir\") pod \"e2e-same-id-b-a5f19aab-kserve-8bc96b7d7-2h6fm\" (UID: \"eba28cd5-8b79-4601-be5b-287072ca4d2f\") " pod="llm/e2e-same-id-b-a5f19aab-kserve-8bc96b7d7-2h6fm"
Sep 08 02:32:00.074697 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:00.074665    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eba28cd5-8b79-4601-be5b-287072ca4d2f-model-cache\") pod \"e2e-same-id-b-a5f19aab-kserve-8bc96b7d7-2h6fm\" (UID: \"eba28cd5-8b79-4601-be5b-287072ca4d2f\") " pod="llm/e2e-same-id-b-a5f19aab-kserve-8bc96b7d7-2h6fm"
Sep 08 02:32:00.076434 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:00.076413    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eba28cd5-8b79-4601-be5b-287072ca4d2f-dshm\") pod \"e2e-same-id-b-a5f19aab-kserve-8bc96b7d7-2h6fm\" (UID: \"eba28cd5-8b79-4601-be5b-287072ca4d2f\") " pod="llm/e2e-same-id-b-a5f19aab-kserve-8bc96b7d7-2h6fm"
Sep 08 02:32:00.076517 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:00.076482    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eba28cd5-8b79-4601-be5b-287072ca4d2f-tls-certs\") pod \"e2e-same-id-b-a5f19aab-kserve-8bc96b7d7-2h6fm\" (UID: \"eba28cd5-8b79-4601-be5b-287072ca4d2f\") " pod="llm/e2e-same-id-b-a5f19aab-kserve-8bc96b7d7-2h6fm"
Sep 08 02:32:00.083082 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:00.083059    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfqn8\" (UniqueName: \"kubernetes.io/projected/eba28cd5-8b79-4601-be5b-287072ca4d2f-kube-api-access-jfqn8\") pod \"e2e-same-id-b-a5f19aab-kserve-8bc96b7d7-2h6fm\" (UID: \"eba28cd5-8b79-4601-be5b-287072ca4d2f\") " pod="llm/e2e-same-id-b-a5f19aab-kserve-8bc96b7d7-2h6fm"
Sep 08 02:32:00.229044 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:00.228957    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-a5f19aab-kserve-8bc96b7d7-2h6fm"
Sep 08 02:32:00.500339 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:00.500256    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-a5f19aab-kserve-8bc96b7d7-2h6fm"]
Sep 08 02:32:00.503664 ip-10-0-142-78 kubenswrapper[2547]: W0908 02:32:00.503637    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeba28cd5_8b79_4601_be5b_287072ca4d2f.slice/crio-608b7b8bfa84e73399d8d94ebd008838369efafab3423c569cefad22db6fcd96 WatchSource:0}: Error finding container 608b7b8bfa84e73399d8d94ebd008838369efafab3423c569cefad22db6fcd96: Status 404 returned error can't find the container with id 608b7b8bfa84e73399d8d94ebd008838369efafab3423c569cefad22db6fcd96
Sep 08 02:32:00.793112 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:00.793016    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-a5f19aab-kserve-8bc96b7d7-2h6fm" event={"ID":"eba28cd5-8b79-4601-be5b-287072ca4d2f","Type":"ContainerStarted","Data":"4e39c005aa6c3673a4b7fd1aedff1e466dfd462077333ff9f55e17d682d26936"}
Sep 08 02:32:00.793112 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:00.793061    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-a5f19aab-kserve-8bc96b7d7-2h6fm" event={"ID":"eba28cd5-8b79-4601-be5b-287072ca4d2f","Type":"ContainerStarted","Data":"608b7b8bfa84e73399d8d94ebd008838369efafab3423c569cefad22db6fcd96"}
Sep 08 02:32:00.810117 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:00.810064    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-a5f19aab-kserve-8bc96b7d7-2h6fm" podStartSLOduration=1.810045914 podStartE2EDuration="1.810045914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:31:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:32:00.809219049 +0000 UTC m=+1992.332994131" watchObservedRunningTime="2026-09-08 02:32:00.810045914 +0000 UTC m=+1992.333820995"
Sep 08 02:32:01.737399 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:01.737353    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-a5f19aab-kserve-c4d6fdb79-dltgw"
Sep 08 02:32:01.737901 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:01.737409    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-a5f19aab-kserve-c4d6fdb79-dltgw"
Sep 08 02:32:01.750379 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:01.750343    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-a5f19aab-kserve-c4d6fdb79-dltgw"
Sep 08 02:32:01.808980 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:01.808952    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-a5f19aab-kserve-c4d6fdb79-dltgw"
Sep 08 02:32:10.229182 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:10.229103    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-a5f19aab-kserve-8bc96b7d7-2h6fm"
Sep 08 02:32:10.229632 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:10.229195    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-a5f19aab-kserve-8bc96b7d7-2h6fm"
Sep 08 02:32:10.241597 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:10.241567    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-a5f19aab-kserve-8bc96b7d7-2h6fm"
Sep 08 02:32:10.848347 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:10.848311    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-a5f19aab-kserve-8bc96b7d7-2h6fm"
Sep 08 02:32:16.519858 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:16.519822    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-a5f19aab-kserve-c4d6fdb79-dltgw"]
Sep 08 02:32:17.620234 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:17.620197    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-a5f19aab-kserve-8bc96b7d7-2h6fm"]
Sep 08 02:32:31.538867 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:31.538820    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-a5f19aab-kserve-c4d6fdb79-dltgw" podUID="0ae6a9c3-b15c-439f-9364-c9f38c3fb360" containerName="main" containerID="cri-o://ae8cf20d61501b0eed45847e29da49e9c894a7da82c3d705abefeedd7e3d8cba" gracePeriod=45
Sep 08 02:32:31.776496 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:31.776470    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-a5f19aab-kserve-c4d6fdb79-dltgw"
Sep 08 02:32:31.844661 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:31.844573    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8ntgk\" (UniqueName: \"kubernetes.io/projected/0ae6a9c3-b15c-439f-9364-c9f38c3fb360-kube-api-access-8ntgk\") pod \"0ae6a9c3-b15c-439f-9364-c9f38c3fb360\" (UID: \"0ae6a9c3-b15c-439f-9364-c9f38c3fb360\") "
Sep 08 02:32:31.844661 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:31.844618    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ae6a9c3-b15c-439f-9364-c9f38c3fb360-dshm\") pod \"0ae6a9c3-b15c-439f-9364-c9f38c3fb360\" (UID: \"0ae6a9c3-b15c-439f-9364-c9f38c3fb360\") "
Sep 08 02:32:31.844868 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:31.844742    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ae6a9c3-b15c-439f-9364-c9f38c3fb360-model-cache\") pod \"0ae6a9c3-b15c-439f-9364-c9f38c3fb360\" (UID: \"0ae6a9c3-b15c-439f-9364-c9f38c3fb360\") "
Sep 08 02:32:31.844868 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:31.844781    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ae6a9c3-b15c-439f-9364-c9f38c3fb360-tmp-dir\") pod \"0ae6a9c3-b15c-439f-9364-c9f38c3fb360\" (UID: \"0ae6a9c3-b15c-439f-9364-c9f38c3fb360\") "
Sep 08 02:32:31.844868 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:31.844800    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ae6a9c3-b15c-439f-9364-c9f38c3fb360-tls-certs\") pod \"0ae6a9c3-b15c-439f-9364-c9f38c3fb360\" (UID: \"0ae6a9c3-b15c-439f-9364-c9f38c3fb360\") "
Sep 08 02:32:31.844868 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:31.844830    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ae6a9c3-b15c-439f-9364-c9f38c3fb360-home\") pod \"0ae6a9c3-b15c-439f-9364-c9f38c3fb360\" (UID: \"0ae6a9c3-b15c-439f-9364-c9f38c3fb360\") "
Sep 08 02:32:31.845079 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:31.845037    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ae6a9c3-b15c-439f-9364-c9f38c3fb360-model-cache" (OuterVolumeSpecName: "model-cache") pod "0ae6a9c3-b15c-439f-9364-c9f38c3fb360" (UID: "0ae6a9c3-b15c-439f-9364-c9f38c3fb360"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 02:32:31.845079 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:31.845054    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ae6a9c3-b15c-439f-9364-c9f38c3fb360-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0ae6a9c3-b15c-439f-9364-c9f38c3fb360" (UID: "0ae6a9c3-b15c-439f-9364-c9f38c3fb360"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 02:32:31.845217 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:31.845196    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ae6a9c3-b15c-439f-9364-c9f38c3fb360-home" (OuterVolumeSpecName: "home") pod "0ae6a9c3-b15c-439f-9364-c9f38c3fb360" (UID: "0ae6a9c3-b15c-439f-9364-c9f38c3fb360"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 02:32:31.846920 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:31.846895    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ae6a9c3-b15c-439f-9364-c9f38c3fb360-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0ae6a9c3-b15c-439f-9364-c9f38c3fb360" (UID: "0ae6a9c3-b15c-439f-9364-c9f38c3fb360"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 02:32:31.847275 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:31.847256    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ae6a9c3-b15c-439f-9364-c9f38c3fb360-dshm" (OuterVolumeSpecName: "dshm") pod "0ae6a9c3-b15c-439f-9364-c9f38c3fb360" (UID: "0ae6a9c3-b15c-439f-9364-c9f38c3fb360"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 02:32:31.847353 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:31.847254    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ae6a9c3-b15c-439f-9364-c9f38c3fb360-kube-api-access-8ntgk" (OuterVolumeSpecName: "kube-api-access-8ntgk") pod "0ae6a9c3-b15c-439f-9364-c9f38c3fb360" (UID: "0ae6a9c3-b15c-439f-9364-c9f38c3fb360"). InnerVolumeSpecName "kube-api-access-8ntgk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 02:32:31.925311 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:31.925271    2547 generic.go:356] "Generic (PLEG): container finished" podID="0ae6a9c3-b15c-439f-9364-c9f38c3fb360" containerID="ae8cf20d61501b0eed45847e29da49e9c894a7da82c3d705abefeedd7e3d8cba" exitCode=0
Sep 08 02:32:31.925457 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:31.925313    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-a5f19aab-kserve-c4d6fdb79-dltgw"
Sep 08 02:32:31.925457 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:31.925351    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-a5f19aab-kserve-c4d6fdb79-dltgw" event={"ID":"0ae6a9c3-b15c-439f-9364-c9f38c3fb360","Type":"ContainerDied","Data":"ae8cf20d61501b0eed45847e29da49e9c894a7da82c3d705abefeedd7e3d8cba"}
Sep 08 02:32:31.925457 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:31.925392    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-a5f19aab-kserve-c4d6fdb79-dltgw" event={"ID":"0ae6a9c3-b15c-439f-9364-c9f38c3fb360","Type":"ContainerDied","Data":"237d9060e173815c35522e5d2bed36606072b3733eeaf58f67a883c7483262cf"}
Sep 08 02:32:31.925457 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:31.925408    2547 scope.go:117] "RemoveContainer" containerID="ae8cf20d61501b0eed45847e29da49e9c894a7da82c3d705abefeedd7e3d8cba"
Sep 08 02:32:31.933712 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:31.933693    2547 scope.go:117] "RemoveContainer" containerID="ae8cf20d61501b0eed45847e29da49e9c894a7da82c3d705abefeedd7e3d8cba"
Sep 08 02:32:31.934002 ip-10-0-142-78 kubenswrapper[2547]: E0908 02:32:31.933985    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae8cf20d61501b0eed45847e29da49e9c894a7da82c3d705abefeedd7e3d8cba\": container with ID starting with ae8cf20d61501b0eed45847e29da49e9c894a7da82c3d705abefeedd7e3d8cba not found: ID does not exist" containerID="ae8cf20d61501b0eed45847e29da49e9c894a7da82c3d705abefeedd7e3d8cba"
Sep 08 02:32:31.934052 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:31.934012    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae8cf20d61501b0eed45847e29da49e9c894a7da82c3d705abefeedd7e3d8cba"} err="failed to get container status \"ae8cf20d61501b0eed45847e29da49e9c894a7da82c3d705abefeedd7e3d8cba\": rpc error: code = NotFound desc = could not find container \"ae8cf20d61501b0eed45847e29da49e9c894a7da82c3d705abefeedd7e3d8cba\": container with ID starting with ae8cf20d61501b0eed45847e29da49e9c894a7da82c3d705abefeedd7e3d8cba not found: ID does not exist"
Sep 08 02:32:31.944879 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:31.944853    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-a5f19aab-kserve-c4d6fdb79-dltgw"]
Sep 08 02:32:31.946370 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:31.946351    2547 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ae6a9c3-b15c-439f-9364-c9f38c3fb360-model-cache\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:32:31.946469 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:31.946373    2547 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ae6a9c3-b15c-439f-9364-c9f38c3fb360-tmp-dir\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:32:31.946469 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:31.946386    2547 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ae6a9c3-b15c-439f-9364-c9f38c3fb360-tls-certs\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:32:31.946469 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:31.946398    2547 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ae6a9c3-b15c-439f-9364-c9f38c3fb360-home\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:32:31.946469 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:31.946411    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ntgk\" (UniqueName: \"kubernetes.io/projected/0ae6a9c3-b15c-439f-9364-c9f38c3fb360-kube-api-access-8ntgk\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:32:31.946469 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:31.946425    2547 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ae6a9c3-b15c-439f-9364-c9f38c3fb360-dshm\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:32:31.948953 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:31.948927    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-a5f19aab-kserve-c4d6fdb79-dltgw"]
Sep 08 02:32:32.639680 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:32.639639    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-a5f19aab-kserve-8bc96b7d7-2h6fm" podUID="eba28cd5-8b79-4601-be5b-287072ca4d2f" containerName="main" containerID="cri-o://4e39c005aa6c3673a4b7fd1aedff1e466dfd462077333ff9f55e17d682d26936" gracePeriod=45
Sep 08 02:32:32.896600 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:32.896535    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-a5f19aab-kserve-8bc96b7d7-2h6fm"
Sep 08 02:32:32.931632 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:32.931597    2547 generic.go:356] "Generic (PLEG): container finished" podID="eba28cd5-8b79-4601-be5b-287072ca4d2f" containerID="4e39c005aa6c3673a4b7fd1aedff1e466dfd462077333ff9f55e17d682d26936" exitCode=0
Sep 08 02:32:32.931835 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:32.931637    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-a5f19aab-kserve-8bc96b7d7-2h6fm"
Sep 08 02:32:32.931835 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:32.931650    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-a5f19aab-kserve-8bc96b7d7-2h6fm" event={"ID":"eba28cd5-8b79-4601-be5b-287072ca4d2f","Type":"ContainerDied","Data":"4e39c005aa6c3673a4b7fd1aedff1e466dfd462077333ff9f55e17d682d26936"}
Sep 08 02:32:32.931835 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:32.931681    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-a5f19aab-kserve-8bc96b7d7-2h6fm" event={"ID":"eba28cd5-8b79-4601-be5b-287072ca4d2f","Type":"ContainerDied","Data":"608b7b8bfa84e73399d8d94ebd008838369efafab3423c569cefad22db6fcd96"}
Sep 08 02:32:32.931835 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:32.931699    2547 scope.go:117] "RemoveContainer" containerID="4e39c005aa6c3673a4b7fd1aedff1e466dfd462077333ff9f55e17d682d26936"
Sep 08 02:32:32.940013 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:32.939995    2547 scope.go:117] "RemoveContainer" containerID="4e39c005aa6c3673a4b7fd1aedff1e466dfd462077333ff9f55e17d682d26936"
Sep 08 02:32:32.940304 ip-10-0-142-78 kubenswrapper[2547]: E0908 02:32:32.940285    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e39c005aa6c3673a4b7fd1aedff1e466dfd462077333ff9f55e17d682d26936\": container with ID starting with 4e39c005aa6c3673a4b7fd1aedff1e466dfd462077333ff9f55e17d682d26936 not found: ID does not exist" containerID="4e39c005aa6c3673a4b7fd1aedff1e466dfd462077333ff9f55e17d682d26936"
Sep 08 02:32:32.940348 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:32.940311    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e39c005aa6c3673a4b7fd1aedff1e466dfd462077333ff9f55e17d682d26936"} err="failed to get container status \"4e39c005aa6c3673a4b7fd1aedff1e466dfd462077333ff9f55e17d682d26936\": rpc error: code = NotFound desc = could not find container \"4e39c005aa6c3673a4b7fd1aedff1e466dfd462077333ff9f55e17d682d26936\": container with ID starting with 4e39c005aa6c3673a4b7fd1aedff1e466dfd462077333ff9f55e17d682d26936 not found: ID does not exist"
Sep 08 02:32:32.954682 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:32.954646    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eba28cd5-8b79-4601-be5b-287072ca4d2f-tls-certs\") pod \"eba28cd5-8b79-4601-be5b-287072ca4d2f\" (UID: \"eba28cd5-8b79-4601-be5b-287072ca4d2f\") "
Sep 08 02:32:32.954764 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:32.954694    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eba28cd5-8b79-4601-be5b-287072ca4d2f-dshm\") pod \"eba28cd5-8b79-4601-be5b-287072ca4d2f\" (UID: \"eba28cd5-8b79-4601-be5b-287072ca4d2f\") "
Sep 08 02:32:32.954764 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:32.954745    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eba28cd5-8b79-4601-be5b-287072ca4d2f-model-cache\") pod \"eba28cd5-8b79-4601-be5b-287072ca4d2f\" (UID: \"eba28cd5-8b79-4601-be5b-287072ca4d2f\") "
Sep 08 02:32:32.954866 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:32.954768    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eba28cd5-8b79-4601-be5b-287072ca4d2f-tmp-dir\") pod \"eba28cd5-8b79-4601-be5b-287072ca4d2f\" (UID: \"eba28cd5-8b79-4601-be5b-287072ca4d2f\") "
Sep 08 02:32:32.954866 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:32.954799    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eba28cd5-8b79-4601-be5b-287072ca4d2f-home\") pod \"eba28cd5-8b79-4601-be5b-287072ca4d2f\" (UID: \"eba28cd5-8b79-4601-be5b-287072ca4d2f\") "
Sep 08 02:32:32.954969 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:32.954874    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jfqn8\" (UniqueName: \"kubernetes.io/projected/eba28cd5-8b79-4601-be5b-287072ca4d2f-kube-api-access-jfqn8\") pod \"eba28cd5-8b79-4601-be5b-287072ca4d2f\" (UID: \"eba28cd5-8b79-4601-be5b-287072ca4d2f\") "
Sep 08 02:32:32.955062 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:32.955032    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eba28cd5-8b79-4601-be5b-287072ca4d2f-model-cache" (OuterVolumeSpecName: "model-cache") pod "eba28cd5-8b79-4601-be5b-287072ca4d2f" (UID: "eba28cd5-8b79-4601-be5b-287072ca4d2f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 02:32:32.955120 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:32.955066    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eba28cd5-8b79-4601-be5b-287072ca4d2f-home" (OuterVolumeSpecName: "home") pod "eba28cd5-8b79-4601-be5b-287072ca4d2f" (UID: "eba28cd5-8b79-4601-be5b-287072ca4d2f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 02:32:32.955120 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:32.955046    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eba28cd5-8b79-4601-be5b-287072ca4d2f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "eba28cd5-8b79-4601-be5b-287072ca4d2f" (UID: "eba28cd5-8b79-4601-be5b-287072ca4d2f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 02:32:32.955217 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:32.955162    2547 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eba28cd5-8b79-4601-be5b-287072ca4d2f-model-cache\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:32:32.955217 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:32.955181    2547 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eba28cd5-8b79-4601-be5b-287072ca4d2f-tmp-dir\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:32:32.955217 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:32.955192    2547 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eba28cd5-8b79-4601-be5b-287072ca4d2f-home\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:32:32.956923 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:32.956896    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eba28cd5-8b79-4601-be5b-287072ca4d2f-dshm" (OuterVolumeSpecName: "dshm") pod "eba28cd5-8b79-4601-be5b-287072ca4d2f" (UID: "eba28cd5-8b79-4601-be5b-287072ca4d2f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 02:32:32.957030 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:32.956933    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eba28cd5-8b79-4601-be5b-287072ca4d2f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "eba28cd5-8b79-4601-be5b-287072ca4d2f" (UID: "eba28cd5-8b79-4601-be5b-287072ca4d2f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 02:32:32.957030 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:32.956970    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eba28cd5-8b79-4601-be5b-287072ca4d2f-kube-api-access-jfqn8" (OuterVolumeSpecName: "kube-api-access-jfqn8") pod "eba28cd5-8b79-4601-be5b-287072ca4d2f" (UID: "eba28cd5-8b79-4601-be5b-287072ca4d2f"). InnerVolumeSpecName "kube-api-access-jfqn8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 02:32:32.993759 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:32.993721    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ae6a9c3-b15c-439f-9364-c9f38c3fb360" path="/var/lib/kubelet/pods/0ae6a9c3-b15c-439f-9364-c9f38c3fb360/volumes"
Sep 08 02:32:33.056207 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:33.056163    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfqn8\" (UniqueName: \"kubernetes.io/projected/eba28cd5-8b79-4601-be5b-287072ca4d2f-kube-api-access-jfqn8\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:32:33.056207 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:33.056200    2547 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eba28cd5-8b79-4601-be5b-287072ca4d2f-tls-certs\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:32:33.056207 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:33.056215    2547 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eba28cd5-8b79-4601-be5b-287072ca4d2f-dshm\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Sep 08 02:32:33.245963 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:33.245875    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-a5f19aab-kserve-8bc96b7d7-2h6fm"]
Sep 08 02:32:33.249593 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:33.249566    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-a5f19aab-kserve-8bc96b7d7-2h6fm"]
Sep 08 02:32:34.993865 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:34.993833    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eba28cd5-8b79-4601-be5b-287072ca4d2f" path="/var/lib/kubelet/pods/eba28cd5-8b79-4601-be5b-287072ca4d2f/volumes"
Sep 08 02:32:45.541231 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:45.541187    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-cf5b5fd48-2fp9m"]
Sep 08 02:32:45.541615 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:45.541498    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eba28cd5-8b79-4601-be5b-287072ca4d2f" containerName="main"
Sep 08 02:32:45.541615 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:45.541509    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="eba28cd5-8b79-4601-be5b-287072ca4d2f" containerName="main"
Sep 08 02:32:45.541615 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:45.541523    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ae6a9c3-b15c-439f-9364-c9f38c3fb360" containerName="main"
Sep 08 02:32:45.541615 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:45.541529    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ae6a9c3-b15c-439f-9364-c9f38c3fb360" containerName="main"
Sep 08 02:32:45.541615 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:45.541583    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ae6a9c3-b15c-439f-9364-c9f38c3fb360" containerName="main"
Sep 08 02:32:45.541615 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:45.541590    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="eba28cd5-8b79-4601-be5b-287072ca4d2f" containerName="main"
Sep 08 02:32:45.545746 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:45.545728    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-cf5b5fd48-2fp9m"
Sep 08 02:32:45.549565 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:45.549529    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 08 02:32:45.549565 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:45.549536    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-tfvqz\""
Sep 08 02:32:45.549769 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:45.549590    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 08 02:32:45.549769 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:45.549536    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 08 02:32:45.549769 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:45.549635    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 02:32:45.556782 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:45.556748    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-cf5b5fd48-2fp9m"]
Sep 08 02:32:45.660374 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:45.660341    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqg2g\" (UniqueName: \"kubernetes.io/projected/c96012c8-a15b-479b-aa3d-f40d7dcbad70-kube-api-access-sqg2g\") pod \"maas-api-cf5b5fd48-2fp9m\" (UID: \"c96012c8-a15b-479b-aa3d-f40d7dcbad70\") " pod="odh-ai-gateway-infra/maas-api-cf5b5fd48-2fp9m"
Sep 08 02:32:45.660559 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:45.660382    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c96012c8-a15b-479b-aa3d-f40d7dcbad70-openshift-service-ca\") pod \"maas-api-cf5b5fd48-2fp9m\" (UID: \"c96012c8-a15b-479b-aa3d-f40d7dcbad70\") " pod="odh-ai-gateway-infra/maas-api-cf5b5fd48-2fp9m"
Sep 08 02:32:45.660559 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:45.660404    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c96012c8-a15b-479b-aa3d-f40d7dcbad70-maas-api-tls\") pod \"maas-api-cf5b5fd48-2fp9m\" (UID: \"c96012c8-a15b-479b-aa3d-f40d7dcbad70\") " pod="odh-ai-gateway-infra/maas-api-cf5b5fd48-2fp9m"
Sep 08 02:32:45.660559 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:45.660519    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c96012c8-a15b-479b-aa3d-f40d7dcbad70-metrics-certs\") pod \"maas-api-cf5b5fd48-2fp9m\" (UID: \"c96012c8-a15b-479b-aa3d-f40d7dcbad70\") " pod="odh-ai-gateway-infra/maas-api-cf5b5fd48-2fp9m"
Sep 08 02:32:45.761198 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:45.761155    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqg2g\" (UniqueName: \"kubernetes.io/projected/c96012c8-a15b-479b-aa3d-f40d7dcbad70-kube-api-access-sqg2g\") pod \"maas-api-cf5b5fd48-2fp9m\" (UID: \"c96012c8-a15b-479b-aa3d-f40d7dcbad70\") " pod="odh-ai-gateway-infra/maas-api-cf5b5fd48-2fp9m"
Sep 08 02:32:45.761198 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:45.761201    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c96012c8-a15b-479b-aa3d-f40d7dcbad70-openshift-service-ca\") pod \"maas-api-cf5b5fd48-2fp9m\" (UID: \"c96012c8-a15b-479b-aa3d-f40d7dcbad70\") " pod="odh-ai-gateway-infra/maas-api-cf5b5fd48-2fp9m"
Sep 08 02:32:45.761393 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:45.761221    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c96012c8-a15b-479b-aa3d-f40d7dcbad70-maas-api-tls\") pod \"maas-api-cf5b5fd48-2fp9m\" (UID: \"c96012c8-a15b-479b-aa3d-f40d7dcbad70\") " pod="odh-ai-gateway-infra/maas-api-cf5b5fd48-2fp9m"
Sep 08 02:32:45.761393 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:45.761256    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c96012c8-a15b-479b-aa3d-f40d7dcbad70-metrics-certs\") pod \"maas-api-cf5b5fd48-2fp9m\" (UID: \"c96012c8-a15b-479b-aa3d-f40d7dcbad70\") " pod="odh-ai-gateway-infra/maas-api-cf5b5fd48-2fp9m"
Sep 08 02:32:45.761860 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:45.761828    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c96012c8-a15b-479b-aa3d-f40d7dcbad70-openshift-service-ca\") pod \"maas-api-cf5b5fd48-2fp9m\" (UID: \"c96012c8-a15b-479b-aa3d-f40d7dcbad70\") " pod="odh-ai-gateway-infra/maas-api-cf5b5fd48-2fp9m"
Sep 08 02:32:45.763880 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:45.763859    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c96012c8-a15b-479b-aa3d-f40d7dcbad70-maas-api-tls\") pod \"maas-api-cf5b5fd48-2fp9m\" (UID: \"c96012c8-a15b-479b-aa3d-f40d7dcbad70\") " pod="odh-ai-gateway-infra/maas-api-cf5b5fd48-2fp9m"
Sep 08 02:32:45.764010 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:45.763988    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c96012c8-a15b-479b-aa3d-f40d7dcbad70-metrics-certs\") pod \"maas-api-cf5b5fd48-2fp9m\" (UID: \"c96012c8-a15b-479b-aa3d-f40d7dcbad70\") " pod="odh-ai-gateway-infra/maas-api-cf5b5fd48-2fp9m"
Sep 08 02:32:45.770126 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:45.770097    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqg2g\" (UniqueName: \"kubernetes.io/projected/c96012c8-a15b-479b-aa3d-f40d7dcbad70-kube-api-access-sqg2g\") pod \"maas-api-cf5b5fd48-2fp9m\" (UID: \"c96012c8-a15b-479b-aa3d-f40d7dcbad70\") " pod="odh-ai-gateway-infra/maas-api-cf5b5fd48-2fp9m"
Sep 08 02:32:45.857029 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:45.856931    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-cf5b5fd48-2fp9m"
Sep 08 02:32:46.128632 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:46.128566    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-cf5b5fd48-2fp9m"]
Sep 08 02:32:46.130878 ip-10-0-142-78 kubenswrapper[2547]: W0908 02:32:46.130848    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc96012c8_a15b_479b_aa3d_f40d7dcbad70.slice/crio-467e49cec35442aa70a32ebc2ab5af0a8865c51bba0d3e3aa3f3ce0fb167f6c3 WatchSource:0}: Error finding container 467e49cec35442aa70a32ebc2ab5af0a8865c51bba0d3e3aa3f3ce0fb167f6c3: Status 404 returned error can't find the container with id 467e49cec35442aa70a32ebc2ab5af0a8865c51bba0d3e3aa3f3ce0fb167f6c3
Sep 08 02:32:46.132257 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:46.132240    2547 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 02:32:46.995395 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:46.995358    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-cf5b5fd48-2fp9m" event={"ID":"c96012c8-a15b-479b-aa3d-f40d7dcbad70","Type":"ContainerStarted","Data":"467e49cec35442aa70a32ebc2ab5af0a8865c51bba0d3e3aa3f3ce0fb167f6c3"}
Sep 08 02:32:48.002257 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:48.002223    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-cf5b5fd48-2fp9m" event={"ID":"c96012c8-a15b-479b-aa3d-f40d7dcbad70","Type":"ContainerStarted","Data":"eb71fb7875c3b658b3431fa0f73fcd4f4f32b4889849e3b1c13674c30e66f986"}
Sep 08 02:32:48.002675 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:48.002329    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-cf5b5fd48-2fp9m"
Sep 08 02:32:48.033238 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:48.033182    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-cf5b5fd48-2fp9m" podStartSLOduration=1.7605654259999999 podStartE2EDuration="3.03316629s" totalImagesPullingTime="1.272600864s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 02:32:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:32:48.032769395 +0000 UTC m=+2039.556544503" watchObservedRunningTime="2026-09-08 02:32:48.03316629 +0000 UTC m=+2039.556941369"
Sep 08 02:32:54.012406 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:32:54.012366    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-cf5b5fd48-2fp9m"
Sep 08 02:33:15.783108 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:33:15.783024    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-4fwjq_6aea39a7-0b3c-4f02-b5cc-b86f82d41c9c/server/0.log"
Sep 08 02:33:16.137763 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:33:16.137674    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6566bd6ff-rxqm5_277e92d6-e780-4011-a698-6f00a205e02b/manager/0.log"
Sep 08 02:33:16.144851 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:33:16.144825    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6566bd6ff-rxqm5_277e92d6-e780-4011-a698-6f00a205e02b/copy-manifests/0.log"
Sep 08 02:33:16.584098 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:33:16.584055    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-cf5b5fd48-2fp9m_c96012c8-a15b-479b-aa3d-f40d7dcbad70/maas-api/0.log"
Sep 08 02:33:16.803004 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:33:16.802969    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29813895-dl7f8_bf435bc6-e06f-45f6-a2bc-76001b84f52a/cleanup/0.log"
Sep 08 02:33:19.103636 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:33:19.103603    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-9c6fc765d-j4v4c_21eff83a-e938-4010-b027-0088de0fc9c8/kube-auth-proxy/0.log"
Sep 08 02:33:19.325126 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:33:19.325092    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-fmdgk_6d582b5b-9b6f-4e26-b6be-bafecf8c1c99/payload-pre-processing/0.log"
Sep 08 02:33:19.873682 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:33:19.873651    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-kct29_334aa197-482b-4f94-864b-bd2166f33f06/main/0.log"
Sep 08 02:33:20.086275 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:33:20.086230    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-shnmg_e0ea35bd-ed0a-4c4b-9d04-6eab00f44427/main/0.log"
Sep 08 02:33:20.189307 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:33:20.189280    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-d8vv5_289f142c-f6f5-4b56-990d-e53f69dd5e9e/main/0.log"
Sep 08 02:33:20.510034 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:33:20.509957    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-bwvxn_0e0ddfff-324b-445d-9b2e-92448da092ec/main/0.log"
Sep 08 02:33:28.853641 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:33:28.853605    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-l7hw8_917f6c17-7902-48d3-a0f4-535e5b602d26/global-pull-secret-syncer/0.log"
Sep 08 02:33:29.049751 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:33:29.049714    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-xr9c8_c793f939-4584-46af-a495-36b57bbdff41/konnectivity-agent/0.log"
Sep 08 02:33:29.121378 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:33:29.121303    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-78.ec2.internal_5cfe1f564aebe22c530741e9b588e31d/haproxy/0.log"
Sep 08 02:33:35.601996 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:33:35.601964    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5ngwm_167f56fa-baa2-41bf-8a91-0b63f41213a6/node-exporter/0.log"
Sep 08 02:33:35.642531 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:33:35.642502    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5ngwm_167f56fa-baa2-41bf-8a91-0b63f41213a6/kube-rbac-proxy/0.log"
Sep 08 02:33:35.687221 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:33:35.687193    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5ngwm_167f56fa-baa2-41bf-8a91-0b63f41213a6/init-textfile/0.log"
Sep 08 02:33:38.853194 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:33:38.853156    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-86shj/perf-node-gather-daemonset-6mmrs"]
Sep 08 02:33:38.857155 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:33:38.857100    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-86shj/perf-node-gather-daemonset-6mmrs"
Sep 08 02:33:38.860614 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:33:38.860584    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-86shj\"/\"kube-root-ca.crt\""
Sep 08 02:33:38.860762 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:33:38.860672    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-86shj\"/\"openshift-service-ca.crt\""
Sep 08 02:33:38.860832 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:33:38.860779    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-86shj\"/\"default-dockercfg-5vlsl\""
Sep 08 02:33:38.871508 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:33:38.871481    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-86shj/perf-node-gather-daemonset-6mmrs"]
Sep 08 02:33:39.021115 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:33:39.021077    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/124f08d5-c524-4455-a272-80ff066cbfaa-lib-modules\") pod \"perf-node-gather-daemonset-6mmrs\" (UID: \"124f08d5-c524-4455-a272-80ff066cbfaa\") " pod="openshift-must-gather-86shj/perf-node-gather-daemonset-6mmrs"
Sep 08 02:33:39.021115 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:33:39.021115    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/124f08d5-c524-4455-a272-80ff066cbfaa-podres\") pod \"perf-node-gather-daemonset-6mmrs\" (UID: \"124f08d5-c524-4455-a272-80ff066cbfaa\") " pod="openshift-must-gather-86shj/perf-node-gather-daemonset-6mmrs"
Sep 08 02:33:39.021386 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:33:39.021193    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/124f08d5-c524-4455-a272-80ff066cbfaa-sys\") pod \"perf-node-gather-daemonset-6mmrs\" (UID: \"124f08d5-c524-4455-a272-80ff066cbfaa\") " pod="openshift-must-gather-86shj/perf-node-gather-daemonset-6mmrs"
Sep 08 02:33:39.021386 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:33:39.021212    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/124f08d5-c524-4455-a272-80ff066cbfaa-proc\") pod \"perf-node-gather-daemonset-6mmrs\" (UID: \"124f08d5-c524-4455-a272-80ff066cbfaa\") " pod="openshift-must-gather-86shj/perf-node-gather-daemonset-6mmrs"
Sep 08 02:33:39.021386 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:33:39.021293    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6x24\" (UniqueName: \"kubernetes.io/projected/124f08d5-c524-4455-a272-80ff066cbfaa-kube-api-access-s6x24\") pod \"perf-node-gather-daemonset-6mmrs\" (UID: \"124f08d5-c524-4455-a272-80ff066cbfaa\") " pod="openshift-must-gather-86shj/perf-node-gather-daemonset-6mmrs"
Sep 08 02:33:39.122584 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:33:39.122496    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/124f08d5-c524-4455-a272-80ff066cbfaa-lib-modules\") pod \"perf-node-gather-daemonset-6mmrs\" (UID: \"124f08d5-c524-4455-a272-80ff066cbfaa\") " pod="openshift-must-gather-86shj/perf-node-gather-daemonset-6mmrs"
Sep 08 02:33:39.122584 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:33:39.122532    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/124f08d5-c524-4455-a272-80ff066cbfaa-podres\") pod \"perf-node-gather-daemonset-6mmrs\" (UID: \"124f08d5-c524-4455-a272-80ff066cbfaa\") " pod="openshift-must-gather-86shj/perf-node-gather-daemonset-6mmrs"
Sep 08 02:33:39.122810 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:33:39.122664    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/124f08d5-c524-4455-a272-80ff066cbfaa-lib-modules\") pod \"perf-node-gather-daemonset-6mmrs\" (UID: \"124f08d5-c524-4455-a272-80ff066cbfaa\") " pod="openshift-must-gather-86shj/perf-node-gather-daemonset-6mmrs"
Sep 08 02:33:39.122810 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:33:39.122784    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/124f08d5-c524-4455-a272-80ff066cbfaa-podres\") pod \"perf-node-gather-daemonset-6mmrs\" (UID: \"124f08d5-c524-4455-a272-80ff066cbfaa\") " pod="openshift-must-gather-86shj/perf-node-gather-daemonset-6mmrs"
Sep 08 02:33:39.122925 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:33:39.122831    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/124f08d5-c524-4455-a272-80ff066cbfaa-sys\") pod \"perf-node-gather-daemonset-6mmrs\" (UID: \"124f08d5-c524-4455-a272-80ff066cbfaa\") " pod="openshift-must-gather-86shj/perf-node-gather-daemonset-6mmrs"
Sep 08 02:33:39.122925 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:33:39.122867    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/124f08d5-c524-4455-a272-80ff066cbfaa-proc\") pod \"perf-node-gather-daemonset-6mmrs\" (UID: \"124f08d5-c524-4455-a272-80ff066cbfaa\") " pod="openshift-must-gather-86shj/perf-node-gather-daemonset-6mmrs"
Sep 08 02:33:39.123041 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:33:39.122921    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/124f08d5-c524-4455-a272-80ff066cbfaa-sys\") pod \"perf-node-gather-daemonset-6mmrs\" (UID: \"124f08d5-c524-4455-a272-80ff066cbfaa\") " pod="openshift-must-gather-86shj/perf-node-gather-daemonset-6mmrs"
Sep 08 02:33:39.123041 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:33:39.122927    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6x24\" (UniqueName: \"kubernetes.io/projected/124f08d5-c524-4455-a272-80ff066cbfaa-kube-api-access-s6x24\") pod \"perf-node-gather-daemonset-6mmrs\" (UID: \"124f08d5-c524-4455-a272-80ff066cbfaa\") " pod="openshift-must-gather-86shj/perf-node-gather-daemonset-6mmrs"
Sep 08 02:33:39.123041 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:33:39.122988    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/124f08d5-c524-4455-a272-80ff066cbfaa-proc\") pod \"perf-node-gather-daemonset-6mmrs\" (UID: \"124f08d5-c524-4455-a272-80ff066cbfaa\") " pod="openshift-must-gather-86shj/perf-node-gather-daemonset-6mmrs"
Sep 08 02:33:39.130905 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:33:39.130880    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6x24\" (UniqueName: \"kubernetes.io/projected/124f08d5-c524-4455-a272-80ff066cbfaa-kube-api-access-s6x24\") pod \"perf-node-gather-daemonset-6mmrs\" (UID: \"124f08d5-c524-4455-a272-80ff066cbfaa\") " pod="openshift-must-gather-86shj/perf-node-gather-daemonset-6mmrs"
Sep 08 02:33:39.168031 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:33:39.167993    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-86shj/perf-node-gather-daemonset-6mmrs"
Sep 08 02:33:39.290170 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:33:39.290078    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-85d98c998f-6qfk4_7ad15f65-08f1-4e92-a163-93131006559a/download-server/0.log"
Sep 08 02:33:39.437460 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:33:39.437430    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-86shj/perf-node-gather-daemonset-6mmrs"]
Sep 08 02:33:39.440031 ip-10-0-142-78 kubenswrapper[2547]: W0908 02:33:39.440004    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod124f08d5_c524_4455_a272_80ff066cbfaa.slice/crio-2aa47bff6315b59472803ac83399ff77f73832f71ed7188f56be031eb81eb68c WatchSource:0}: Error finding container 2aa47bff6315b59472803ac83399ff77f73832f71ed7188f56be031eb81eb68c: Status 404 returned error can't find the container with id 2aa47bff6315b59472803ac83399ff77f73832f71ed7188f56be031eb81eb68c
Sep 08 02:33:40.230903 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:33:40.230859    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-86shj/perf-node-gather-daemonset-6mmrs" event={"ID":"124f08d5-c524-4455-a272-80ff066cbfaa","Type":"ContainerStarted","Data":"c3ebec2050970b6a5c9934d12a5f245a66c8e5d2d1831852296a6b30bd5014f0"}
Sep 08 02:33:40.231321 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:33:40.230916    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-86shj/perf-node-gather-daemonset-6mmrs" event={"ID":"124f08d5-c524-4455-a272-80ff066cbfaa","Type":"ContainerStarted","Data":"2aa47bff6315b59472803ac83399ff77f73832f71ed7188f56be031eb81eb68c"}
Sep 08 02:33:40.231321 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:33:40.230942    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-86shj/perf-node-gather-daemonset-6mmrs"
Sep 08 02:33:40.247953 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:33:40.247896    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-86shj/perf-node-gather-daemonset-6mmrs" podStartSLOduration=2.247881749 podStartE2EDuration="2.247881749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 02:33:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 02:33:40.246282987 +0000 UTC m=+2091.770058079" watchObservedRunningTime="2026-09-08 02:33:40.247881749 +0000 UTC m=+2091.771656829"
Sep 08 02:33:41.173552 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:33:41.173524    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-q6gwz_3ba60eb6-74cd-4416-9a15-d04ba2ef2b1e/dns/0.log"
Sep 08 02:33:41.194585 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:33:41.194563    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-q6gwz_3ba60eb6-74cd-4416-9a15-d04ba2ef2b1e/kube-rbac-proxy/0.log"
Sep 08 02:33:41.362000 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:33:41.361967    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4s9cm_0536c61b-c71c-41d9-8b2c-18228a7ddff4/dns-node-resolver/0.log"
Sep 08 02:33:41.932078 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:33:41.932033    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-db5d55684-xlxmr_9ca7a4bc-6b17-4220-95e1-b28e01db798e/registry/0.log"
Sep 08 02:33:41.974350 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:33:41.974326    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-pl5qh_ae4a55d0-ad38-48a9-9a36-279949e2e4c5/node-ca/0.log"
Sep 08 02:33:43.499264 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:33:43.499236    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-9c6fc765d-j4v4c_21eff83a-e938-4010-b027-0088de0fc9c8/kube-auth-proxy/0.log"
Sep 08 02:33:43.591722 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:33:43.591684    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-fmdgk_6d582b5b-9b6f-4e26-b6be-bafecf8c1c99/payload-pre-processing/0.log"
Sep 08 02:33:44.501656 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:33:44.501628    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-87nvd_444897cb-e238-4b58-a0ef-9adce681c307/serve-healthcheck-canary/0.log"
Sep 08 02:33:45.231449 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:33:45.231417    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-sts5m_278c1af4-cf3a-49c4-80b8-a40b5b6ddb72/kube-rbac-proxy/0.log"
Sep 08 02:33:45.255041 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:33:45.255013    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-sts5m_278c1af4-cf3a-49c4-80b8-a40b5b6ddb72/exporter/0.log"
Sep 08 02:33:45.279490 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:33:45.279462    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-sts5m_278c1af4-cf3a-49c4-80b8-a40b5b6ddb72/extractor/0.log"
Sep 08 02:33:46.245541 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:33:46.245514    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-86shj/perf-node-gather-daemonset-6mmrs"
Sep 08 02:33:48.149585 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:33:48.149556    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-4fwjq_6aea39a7-0b3c-4f02-b5cc-b86f82d41c9c/server/0.log"
Sep 08 02:33:48.429413 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:33:48.429377    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6566bd6ff-rxqm5_277e92d6-e780-4011-a698-6f00a205e02b/manager/0.log"
Sep 08 02:33:48.448548 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:33:48.448518    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6566bd6ff-rxqm5_277e92d6-e780-4011-a698-6f00a205e02b/copy-manifests/0.log"
Sep 08 02:33:49.245894 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:33:49.245860    2547 scope.go:117] "RemoveContainer" containerID="1a2bf2071e5feb23445de5b99cf3ab7ec0547641a2625f02930f86f044fabe9d"
Sep 08 02:33:49.819864 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:33:49.819833    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5777c545f5-f4wtw_e95368d2-40b8-48c9-a354-f5e274c7f359/manager/0.log"
Sep 08 02:33:49.914340 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:33:49.914299    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-9c6fc765d-j4v4c_21eff83a-e938-4010-b027-0088de0fc9c8/kube-auth-proxy/0.log"
Sep 08 02:33:49.988760 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:33:49.988727    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-fmdgk_6d582b5b-9b6f-4e26-b6be-bafecf8c1c99/payload-pre-processing/0.log"
Sep 08 02:33:52.152663 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:33:52.152599    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-9c6fc765d-j4v4c_21eff83a-e938-4010-b027-0088de0fc9c8/kube-auth-proxy/0.log"
Sep 08 02:33:52.226171 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:33:52.226125    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-fmdgk_6d582b5b-9b6f-4e26-b6be-bafecf8c1c99/payload-pre-processing/0.log"
Sep 08 02:33:54.957248 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:33:54.957218    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-2j82x_90cc711b-31b7-4417-8bb6-9e827e71bb28/migrator/0.log"
Sep 08 02:33:54.978770 ip-10-0-142-78 kubenswrapper[2547]: I0908 02:33:54.978741    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-2j82x_90cc711b-31b7-4417-8bb6-9e827e71bb28/graceful-termination/0.log"