Sep 11 18:48:43.807908 ip-10-0-134-189 systemd[1]: Starting Kubernetes Kubelet...
Sep 11 18:48:44.181563 ip-10-0-134-189 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 18:48:44.181563 ip-10-0-134-189 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 11 18:48:44.181563 ip-10-0-134-189 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 18:48:44.181563 ip-10-0-134-189 kubenswrapper[2578]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 11 18:48:44.182215 ip-10-0-134-189 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 18:48:44.182319 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.182241    2578 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 11 18:48:44.185709 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.185695    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 18:48:44.185709 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.185710    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 18:48:44.185768 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.185713    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 18:48:44.185768 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.185716    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 18:48:44.185768 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.185719    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 18:48:44.185768 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.185722    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 18:48:44.185768 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.185725    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 18:48:44.185768 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.185727    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 18:48:44.185768 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.185730    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 18:48:44.185768 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.185732    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 18:48:44.185768 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.185735    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 18:48:44.185768 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.185737    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 18:48:44.185768 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.185740    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 18:48:44.185768 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.185742    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 18:48:44.185768 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.185745    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 18:48:44.185768 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.185748    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 18:48:44.185768 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.185751    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 18:48:44.185768 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.185754    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 18:48:44.185768 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.185757    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 18:48:44.185768 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.185760    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 18:48:44.185768 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.185763    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 18:48:44.185768 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.185765    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 18:48:44.186354 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.185767    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 18:48:44.186354 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.185770    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 18:48:44.186354 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.185773    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 18:48:44.186354 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.185776    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 18:48:44.186354 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.185778    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 18:48:44.186354 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.185781    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 18:48:44.186354 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.185784    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 18:48:44.186354 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.185786    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 18:48:44.186354 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.185788    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 18:48:44.186354 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.185791    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 18:48:44.186354 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.185794    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 18:48:44.186354 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.185797    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 18:48:44.186354 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.185799    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 18:48:44.186354 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.185802    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 18:48:44.186354 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.185804    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 18:48:44.186354 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.185806    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 18:48:44.186354 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.185809    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 18:48:44.186354 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.185811    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 18:48:44.186354 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.185814    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 18:48:44.186354 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.185816    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 18:48:44.187124 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.185818    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 18:48:44.187124 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.185821    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 18:48:44.187124 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.185823    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 18:48:44.187124 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.185826    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 18:48:44.187124 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.185828    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 18:48:44.187124 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.185830    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 18:48:44.187124 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.185833    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 18:48:44.187124 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.185835    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 18:48:44.187124 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.185838    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 18:48:44.187124 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.185840    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 18:48:44.187124 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.185843    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 18:48:44.187124 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.185845    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 18:48:44.187124 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.185847    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 18:48:44.187124 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.185850    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 18:48:44.187124 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.185853    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 18:48:44.187124 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.185857    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 18:48:44.187124 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.185861    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 18:48:44.187124 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.185864    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 18:48:44.187124 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.185867    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 18:48:44.187589 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.185870    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 18:48:44.187589 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.185873    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 18:48:44.187589 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.185875    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 18:48:44.187589 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.185877    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 18:48:44.187589 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.185880    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 18:48:44.187589 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.185883    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 18:48:44.187589 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.185885    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 18:48:44.187589 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.185888    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 18:48:44.187589 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.185890    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 18:48:44.187589 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.185892    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 18:48:44.187589 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.185895    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 18:48:44.187589 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.185897    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 18:48:44.187589 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.185899    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 18:48:44.187589 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.185902    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 18:48:44.187589 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.185904    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 18:48:44.187589 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.185907    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 18:48:44.187589 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.185909    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 18:48:44.187589 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.185911    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 18:48:44.187589 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.185916    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 18:48:44.187589 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.185919    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 18:48:44.188211 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.185922    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 18:48:44.188211 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.185925    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 18:48:44.188211 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.185928    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 18:48:44.188211 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.185930    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 18:48:44.188211 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.185933    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 18:48:44.188211 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.185936    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 18:48:44.188211 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.185939    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 18:48:44.188211 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.185942    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 18:48:44.188211 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.185945    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 18:48:44.188211 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.185949    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 18:48:44.188211 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.185953    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 18:48:44.188211 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.185956    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 18:48:44.188211 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.185959    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 18:48:44.188211 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.185962    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 18:48:44.188211 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.185964    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 18:48:44.188211 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.185967    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 18:48:44.188211 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.185970    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 18:48:44.188211 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.185973    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 18:48:44.188724 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.188709    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 18:48:44.188724 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.188724    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 18:48:44.188786 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.188728    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 18:48:44.188786 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.188732    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 18:48:44.188786 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.188735    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 18:48:44.188786 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.188738    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 18:48:44.188786 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.188741    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 18:48:44.188786 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.188744    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 18:48:44.188786 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.188747    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 18:48:44.188786 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.188749    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 18:48:44.188786 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.188752    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 18:48:44.188786 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.188754    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 18:48:44.188786 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.188757    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 18:48:44.188786 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.188760    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 18:48:44.188786 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.188762    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 18:48:44.188786 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.188765    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 18:48:44.188786 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.188767    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 18:48:44.188786 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.188770    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 18:48:44.188786 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.188773    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 18:48:44.188786 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.188775    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 18:48:44.188786 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.188778    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 18:48:44.188786 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.188780    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 18:48:44.189273 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.188783    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 18:48:44.189273 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.188786    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 18:48:44.189273 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.188789    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 18:48:44.189273 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.188792    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 18:48:44.189273 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.188795    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 18:48:44.189273 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.188798    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 18:48:44.189273 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.188801    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 18:48:44.189273 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.188803    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 18:48:44.189273 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.188806    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 18:48:44.189273 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.188808    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 18:48:44.189273 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.188811    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 18:48:44.189273 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.188814    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 18:48:44.189273 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.188816    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 18:48:44.189273 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.188819    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 18:48:44.189273 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.188821    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 18:48:44.189273 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.188823    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 18:48:44.189273 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.188826    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 18:48:44.189273 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.188828    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 18:48:44.189273 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.188832    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 18:48:44.189754 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.188836    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 18:48:44.189754 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.188839    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 18:48:44.189754 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.188842    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 18:48:44.189754 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.188845    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 18:48:44.189754 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.188847    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 18:48:44.189754 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.188850    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 18:48:44.189754 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.188854    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 18:48:44.189754 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.188858    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 18:48:44.189754 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.188861    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 18:48:44.189754 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.188864    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 18:48:44.189754 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.188867    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 18:48:44.189754 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.188871    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 18:48:44.189754 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.188874    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 18:48:44.189754 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.188876    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 18:48:44.189754 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.188879    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 18:48:44.189754 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.188881    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 18:48:44.189754 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.188884    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 18:48:44.189754 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.188886    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 18:48:44.189754 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.188889    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 18:48:44.190220 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.188891    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 18:48:44.190220 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.188894    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 18:48:44.190220 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.188896    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 18:48:44.190220 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.188899    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 18:48:44.190220 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.188901    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 18:48:44.190220 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.188904    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 18:48:44.190220 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.188906    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 18:48:44.190220 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.188908    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 18:48:44.190220 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.188911    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 18:48:44.190220 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.188913    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 18:48:44.190220 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.188915    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 18:48:44.190220 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.188918    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 18:48:44.190220 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.188921    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 18:48:44.190220 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.188923    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 18:48:44.190220 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.188926    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 18:48:44.190220 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.188928    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 18:48:44.190220 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.188931    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 18:48:44.190220 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.188933    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 18:48:44.190220 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.188935    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 18:48:44.190220 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.188939    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 18:48:44.190745 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.188942    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 18:48:44.190745 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.188945    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 18:48:44.190745 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.188949    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 18:48:44.190745 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.188953    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 18:48:44.190745 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.188956    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 18:48:44.190745 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.188958    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 18:48:44.190745 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.188961    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 18:48:44.190745 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.188963    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 18:48:44.190745 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.188967    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 18:48:44.190745 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.188970    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 18:48:44.190745 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.188972    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 18:48:44.190745 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.188975    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 18:48:44.190745 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.188978    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 18:48:44.190745 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.188980    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 18:48:44.190745 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.188983    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 18:48:44.190745 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.188985    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 18:48:44.190745 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.188987    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 18:48:44.190745 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.188990    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 18:48:44.190745 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.188992    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 18:48:44.190745 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.189339    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 18:48:44.191242 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.189343    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 18:48:44.191242 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.189346    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 18:48:44.191242 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.189349    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 18:48:44.191242 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.189351    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 18:48:44.191242 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.189354    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 18:48:44.191242 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.189356    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 18:48:44.191242 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.189359    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 18:48:44.191242 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.189362    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 18:48:44.191242 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.189364    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 18:48:44.191242 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.189366    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 18:48:44.191242 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.189371    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 18:48:44.191242 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.189376    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 18:48:44.191242 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.189379    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 18:48:44.191242 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.189382    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 18:48:44.191242 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.189385    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 18:48:44.191242 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.189389    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 18:48:44.191242 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.189392    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 18:48:44.191242 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.189394    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 18:48:44.191242 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.189397    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 18:48:44.191242 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.189400    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 18:48:44.191754 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.189402    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 18:48:44.191754 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.189405    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 18:48:44.191754 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.189408    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 18:48:44.191754 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.189411    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 18:48:44.191754 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.189414    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 18:48:44.191754 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.189416    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 18:48:44.191754 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.189419    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 18:48:44.191754 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.189422    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 18:48:44.191754 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.189425    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 18:48:44.191754 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.189427    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 18:48:44.191754 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.189430    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 18:48:44.191754 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.189433    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 18:48:44.191754 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.189435    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 18:48:44.191754 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.189438    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 18:48:44.191754 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.189441    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 18:48:44.191754 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.189443    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 18:48:44.191754 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.189446    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 18:48:44.191754 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.189449    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 18:48:44.191754 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.189452    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 18:48:44.191754 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.189455    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 18:48:44.192282 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.189457    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 18:48:44.192282 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.189460    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 18:48:44.192282 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.189462    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 18:48:44.192282 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.189465    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 18:48:44.192282 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.189468    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 18:48:44.192282 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.189470    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 18:48:44.192282 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.189473    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 18:48:44.192282 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.189475    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 18:48:44.192282 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.189478    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 18:48:44.192282 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.189481    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 18:48:44.192282 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.189484    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 18:48:44.192282 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.189488    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 18:48:44.192282 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.189491    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 18:48:44.192282 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.189494    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 18:48:44.192282 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.189497    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 18:48:44.192282 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.189499    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 18:48:44.192282 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.189502    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 18:48:44.192282 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.189504    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 18:48:44.192282 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.189506    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 18:48:44.192282 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.189509    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 18:48:44.192792 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.189511    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 18:48:44.192792 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.189514    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 18:48:44.192792 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.189516    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 18:48:44.192792 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.189519    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 18:48:44.192792 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.189521    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 18:48:44.192792 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.189523    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 18:48:44.192792 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.189526    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 18:48:44.192792 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.189530    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 18:48:44.192792 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.189532    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 18:48:44.192792 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.189535    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 18:48:44.192792 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.189537    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 18:48:44.192792 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.189540    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 18:48:44.192792 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.189542    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 18:48:44.192792 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.189545    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 18:48:44.192792 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.189547    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 18:48:44.192792 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.189550    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 18:48:44.192792 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.189553    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 18:48:44.192792 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.189556    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 18:48:44.192792 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.189558    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 18:48:44.193252 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.189561    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 18:48:44.193252 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.189563    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 18:48:44.193252 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.189566    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 18:48:44.193252 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.189569    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 18:48:44.193252 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.189571    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 18:48:44.193252 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.189573    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 18:48:44.193252 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.189576    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 18:48:44.193252 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.189578    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 18:48:44.193252 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.189581    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 18:48:44.193252 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.189584    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 18:48:44.193252 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.189586    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 18:48:44.193252 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.189590    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 18:48:44.193252 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.189606    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 18:48:44.193252 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.189610    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 18:48:44.193252 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.189612    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 18:48:44.193252 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.189615    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 18:48:44.193252 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.189618    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 18:48:44.193252 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.189621    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 18:48:44.193252 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.189623    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 18:48:44.193252 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.189686    2578 flags.go:64] FLAG: --address="0.0.0.0"
Sep 11 18:48:44.193780 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.189724    2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 11 18:48:44.193780 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.189733    2578 flags.go:64] FLAG: --anonymous-auth="true"
Sep 11 18:48:44.193780 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.189737    2578 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 11 18:48:44.193780 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.189740    2578 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 11 18:48:44.193780 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.189743    2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 11 18:48:44.193780 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.189747    2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 11 18:48:44.193780 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.189751    2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 11 18:48:44.193780 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.189754    2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 11 18:48:44.193780 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.189757    2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 11 18:48:44.193780 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.189761    2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 11 18:48:44.193780 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.189764    2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 11 18:48:44.193780 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.189767    2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 11 18:48:44.193780 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.189770    2578 flags.go:64] FLAG: --cgroup-root=""
Sep 11 18:48:44.193780 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.189774    2578 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 11 18:48:44.193780 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.189777    2578 flags.go:64] FLAG: --client-ca-file=""
Sep 11 18:48:44.193780 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.189780    2578 flags.go:64] FLAG: --cloud-provider="external"
Sep 11 18:48:44.193780 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.189783    2578 flags.go:64] FLAG: --cluster-dns="[]"
Sep 11 18:48:44.193780 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.189788    2578 flags.go:64] FLAG: --cluster-domain=""
Sep 11 18:48:44.193780 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.189791    2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 11 18:48:44.193780 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.189794    2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 11 18:48:44.193780 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.189796    2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 11 18:48:44.193780 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.189800    2578 flags.go:64] FLAG: --container-log-max-files="5"
Sep 11 18:48:44.193780 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.189803    2578 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 11 18:48:44.193780 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.189806    2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 11 18:48:44.194355 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.189809    2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 11 18:48:44.194355 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.189812    2578 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 11 18:48:44.194355 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.189815    2578 flags.go:64] FLAG: --contention-profiling="false"
Sep 11 18:48:44.194355 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.189817    2578 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 11 18:48:44.194355 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.189820    2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 11 18:48:44.194355 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.189823    2578 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 11 18:48:44.194355 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.189826    2578 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 11 18:48:44.194355 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.189830    2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 11 18:48:44.194355 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.189833    2578 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 11 18:48:44.194355 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.189836    2578 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 11 18:48:44.194355 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.189839    2578 flags.go:64] FLAG: --enable-load-reader="false"
Sep 11 18:48:44.194355 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.189842    2578 flags.go:64] FLAG: --enable-server="true"
Sep 11 18:48:44.194355 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.189845    2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 11 18:48:44.194355 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.189848    2578 flags.go:64] FLAG: --event-burst="100"
Sep 11 18:48:44.194355 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.189851    2578 flags.go:64] FLAG: --event-qps="50"
Sep 11 18:48:44.194355 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.189854    2578 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 11 18:48:44.194355 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.189857    2578 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 11 18:48:44.194355 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.189860    2578 flags.go:64] FLAG: --eviction-hard=""
Sep 11 18:48:44.194355 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.189863    2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 11 18:48:44.194355 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.189866    2578 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 11 18:48:44.194355 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.189869    2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 11 18:48:44.194355 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.189872    2578 flags.go:64] FLAG: --eviction-soft=""
Sep 11 18:48:44.194355 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.189874    2578 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 11 18:48:44.194355 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.189877    2578 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 11 18:48:44.194355 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.189881    2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 11 18:48:44.194992 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.189886    2578 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 11 18:48:44.194992 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.189889    2578 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 11 18:48:44.194992 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.189892    2578 flags.go:64] FLAG: --fail-swap-on="true"
Sep 11 18:48:44.194992 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.189894    2578 flags.go:64] FLAG: --feature-gates=""
Sep 11 18:48:44.194992 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.189898    2578 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 11 18:48:44.194992 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.189901    2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 11 18:48:44.194992 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.189904    2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 11 18:48:44.194992 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.189907    2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 11 18:48:44.194992 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.189910    2578 flags.go:64] FLAG: --healthz-port="10248"
Sep 11 18:48:44.194992 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.189913    2578 flags.go:64] FLAG: --help="false"
Sep 11 18:48:44.194992 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.189916    2578 flags.go:64] FLAG: --hostname-override="ip-10-0-134-189.ec2.internal"
Sep 11 18:48:44.194992 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.189919    2578 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 11 18:48:44.194992 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.189921    2578 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 11 18:48:44.194992 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.189924    2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 11 18:48:44.194992 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.189927    2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 11 18:48:44.194992 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.189930    2578 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 11 18:48:44.194992 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.189933    2578 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 11 18:48:44.194992 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.189936    2578 flags.go:64] FLAG: --image-service-endpoint=""
Sep 11 18:48:44.194992 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.189939    2578 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 11 18:48:44.194992 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.189942    2578 flags.go:64] FLAG: --kube-api-burst="100"
Sep 11 18:48:44.194992 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.189945    2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 11 18:48:44.194992 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.189948    2578 flags.go:64] FLAG: --kube-api-qps="50"
Sep 11 18:48:44.194992 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.189950    2578 flags.go:64] FLAG: --kube-reserved=""
Sep 11 18:48:44.194992 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.189953    2578 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 11 18:48:44.195584 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.189956    2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 11 18:48:44.195584 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.189959    2578 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 11 18:48:44.195584 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.189962    2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 11 18:48:44.195584 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.189964    2578 flags.go:64] FLAG: --lock-file=""
Sep 11 18:48:44.195584 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.189967    2578 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 11 18:48:44.195584 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.189970    2578 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 11 18:48:44.195584 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.189972    2578 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 11 18:48:44.195584 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.189977    2578 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 11 18:48:44.195584 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.189986    2578 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 11 18:48:44.195584 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.189988    2578 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 11 18:48:44.195584 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.189991    2578 flags.go:64] FLAG: --logging-format="text"
Sep 11 18:48:44.195584 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.189994    2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 11 18:48:44.195584 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.189997    2578 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 11 18:48:44.195584 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.190000    2578 flags.go:64] FLAG: --manifest-url=""
Sep 11 18:48:44.195584 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.190003    2578 flags.go:64] FLAG: --manifest-url-header=""
Sep 11 18:48:44.195584 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.190007    2578 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 11 18:48:44.195584 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.190010    2578 flags.go:64] FLAG: --max-open-files="1000000"
Sep 11 18:48:44.195584 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.190013    2578 flags.go:64] FLAG: --max-pods="110"
Sep 11 18:48:44.195584 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.190016    2578 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 11 18:48:44.195584 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.190019    2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 11 18:48:44.195584 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.190022    2578 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 11 18:48:44.195584 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.190025    2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 11 18:48:44.195584 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.190028    2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 11 18:48:44.195584 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.190031    2578 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 11 18:48:44.195584 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.190034    2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 11 18:48:44.196234 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.190040    2578 flags.go:64] FLAG: --node-status-max-images="50"
Sep 11 18:48:44.196234 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.190043    2578 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 11 18:48:44.196234 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.190046    2578 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 11 18:48:44.196234 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.190049    2578 flags.go:64] FLAG: --pod-cidr=""
Sep 11 18:48:44.196234 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.190052    2578 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 11 18:48:44.196234 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.190057    2578 flags.go:64] FLAG: --pod-manifest-path=""
Sep 11 18:48:44.196234 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.190060    2578 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 11 18:48:44.196234 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.190063    2578 flags.go:64] FLAG: --pods-per-core="0"
Sep 11 18:48:44.196234 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.190066    2578 flags.go:64] FLAG: --port="10250"
Sep 11 18:48:44.196234 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.190068    2578 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 11 18:48:44.196234 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.190071    2578 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0fe5f89d161e661b8"
Sep 11 18:48:44.196234 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.190074    2578 flags.go:64] FLAG: --qos-reserved=""
Sep 11 18:48:44.196234 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.190077    2578 flags.go:64] FLAG: --read-only-port="10255"
Sep 11 18:48:44.196234 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.190080    2578 flags.go:64] FLAG: --register-node="true"
Sep 11 18:48:44.196234 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.190082    2578 flags.go:64] FLAG: --register-with-taints=""
Sep 11 18:48:44.196234 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.190087    2578 flags.go:64] FLAG: --registry-burst="10"
Sep 11 18:48:44.196234 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.190090    2578 flags.go:64] FLAG: --registry-qps="5"
Sep 11 18:48:44.196234 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.190093    2578 flags.go:64] FLAG: --reserved-cpus=""
Sep 11 18:48:44.196234 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.190095    2578 flags.go:64] FLAG: --reserved-memory=""
Sep 11 18:48:44.196234 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.190100    2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 11 18:48:44.196234 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.190102    2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 11 18:48:44.196234 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.190105    2578 flags.go:64] FLAG: --rotate-certificates="false"
Sep 11 18:48:44.196234 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.190108    2578 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 11 18:48:44.196234 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.190111    2578 flags.go:64] FLAG: --runonce="false"
Sep 11 18:48:44.196234 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.190113    2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 11 18:48:44.196952 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.190116    2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 11 18:48:44.196952 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.190119    2578 flags.go:64] FLAG: --seccomp-default="false"
Sep 11 18:48:44.196952 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.190122    2578 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 11 18:48:44.196952 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.190124    2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 11 18:48:44.196952 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.190127    2578 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 11 18:48:44.196952 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.190130    2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 11 18:48:44.196952 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.190133    2578 flags.go:64] FLAG: --storage-driver-password="root"
Sep 11 18:48:44.196952 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.190136    2578 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 11 18:48:44.196952 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.190139    2578 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 11 18:48:44.196952 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.190141    2578 flags.go:64] FLAG: --storage-driver-user="root"
Sep 11 18:48:44.196952 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.190144    2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 11 18:48:44.196952 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.190147    2578 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 11 18:48:44.196952 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.190149    2578 flags.go:64] FLAG: --system-cgroups=""
Sep 11 18:48:44.196952 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.190152    2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 11 18:48:44.196952 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.190157    2578 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 11 18:48:44.196952 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.190160    2578 flags.go:64] FLAG: --tls-cert-file=""
Sep 11 18:48:44.196952 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.190163    2578 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 11 18:48:44.196952 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.190167    2578 flags.go:64] FLAG: --tls-min-version=""
Sep 11 18:48:44.196952 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.190170    2578 flags.go:64] FLAG: --tls-private-key-file=""
Sep 11 18:48:44.196952 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.190172    2578 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 11 18:48:44.196952 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.190175    2578 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 11 18:48:44.196952 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.190178    2578 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 11 18:48:44.196952 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.190181    2578 flags.go:64] FLAG: --v="2"
Sep 11 18:48:44.196952 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.190190    2578 flags.go:64] FLAG: --version="false"
Sep 11 18:48:44.196952 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.190194    2578 flags.go:64] FLAG: --vmodule=""
Sep 11 18:48:44.197622 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.190197    2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 11 18:48:44.197622 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.190201    2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 11 18:48:44.197622 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.190306    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 18:48:44.197622 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.190310    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 18:48:44.197622 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.190312    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 18:48:44.197622 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.190315    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 18:48:44.197622 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.190318    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 18:48:44.197622 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.190320    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 18:48:44.197622 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.190322    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 18:48:44.197622 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.190325    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 18:48:44.197622 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.190327    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 18:48:44.197622 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.190330    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 18:48:44.197622 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.190332    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 18:48:44.197622 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.190335    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 18:48:44.197622 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.190337    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 18:48:44.197622 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.190340    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 18:48:44.197622 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.190343    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 18:48:44.197622 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.190345    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 18:48:44.197622 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.190348    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 18:48:44.198088 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.190352    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 18:48:44.198088 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.190354    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 18:48:44.198088 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.190357    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 18:48:44.198088 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.190360    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 18:48:44.198088 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.190363    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 18:48:44.198088 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.190366    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 18:48:44.198088 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.190368    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 18:48:44.198088 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.190371    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 18:48:44.198088 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.190373    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 18:48:44.198088 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.190376    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 18:48:44.198088 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.190380    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 18:48:44.198088 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.190382    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 18:48:44.198088 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.190385    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 18:48:44.198088 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.190388    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 18:48:44.198088 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.190390    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 18:48:44.198088 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.190392    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 18:48:44.198088 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.190395    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 18:48:44.198088 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.190398    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 18:48:44.198088 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.190400    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 18:48:44.198088 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.190403    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 18:48:44.198578 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.190405    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 18:48:44.198578 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.190408    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 18:48:44.198578 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.190410    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 18:48:44.198578 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.190413    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 18:48:44.198578 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.190415    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 18:48:44.198578 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.190418    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 18:48:44.198578 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.190420    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 18:48:44.198578 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.190423    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 18:48:44.198578 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.190426    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 18:48:44.198578 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.190428    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 18:48:44.198578 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.190430    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 18:48:44.198578 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.190433    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 18:48:44.198578 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.190437    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 18:48:44.198578 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.190440    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 18:48:44.198578 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.190442    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 18:48:44.198578 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.190445    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 18:48:44.198578 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.190447    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 18:48:44.198578 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.190450    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 18:48:44.198578 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.190452    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 18:48:44.198578 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.190455    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 18:48:44.199077 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.190458    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 18:48:44.199077 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.190460    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 18:48:44.199077 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.190464    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 18:48:44.199077 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.190467    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 18:48:44.199077 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.190471    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 18:48:44.199077 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.190474    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 18:48:44.199077 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.190477    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 18:48:44.199077 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.190480    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 18:48:44.199077 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.190483    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 18:48:44.199077 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.190486    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 18:48:44.199077 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.190489    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 18:48:44.199077 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.190491    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 18:48:44.199077 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.190494    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 18:48:44.199077 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.190497    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 18:48:44.199077 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.190499    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 18:48:44.199077 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.190501    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 18:48:44.199077 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.190504    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 18:48:44.199077 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.190506    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 18:48:44.199077 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.190509    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 18:48:44.199077 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.190512    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 18:48:44.199560 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.190516    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 18:48:44.199560 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.190520    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 18:48:44.199560 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.190524    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 18:48:44.199560 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.190527    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 18:48:44.199560 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.190531    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 18:48:44.199560 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.190534    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 18:48:44.199560 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.190536    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 18:48:44.199560 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.190539    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 18:48:44.199560 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.190542    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 18:48:44.199560 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.190544    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 18:48:44.199560 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.190547    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 18:48:44.199560 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.190549    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 18:48:44.199560 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.190552    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 18:48:44.199560 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.190554    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 18:48:44.199560 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.190557    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 18:48:44.199560 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.190560    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 18:48:44.199560 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.190562    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 18:48:44.199560 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.190565    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 18:48:44.199560 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.190568    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 18:48:44.200140 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.190571    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 18:48:44.200140 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.190573    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 18:48:44.200140 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.190576    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 18:48:44.200140 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.191136    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 18:48:44.200140 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.198777    2578 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 11 18:48:44.200140 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.198794    2578 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 11 18:48:44.200140 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.198814    2578 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 11 18:48:44.200140 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.198819    2578 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 11 18:48:44.200140 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.198872    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 18:48:44.200140 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.198877    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 18:48:44.200140 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.198880    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 18:48:44.200140 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.198884    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 18:48:44.200140 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.198887    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 18:48:44.200140 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.198889    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 18:48:44.200490 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.198892    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 18:48:44.200490 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.198896    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 18:48:44.200490 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.198899    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 18:48:44.200490 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.198902    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 18:48:44.200490 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.198905    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 18:48:44.200490 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.198907    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 18:48:44.200490 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.198910    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 18:48:44.200490 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.198913    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 18:48:44.200490 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.198915    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 18:48:44.200490 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.198918    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 18:48:44.200490 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.198920    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 18:48:44.200490 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.198923    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 18:48:44.200490 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.198926    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 18:48:44.200490 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.198928    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 18:48:44.200490 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.198931    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 18:48:44.200490 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.198933    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 18:48:44.200490 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.198936    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 18:48:44.200490 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.198938    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 18:48:44.200490 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.198941    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 18:48:44.200490 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.198944    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 18:48:44.201032 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.198946    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 18:48:44.201032 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.198949    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 18:48:44.201032 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.198951    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 18:48:44.201032 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.198955    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 18:48:44.201032 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.198958    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 18:48:44.201032 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.198961    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 18:48:44.201032 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.198965    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 18:48:44.201032 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.198968    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 18:48:44.201032 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.198971    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 18:48:44.201032 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.198974    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 18:48:44.201032 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.198977    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 18:48:44.201032 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.198980    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 18:48:44.201032 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.198983    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 18:48:44.201032 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.198985    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 18:48:44.201032 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.198988    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 18:48:44.201032 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.198990    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 18:48:44.201032 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.198993    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 18:48:44.201032 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.198996    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 18:48:44.201032 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.198998    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 18:48:44.201493 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199001    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 18:48:44.201493 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199003    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 18:48:44.201493 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199006    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 18:48:44.201493 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199009    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 18:48:44.201493 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199012    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 18:48:44.201493 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199014    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 18:48:44.201493 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199017    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 18:48:44.201493 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199019    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 18:48:44.201493 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199023    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 18:48:44.201493 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199026    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 18:48:44.201493 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199029    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 18:48:44.201493 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199032    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 18:48:44.201493 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199035    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 18:48:44.201493 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199037    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 18:48:44.201493 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199040    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 18:48:44.201493 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199042    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 18:48:44.201493 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199046    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 18:48:44.201493 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199049    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 18:48:44.201493 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199051    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 18:48:44.201493 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199054    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 18:48:44.201990 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199056    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 18:48:44.201990 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199059    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 18:48:44.201990 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199061    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 18:48:44.201990 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199064    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 18:48:44.201990 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199066    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 18:48:44.201990 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199069    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 18:48:44.201990 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199071    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 18:48:44.201990 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199074    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 18:48:44.201990 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199076    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 18:48:44.201990 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199079    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 18:48:44.201990 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199082    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 18:48:44.201990 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199084    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 18:48:44.201990 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199087    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 18:48:44.201990 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199090    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 18:48:44.201990 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199092    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 18:48:44.201990 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199095    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 18:48:44.201990 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199097    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 18:48:44.201990 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199100    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 18:48:44.201990 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199102    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 18:48:44.202448 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199104    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 18:48:44.202448 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199107    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 18:48:44.202448 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199110    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 18:48:44.202448 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199112    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 18:48:44.202448 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199115    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 18:48:44.202448 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199118    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 18:48:44.202448 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199120    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 18:48:44.202448 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199122    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 18:48:44.202448 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199125    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 18:48:44.202448 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199128    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 18:48:44.202448 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199130    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 18:48:44.202448 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199133    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 18:48:44.202448 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199136    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 18:48:44.202448 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199138    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 18:48:44.202448 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199141    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 18:48:44.202827 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.199146    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 18:48:44.202827 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199623    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 18:48:44.202827 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199627    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 18:48:44.202827 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199630    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 18:48:44.202827 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199633    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 18:48:44.202827 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199636    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 18:48:44.202827 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199638    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 18:48:44.202827 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199641    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 18:48:44.202827 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199643    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 18:48:44.202827 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199646    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 18:48:44.202827 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199648    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 18:48:44.202827 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199651    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 18:48:44.202827 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199654    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 18:48:44.202827 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199656    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 18:48:44.202827 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199659    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 18:48:44.203196 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199661    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 18:48:44.203196 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199664    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 18:48:44.203196 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199666    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 18:48:44.203196 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199669    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 18:48:44.203196 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199671    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 18:48:44.203196 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199675    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 18:48:44.203196 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199678    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 18:48:44.203196 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199680    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 18:48:44.203196 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199683    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 18:48:44.203196 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199686    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 18:48:44.203196 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199688    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 18:48:44.203196 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199691    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 18:48:44.203196 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199694    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 18:48:44.203196 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199696    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 18:48:44.203196 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199699    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 18:48:44.203196 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199701    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 18:48:44.203196 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199704    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 18:48:44.203196 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199707    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 18:48:44.203196 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199709    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 18:48:44.203674 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199711    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 18:48:44.203674 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199714    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 18:48:44.203674 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199716    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 18:48:44.203674 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199719    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 18:48:44.203674 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199721    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 18:48:44.203674 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199723    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 18:48:44.203674 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199725    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 18:48:44.203674 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199728    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 18:48:44.203674 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199730    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 18:48:44.203674 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199733    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 18:48:44.203674 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199736    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 18:48:44.203674 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199738    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 18:48:44.203674 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199740    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 18:48:44.203674 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199743    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 18:48:44.203674 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199745    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 18:48:44.203674 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199748    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 18:48:44.203674 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199750    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 18:48:44.203674 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199752    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 18:48:44.203674 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199755    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 18:48:44.203674 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199757    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 18:48:44.204179 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199759    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 18:48:44.204179 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199762    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 18:48:44.204179 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199765    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 18:48:44.204179 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199767    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 18:48:44.204179 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199770    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 18:48:44.204179 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199772    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 18:48:44.204179 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199774    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 18:48:44.204179 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199777    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 18:48:44.204179 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199780    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 18:48:44.204179 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199783    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 18:48:44.204179 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199785    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 18:48:44.204179 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199788    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 18:48:44.204179 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199790    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 18:48:44.204179 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199792    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 18:48:44.204179 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199795    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 18:48:44.204179 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199797    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 18:48:44.204179 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199800    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 18:48:44.204179 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199802    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 18:48:44.204179 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199805    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 18:48:44.204647 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199807    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 18:48:44.204647 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199810    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 18:48:44.204647 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199812    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 18:48:44.204647 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199815    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 18:48:44.204647 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199817    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 18:48:44.204647 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199820    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 18:48:44.204647 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199824    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 18:48:44.204647 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199827    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 18:48:44.204647 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199831    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 18:48:44.204647 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199833    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 18:48:44.204647 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199836    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 18:48:44.204647 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199839    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 18:48:44.204647 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199842    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 18:48:44.204647 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199846    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 18:48:44.204647 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199849    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 18:48:44.204647 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199852    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 18:48:44.204647 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199855    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 18:48:44.204647 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199858    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 18:48:44.204647 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199863    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 18:48:44.205125 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199865    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 18:48:44.205125 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199868    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 18:48:44.205125 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199871    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 18:48:44.205125 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199874    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 18:48:44.205125 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199876    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 18:48:44.205125 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199879    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 18:48:44.205125 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199881    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 18:48:44.205125 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:44.199884    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 18:48:44.205125 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.199888    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 18:48:44.205125 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.199995    2578 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 11 18:48:44.205125 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.201913    2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 11 18:48:44.205125 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.202675    2578 server.go:1019] "Starting client certificate rotation"
Sep 11 18:48:44.205125 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.202773    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 18:48:44.205125 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.202805    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 18:48:44.205472 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.204817    2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 18:48:44.207457 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.207429    2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 18:48:44.219840 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.219811    2578 log.go:25] "Validated CRI v1 runtime API"
Sep 11 18:48:44.222914 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.222901    2578 log.go:25] "Validated CRI v1 image API"
Sep 11 18:48:44.223740 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.223724    2578 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 11 18:48:44.225861 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.225842    2578 fs.go:135] Filesystem UUIDs: map[4c66f408-262e-477d-a3f8-4d649cda7e1a:/dev/nvme0n1p4 75366465-9556-4624-ad0b-b64a4ba934c2:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Sep 11 18:48:44.225919 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.225862    2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 11 18:48:44.228902 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.228886    2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 18:48:44.231391 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.231287    2578 manager.go:216] Machine: {Timestamp:2026-09-11 18:48:44.229546129 +0000 UTC m=+0.307136690 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3117794 MemoryCapacity:32812150784 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec25ff64fad70f4b258e028c88d67f51 SystemUUID:ec25ff64-fad7-0f4b-258e-028c88d67f51 BootID:ca652bdf-d543-4a56-8378-3a4d8bf5c967 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406073344 Type:vfs Inodes:4005389 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:e8:23:23:11:a9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:e8:23:23:11:a9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:66:a0:c0:2f:5f:8b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812150784 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 11 18:48:44.231391 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.231387    2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 11 18:48:44.231495 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.231459    2578 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 11 18:48:44.234625 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.234590    2578 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 11 18:48:44.234757 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.234628    2578 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-189.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 11 18:48:44.236036 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.236026    2578 topology_manager.go:138] "Creating topology manager with none policy"
Sep 11 18:48:44.236073 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.236039    2578 container_manager_linux.go:309] "Creating device plugin manager"
Sep 11 18:48:44.236073 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.236047    2578 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 18:48:44.236614 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.236605    2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 18:48:44.237361 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.237353    2578 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 11 18:48:44.240004 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.239993    2578 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 18:48:44.240125 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.240114    2578 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 11 18:48:44.240159 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.240143    2578 kubelet.go:486] "Attempting to sync node with API server"
Sep 11 18:48:44.240196 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.240168    2578 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 11 18:48:44.240196 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.240184    2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 11 18:48:44.240196 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.240196    2578 kubelet.go:398] "Adding apiserver pod source"
Sep 11 18:48:44.240320 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.240205    2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 11 18:48:44.240545 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.240534    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 18:48:44.240577 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.240555    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 18:48:44.242532 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.242505    2578 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 11 18:48:44.245867 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.245845    2578 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 11 18:48:44.245944 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.245878    2578 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 11 18:48:44.248178 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.248162    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 11 18:48:44.248238 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.248181    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 11 18:48:44.248238 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.248188    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 11 18:48:44.248238 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.248193    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 11 18:48:44.248238 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.248201    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 11 18:48:44.248238 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.248207    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 11 18:48:44.248238 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.248220    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 11 18:48:44.248238 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.248225    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 11 18:48:44.248238 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.248231    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 11 18:48:44.248238 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.248238    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 11 18:48:44.248475 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.248247    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 11 18:48:44.248475 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.248252    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 11 18:48:44.249754 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.249742    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 11 18:48:44.249754 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.249752    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 11 18:48:44.251186 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:48:44.251167    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-189.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 11 18:48:44.251186 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:48:44.251187    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 11 18:48:44.252943 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.252928    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-pfzzh"
Sep 11 18:48:44.253179 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.253159    2578 server.go:1268] "Started kubelet"
Sep 11 18:48:44.253286 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.253246    2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 11 18:48:44.253331 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.253308    2578 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 11 18:48:44.253415 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.253393    2578 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 11 18:48:44.253911 ip-10-0-134-189 systemd[1]: Started Kubernetes Kubelet.
Sep 11 18:48:44.254499 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.254485    2578 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 11 18:48:44.254708 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.254648    2578 server.go:310] "Adding debug handlers to kubelet server"
Sep 11 18:48:44.258193 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:48:44.258091    2578 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 11 18:48:44.259530 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.259516    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 11 18:48:44.260008 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.259992    2578 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 11 18:48:44.260300 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:48:44.260160    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-189.ec2.internal\" not found"
Sep 11 18:48:44.261127 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.260735    2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 11 18:48:44.261127 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.260735    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-189.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 11 18:48:44.261127 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.260751    2578 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 11 18:48:44.261127 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.260772    2578 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 11 18:48:44.261127 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.260896    2578 reconstruct.go:97] "Volume reconstruction finished"
Sep 11 18:48:44.261127 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.260905    2578 reconciler.go:29] "Reconciler: start to sync state"
Sep 11 18:48:44.261127 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.260976    2578 factory.go:153] Registering CRI-O factory
Sep 11 18:48:44.261127 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.260995    2578 factory.go:223] Registration of the crio container factory successfully
Sep 11 18:48:44.261127 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.261039    2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 11 18:48:44.261127 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.261049    2578 factory.go:55] Registering systemd factory
Sep 11 18:48:44.261127 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.261056    2578 factory.go:223] Registration of the systemd container factory successfully
Sep 11 18:48:44.261127 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.261082    2578 factory.go:103] Registering Raw factory
Sep 11 18:48:44.261127 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.261094    2578 manager.go:1196] Started watching for new ooms in manager
Sep 11 18:48:44.261127 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.261121    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-pfzzh"
Sep 11 18:48:44.261893 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.261383    2578 manager.go:349] Starting recovery of all containers
Sep 11 18:48:44.261893 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:48:44.260775    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-189.ec2.internal.18d458bdd119f650  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-189.ec2.internal,UID:ip-10-0-134-189.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-189.ec2.internal,},FirstTimestamp:2026-09-11 18:48:44.253132368 +0000 UTC m=+0.330722913,LastTimestamp:2026-09-11 18:48:44.253132368 +0000 UTC m=+0.330722913,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-189.ec2.internal,}"
Sep 11 18:48:44.268191 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.268174    2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 18:48:44.272046 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:48:44.272022    2578 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-134-189.ec2.internal\" not found" node="ip-10-0-134-189.ec2.internal"
Sep 11 18:48:44.273221 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.273203    2578 manager.go:354] Recovery completed
Sep 11 18:48:44.277609 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.277582    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 18:48:44.279202 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.279181    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-189.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 18:48:44.279261 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.279211    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-189.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 18:48:44.279261 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.279223    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-189.ec2.internal" event="NodeHasSufficientPID"
Sep 11 18:48:44.279766 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.279752    2578 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 11 18:48:44.279766 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.279765    2578 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 11 18:48:44.279892 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.279773    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 18:48:44.279892 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.279784    2578 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 18:48:44.280382 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.280366    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-189.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 18:48:44.280464 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.280393    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-189.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 18:48:44.280464 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.280405    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-189.ec2.internal" event="NodeHasSufficientPID"
Sep 11 18:48:44.281167 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.281156    2578 policy_none.go:49] "None policy: Start"
Sep 11 18:48:44.281208 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.281172    2578 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 11 18:48:44.281208 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.281181    2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 11 18:48:44.282263 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.282253    2578 policy_none.go:47] "Start"
Sep 11 18:48:44.306190 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.306161    2578 manager.go:337] "Starting Device Plugin manager"
Sep 11 18:48:44.306298 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:48:44.306193    2578 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 11 18:48:44.306298 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.306203    2578 server.go:87] "Starting device plugin registration server"
Sep 11 18:48:44.306483 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.306469    2578 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 11 18:48:44.306543 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.306486    2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 11 18:48:44.306792 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.306660    2578 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 11 18:48:44.306792 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.306740    2578 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 11 18:48:44.306792 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.306748    2578 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 11 18:48:44.307350 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:48:44.307284    2578 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 11 18:48:44.307350 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:48:44.307334    2578 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-189.ec2.internal\" not found"
Sep 11 18:48:44.388608 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.388551    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 11 18:48:44.389932 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.389916    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 11 18:48:44.389990 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.389973    2578 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 11 18:48:44.390027 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.390001    2578 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 11 18:48:44.390068 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:48:44.390047    2578 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 11 18:48:44.392634 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.392589    2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 18:48:44.406981 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.406967    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 18:48:44.407832 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.407813    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-189.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 18:48:44.407921 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.407840    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-189.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 18:48:44.407921 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.407851    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-189.ec2.internal" event="NodeHasSufficientPID"
Sep 11 18:48:44.407921 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.407874    2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-189.ec2.internal"
Sep 11 18:48:44.416367 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.416351    2578 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-134-189.ec2.internal"
Sep 11 18:48:44.416428 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:48:44.416374    2578 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-189.ec2.internal\": node \"ip-10-0-134-189.ec2.internal\" not found"
Sep 11 18:48:44.427768 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:48:44.427747    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-189.ec2.internal\" not found"
Sep 11 18:48:44.490333 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.490296    2578 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-189.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-189.ec2.internal"]
Sep 11 18:48:44.490435 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.490378    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 18:48:44.491232 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.491214    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-189.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 18:48:44.491348 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.491249    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-189.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 18:48:44.491348 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.491264    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-189.ec2.internal" event="NodeHasSufficientPID"
Sep 11 18:48:44.492574 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.492561    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 18:48:44.492718 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.492703    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-189.ec2.internal"
Sep 11 18:48:44.492781 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.492763    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 18:48:44.493300 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.493286    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-189.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 18:48:44.493366 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.493315    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-189.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 18:48:44.493366 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.493330    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-189.ec2.internal" event="NodeHasSufficientPID"
Sep 11 18:48:44.493366 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.493336    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-189.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 18:48:44.493366 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.493355    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-189.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 18:48:44.493366 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.493364    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-189.ec2.internal" event="NodeHasSufficientPID"
Sep 11 18:48:44.498235 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.498141    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-189.ec2.internal"
Sep 11 18:48:44.498235 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.498164    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 18:48:44.498835 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.498820    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-189.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 18:48:44.498909 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.498843    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-189.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 18:48:44.498909 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.498854    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-189.ec2.internal" event="NodeHasSufficientPID"
Sep 11 18:48:44.511131 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:48:44.511106    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-189.ec2.internal\" not found" node="ip-10-0-134-189.ec2.internal"
Sep 11 18:48:44.516874 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:48:44.516859    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-189.ec2.internal\" not found" node="ip-10-0-134-189.ec2.internal"
Sep 11 18:48:44.527924 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:48:44.527909    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-189.ec2.internal\" not found"
Sep 11 18:48:44.562943 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.562920    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4bf36dcd150da66bbf96efc4d354ac7b-config\") pod \"kube-apiserver-proxy-ip-10-0-134-189.ec2.internal\" (UID: \"4bf36dcd150da66bbf96efc4d354ac7b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-189.ec2.internal"
Sep 11 18:48:44.563029 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.562945    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4c97f908167f20c09da5c0e0087b3f35-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-189.ec2.internal\" (UID: \"4c97f908167f20c09da5c0e0087b3f35\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-189.ec2.internal"
Sep 11 18:48:44.563029 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.562965    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4c97f908167f20c09da5c0e0087b3f35-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-189.ec2.internal\" (UID: \"4c97f908167f20c09da5c0e0087b3f35\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-189.ec2.internal"
Sep 11 18:48:44.627999 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:48:44.627977    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-189.ec2.internal\" not found"
Sep 11 18:48:44.663267 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.663244    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4c97f908167f20c09da5c0e0087b3f35-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-189.ec2.internal\" (UID: \"4c97f908167f20c09da5c0e0087b3f35\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-189.ec2.internal"
Sep 11 18:48:44.663353 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.663271    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4bf36dcd150da66bbf96efc4d354ac7b-config\") pod \"kube-apiserver-proxy-ip-10-0-134-189.ec2.internal\" (UID: \"4bf36dcd150da66bbf96efc4d354ac7b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-189.ec2.internal"
Sep 11 18:48:44.663353 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.663289    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4c97f908167f20c09da5c0e0087b3f35-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-189.ec2.internal\" (UID: \"4c97f908167f20c09da5c0e0087b3f35\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-189.ec2.internal"
Sep 11 18:48:44.663353 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.663320    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4c97f908167f20c09da5c0e0087b3f35-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-189.ec2.internal\" (UID: \"4c97f908167f20c09da5c0e0087b3f35\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-189.ec2.internal"
Sep 11 18:48:44.663353 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.663333    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4c97f908167f20c09da5c0e0087b3f35-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-189.ec2.internal\" (UID: \"4c97f908167f20c09da5c0e0087b3f35\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-189.ec2.internal"
Sep 11 18:48:44.663353 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.663339    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4bf36dcd150da66bbf96efc4d354ac7b-config\") pod \"kube-apiserver-proxy-ip-10-0-134-189.ec2.internal\" (UID: \"4bf36dcd150da66bbf96efc4d354ac7b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-189.ec2.internal"
Sep 11 18:48:44.728556 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:48:44.728515    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-189.ec2.internal\" not found"
Sep 11 18:48:44.814156 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.814089    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-189.ec2.internal"
Sep 11 18:48:44.819847 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:44.819832    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-189.ec2.internal"
Sep 11 18:48:44.829388 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:48:44.829370    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-189.ec2.internal\" not found"
Sep 11 18:48:44.929976 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:48:44.929930    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-189.ec2.internal\" not found"
Sep 11 18:48:45.030402 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:48:45.030368    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-189.ec2.internal\" not found"
Sep 11 18:48:45.130950 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:48:45.130883    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-189.ec2.internal\" not found"
Sep 11 18:48:45.202333 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:45.202300    2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 11 18:48:45.202753 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:45.202436    2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 11 18:48:45.202753 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:45.202470    2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 11 18:48:45.231612 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:48:45.231575    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-189.ec2.internal\" not found"
Sep 11 18:48:45.263667 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:45.263635    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-10 18:43:44 +0000 UTC" deadline="2028-06-16 15:07:04.683466834 +0000 UTC"
Sep 11 18:48:45.263667 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:45.263667    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15452h18m19.419803628s"
Sep 11 18:48:45.332236 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:48:45.332202    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-189.ec2.internal\" not found"
Sep 11 18:48:45.352773 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:45.352748    2578 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 18:48:45.360536 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:45.360515    2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-189.ec2.internal"
Sep 11 18:48:45.372833 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:45.372810    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 18:48:45.374989 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:45.374971    2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-189.ec2.internal"
Sep 11 18:48:45.383238 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:45.383185    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 18:48:45.387710 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:45.387682    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c97f908167f20c09da5c0e0087b3f35.slice/crio-b1548c81ca02449ced1ee6c7ee37938bb89eccf3bfdfba33bddb210e10a364d0 WatchSource:0}: Error finding container b1548c81ca02449ced1ee6c7ee37938bb89eccf3bfdfba33bddb210e10a364d0: Status 404 returned error can't find the container with id b1548c81ca02449ced1ee6c7ee37938bb89eccf3bfdfba33bddb210e10a364d0
Sep 11 18:48:45.387909 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:45.387895    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bf36dcd150da66bbf96efc4d354ac7b.slice/crio-b0ac8b21cccd450c60804972295290837f3a619d735dfcc3ac79987f2a7c45cd WatchSource:0}: Error finding container b0ac8b21cccd450c60804972295290837f3a619d735dfcc3ac79987f2a7c45cd: Status 404 returned error can't find the container with id b0ac8b21cccd450c60804972295290837f3a619d735dfcc3ac79987f2a7c45cd
Sep 11 18:48:45.392207 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:45.392193    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 18:48:45.393298 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:45.393258    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-189.ec2.internal" event={"ID":"4bf36dcd150da66bbf96efc4d354ac7b","Type":"ContainerStarted","Data":"b0ac8b21cccd450c60804972295290837f3a619d735dfcc3ac79987f2a7c45cd"}
Sep 11 18:48:45.394639 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:45.394620    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-189.ec2.internal" event={"ID":"4c97f908167f20c09da5c0e0087b3f35","Type":"ContainerStarted","Data":"b1548c81ca02449ced1ee6c7ee37938bb89eccf3bfdfba33bddb210e10a364d0"}
Sep 11 18:48:45.444197 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:45.444171    2578 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 18:48:46.055454 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.055424    2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 18:48:46.240820 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.240776    2578 apiserver.go:52] "Watching apiserver"
Sep 11 18:48:46.249503 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.249476    2578 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 11 18:48:46.249980 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.249914    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-dnzn9","openshift-image-registry/node-ca-wmk8r","openshift-multus/multus-additional-cni-plugins-dc7p2","openshift-multus/multus-dcbq8","openshift-multus/network-metrics-daemon-bjcnp","openshift-network-diagnostics/network-check-target-7c8ns","kube-system/global-pull-secret-syncer-vnkmt","kube-system/kube-apiserver-proxy-ip-10-0-134-189.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plbjn","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-189.ec2.internal","openshift-network-operator/iptables-alerter-hh9vt","openshift-ovn-kubernetes/ovnkube-node-7xkpg","kube-system/konnectivity-agent-8lq2v","openshift-cluster-node-tuning-operator/tuned-fqd6d"]
Sep 11 18:48:46.251640 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.251617    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hh9vt"
Sep 11 18:48:46.253166 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.253141    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-wmk8r"
Sep 11 18:48:46.253938 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.253908    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 11 18:48:46.254485 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.254227    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 11 18:48:46.254485 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.254271    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-9j8fz\""
Sep 11 18:48:46.254485 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.254316    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 11 18:48:46.254703 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.254545    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dc7p2"
Sep 11 18:48:46.255106 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.255085    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 11 18:48:46.255502 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.255484    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-kml6w\""
Sep 11 18:48:46.255637 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.255620    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 11 18:48:46.256026 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.256007    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 11 18:48:46.256328 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.256311    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fqd6d"
Sep 11 18:48:46.257518 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.257497    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 11 18:48:46.258519 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.257767    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bjcnp"
Sep 11 18:48:46.258519 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.257823    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 11 18:48:46.258519 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:48:46.257840    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bjcnp" podUID="8498e9d1-1eeb-4ce9-b225-d18452e49150"
Sep 11 18:48:46.258519 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.257895    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 11 18:48:46.258519 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.257772    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-rx5lr\""
Sep 11 18:48:46.258519 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.258036    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 11 18:48:46.258519 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.258092    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 11 18:48:46.258519 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.258194    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-nnv5w\""
Sep 11 18:48:46.258519 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.258199    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 11 18:48:46.258519 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.258301    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7c8ns"
Sep 11 18:48:46.258519 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:48:46.258356    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7c8ns" podUID="c7382740-2bba-4368-8e7f-508c64d170a7"
Sep 11 18:48:46.258519 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.258424    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 11 18:48:46.259668 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.259586    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dnzn9"
Sep 11 18:48:46.260059 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.260041    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 11 18:48:46.261254 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.260930    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8lq2v"
Sep 11 18:48:46.262494 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.262444    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plbjn"
Sep 11 18:48:46.263040 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.262858    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 11 18:48:46.263040 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.262866    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-s9nm6\""
Sep 11 18:48:46.263040 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.262982    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 11 18:48:46.263332 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.263273    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 11 18:48:46.263680 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.263432    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-mgfth\""
Sep 11 18:48:46.263680 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.263583    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 11 18:48:46.264088 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.264072    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7xkpg"
Sep 11 18:48:46.264181 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.264114    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 11 18:48:46.264422 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.264393    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 11 18:48:46.264827 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.264810    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 11 18:48:46.264827 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.264809    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rvf2k\""
Sep 11 18:48:46.265879 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.265858    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnkmt"
Sep 11 18:48:46.265973 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:48:46.265919    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vnkmt" podUID="90acf5d2-d7df-46bb-a380-73f3ae079b75"
Sep 11 18:48:46.266218 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.266199    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 11 18:48:46.266458 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.266418    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 11 18:48:46.266945 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.266925    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 11 18:48:46.267204 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.267182    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 11 18:48:46.267416 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.267394    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-lw4mw\""
Sep 11 18:48:46.267416 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.267404    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dcbq8"
Sep 11 18:48:46.267607 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.267591    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 11 18:48:46.268081 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.268065    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 11 18:48:46.269041 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.269025    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 11 18:48:46.269638 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.269620    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-5x2tk\""
Sep 11 18:48:46.272548 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.272530    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c088609c-51c3-4070-88eb-a329ef4204a7-cni-binary-copy\") pod \"multus-additional-cni-plugins-dc7p2\" (UID: \"c088609c-51c3-4070-88eb-a329ef4204a7\") " pod="openshift-multus/multus-additional-cni-plugins-dc7p2"
Sep 11 18:48:46.272677 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.272559    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/679f1bfe-400c-46c5-837e-39d1dccccf5f-agent-certs\") pod \"konnectivity-agent-8lq2v\" (UID: \"679f1bfe-400c-46c5-837e-39d1dccccf5f\") " pod="kube-system/konnectivity-agent-8lq2v"
Sep 11 18:48:46.272677 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.272584    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4c6ece11-f761-4493-ba21-fa07150f8ec3-sys-fs\") pod \"aws-ebs-csi-driver-node-plbjn\" (UID: \"4c6ece11-f761-4493-ba21-fa07150f8ec3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plbjn"
Sep 11 18:48:46.272677 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.272608    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78kr7\" (UniqueName: \"kubernetes.io/projected/0905525e-5ad4-4f0a-882d-d9d55c922e0b-kube-api-access-78kr7\") pod \"iptables-alerter-hh9vt\" (UID: \"0905525e-5ad4-4f0a-882d-d9d55c922e0b\") " pod="openshift-network-operator/iptables-alerter-hh9vt"
Sep 11 18:48:46.272975 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.272931    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/107db790-53d5-4969-8bf9-2ee3b9f9df09-serviceca\") pod \"node-ca-wmk8r\" (UID: \"107db790-53d5-4969-8bf9-2ee3b9f9df09\") " pod="openshift-image-registry/node-ca-wmk8r"
Sep 11 18:48:46.273072 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.272994    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tl8kx\" (UniqueName: \"kubernetes.io/projected/107db790-53d5-4969-8bf9-2ee3b9f9df09-kube-api-access-tl8kx\") pod \"node-ca-wmk8r\" (UID: \"107db790-53d5-4969-8bf9-2ee3b9f9df09\") " pod="openshift-image-registry/node-ca-wmk8r"
Sep 11 18:48:46.273072 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.273047    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8h4f\" (UniqueName: \"kubernetes.io/projected/c7382740-2bba-4368-8e7f-508c64d170a7-kube-api-access-f8h4f\") pod \"network-check-target-7c8ns\" (UID: \"c7382740-2bba-4368-8e7f-508c64d170a7\") " pod="openshift-network-diagnostics/network-check-target-7c8ns"
Sep 11 18:48:46.273190 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.273091    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4c6ece11-f761-4493-ba21-fa07150f8ec3-device-dir\") pod \"aws-ebs-csi-driver-node-plbjn\" (UID: \"4c6ece11-f761-4493-ba21-fa07150f8ec3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plbjn"
Sep 11 18:48:46.273190 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.273121    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c088609c-51c3-4070-88eb-a329ef4204a7-cnibin\") pod \"multus-additional-cni-plugins-dc7p2\" (UID: \"c088609c-51c3-4070-88eb-a329ef4204a7\") " pod="openshift-multus/multus-additional-cni-plugins-dc7p2"
Sep 11 18:48:46.273190 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.273149    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c088609c-51c3-4070-88eb-a329ef4204a7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dc7p2\" (UID: \"c088609c-51c3-4070-88eb-a329ef4204a7\") " pod="openshift-multus/multus-additional-cni-plugins-dc7p2"
Sep 11 18:48:46.273190 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.273177    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkk6g\" (UniqueName: \"kubernetes.io/projected/c088609c-51c3-4070-88eb-a329ef4204a7-kube-api-access-rkk6g\") pod \"multus-additional-cni-plugins-dc7p2\" (UID: \"c088609c-51c3-4070-88eb-a329ef4204a7\") " pod="openshift-multus/multus-additional-cni-plugins-dc7p2"
Sep 11 18:48:46.273516 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.273203    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/240ee9ec-9e58-435f-938f-62cf71acfb5b-etc-systemd\") pod \"tuned-fqd6d\" (UID: \"240ee9ec-9e58-435f-938f-62cf71acfb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-fqd6d"
Sep 11 18:48:46.273516 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.273226    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/240ee9ec-9e58-435f-938f-62cf71acfb5b-run\") pod \"tuned-fqd6d\" (UID: \"240ee9ec-9e58-435f-938f-62cf71acfb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-fqd6d"
Sep 11 18:48:46.273516 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.273251    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nphws\" (UniqueName: \"kubernetes.io/projected/43683762-3da6-4f69-a5ad-d685d63bbe4f-kube-api-access-nphws\") pod \"node-resolver-dnzn9\" (UID: \"43683762-3da6-4f69-a5ad-d685d63bbe4f\") " pod="openshift-dns/node-resolver-dnzn9"
Sep 11 18:48:46.273516 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.273278    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7nng\" (UniqueName: \"kubernetes.io/projected/8498e9d1-1eeb-4ce9-b225-d18452e49150-kube-api-access-r7nng\") pod \"network-metrics-daemon-bjcnp\" (UID: \"8498e9d1-1eeb-4ce9-b225-d18452e49150\") " pod="openshift-multus/network-metrics-daemon-bjcnp"
Sep 11 18:48:46.273516 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.273316    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/240ee9ec-9e58-435f-938f-62cf71acfb5b-lib-modules\") pod \"tuned-fqd6d\" (UID: \"240ee9ec-9e58-435f-938f-62cf71acfb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-fqd6d"
Sep 11 18:48:46.273516 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.273343    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hrvd\" (UniqueName: \"kubernetes.io/projected/240ee9ec-9e58-435f-938f-62cf71acfb5b-kube-api-access-9hrvd\") pod \"tuned-fqd6d\" (UID: \"240ee9ec-9e58-435f-938f-62cf71acfb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-fqd6d"
Sep 11 18:48:46.273516 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.273369    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/679f1bfe-400c-46c5-837e-39d1dccccf5f-konnectivity-ca\") pod \"konnectivity-agent-8lq2v\" (UID: \"679f1bfe-400c-46c5-837e-39d1dccccf5f\") " pod="kube-system/konnectivity-agent-8lq2v"
Sep 11 18:48:46.273516 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.273401    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4c6ece11-f761-4493-ba21-fa07150f8ec3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-plbjn\" (UID: \"4c6ece11-f761-4493-ba21-fa07150f8ec3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plbjn"
Sep 11 18:48:46.273516 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.273429    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4c6ece11-f761-4493-ba21-fa07150f8ec3-registration-dir\") pod \"aws-ebs-csi-driver-node-plbjn\" (UID: \"4c6ece11-f761-4493-ba21-fa07150f8ec3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plbjn"
Sep 11 18:48:46.273516 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.273459    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0905525e-5ad4-4f0a-882d-d9d55c922e0b-iptables-alerter-script\") pod \"iptables-alerter-hh9vt\" (UID: \"0905525e-5ad4-4f0a-882d-d9d55c922e0b\") " pod="openshift-network-operator/iptables-alerter-hh9vt"
Sep 11 18:48:46.273516 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.273483    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8498e9d1-1eeb-4ce9-b225-d18452e49150-metrics-certs\") pod \"network-metrics-daemon-bjcnp\" (UID: \"8498e9d1-1eeb-4ce9-b225-d18452e49150\") " pod="openshift-multus/network-metrics-daemon-bjcnp"
Sep 11 18:48:46.273516 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.273508    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c088609c-51c3-4070-88eb-a329ef4204a7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dc7p2\" (UID: \"c088609c-51c3-4070-88eb-a329ef4204a7\") " pod="openshift-multus/multus-additional-cni-plugins-dc7p2"
Sep 11 18:48:46.274144 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.273531    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/240ee9ec-9e58-435f-938f-62cf71acfb5b-etc-sysconfig\") pod \"tuned-fqd6d\" (UID: \"240ee9ec-9e58-435f-938f-62cf71acfb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-fqd6d"
Sep 11 18:48:46.274144 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.273554    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/240ee9ec-9e58-435f-938f-62cf71acfb5b-etc-sysctl-d\") pod \"tuned-fqd6d\" (UID: \"240ee9ec-9e58-435f-938f-62cf71acfb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-fqd6d"
Sep 11 18:48:46.274144 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.273577    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/43683762-3da6-4f69-a5ad-d685d63bbe4f-hosts-file\") pod \"node-resolver-dnzn9\" (UID: \"43683762-3da6-4f69-a5ad-d685d63bbe4f\") " pod="openshift-dns/node-resolver-dnzn9"
Sep 11 18:48:46.274144 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.273601    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43683762-3da6-4f69-a5ad-d685d63bbe4f-tmp-dir\") pod \"node-resolver-dnzn9\" (UID: \"43683762-3da6-4f69-a5ad-d685d63bbe4f\") " pod="openshift-dns/node-resolver-dnzn9"
Sep 11 18:48:46.274144 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.273627    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c088609c-51c3-4070-88eb-a329ef4204a7-system-cni-dir\") pod \"multus-additional-cni-plugins-dc7p2\" (UID: \"c088609c-51c3-4070-88eb-a329ef4204a7\") " pod="openshift-multus/multus-additional-cni-plugins-dc7p2"
Sep 11 18:48:46.274144 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.273227    2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 18:48:46.274144 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.273701    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/240ee9ec-9e58-435f-938f-62cf71acfb5b-etc-sysctl-conf\") pod \"tuned-fqd6d\" (UID: \"240ee9ec-9e58-435f-938f-62cf71acfb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-fqd6d"
Sep 11 18:48:46.274144 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.273729    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/240ee9ec-9e58-435f-938f-62cf71acfb5b-etc-tuned\") pod \"tuned-fqd6d\" (UID: \"240ee9ec-9e58-435f-938f-62cf71acfb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-fqd6d"
Sep 11 18:48:46.274144 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.273755    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4c6ece11-f761-4493-ba21-fa07150f8ec3-etc-selinux\") pod \"aws-ebs-csi-driver-node-plbjn\" (UID: \"4c6ece11-f761-4493-ba21-fa07150f8ec3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plbjn"
Sep 11 18:48:46.274144 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.273798    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c088609c-51c3-4070-88eb-a329ef4204a7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dc7p2\" (UID: \"c088609c-51c3-4070-88eb-a329ef4204a7\") " pod="openshift-multus/multus-additional-cni-plugins-dc7p2"
Sep 11 18:48:46.274144 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.273858    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/240ee9ec-9e58-435f-938f-62cf71acfb5b-etc-modprobe-d\") pod \"tuned-fqd6d\" (UID: \"240ee9ec-9e58-435f-938f-62cf71acfb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-fqd6d"
Sep 11 18:48:46.274144 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.273885    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/240ee9ec-9e58-435f-938f-62cf71acfb5b-sys\") pod \"tuned-fqd6d\" (UID: \"240ee9ec-9e58-435f-938f-62cf71acfb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-fqd6d"
Sep 11 18:48:46.274144 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.273902    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/240ee9ec-9e58-435f-938f-62cf71acfb5b-var-lib-kubelet\") pod \"tuned-fqd6d\" (UID: \"240ee9ec-9e58-435f-938f-62cf71acfb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-fqd6d"
Sep 11 18:48:46.274144 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.273926    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4c6ece11-f761-4493-ba21-fa07150f8ec3-socket-dir\") pod \"aws-ebs-csi-driver-node-plbjn\" (UID: \"4c6ece11-f761-4493-ba21-fa07150f8ec3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plbjn"
Sep 11 18:48:46.274144 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.273962    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/107db790-53d5-4969-8bf9-2ee3b9f9df09-host\") pod \"node-ca-wmk8r\" (UID: \"107db790-53d5-4969-8bf9-2ee3b9f9df09\") " pod="openshift-image-registry/node-ca-wmk8r"
Sep 11 18:48:46.274144 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.273994    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/240ee9ec-9e58-435f-938f-62cf71acfb5b-etc-kubernetes\") pod \"tuned-fqd6d\" (UID: \"240ee9ec-9e58-435f-938f-62cf71acfb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-fqd6d"
Sep 11 18:48:46.274144 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.274020    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/240ee9ec-9e58-435f-938f-62cf71acfb5b-host\") pod \"tuned-fqd6d\" (UID: \"240ee9ec-9e58-435f-938f-62cf71acfb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-fqd6d"
Sep 11 18:48:46.275032 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.274081    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/240ee9ec-9e58-435f-938f-62cf71acfb5b-tmp\") pod \"tuned-fqd6d\" (UID: \"240ee9ec-9e58-435f-938f-62cf71acfb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-fqd6d"
Sep 11 18:48:46.275032 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.274106    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2kst\" (UniqueName: \"kubernetes.io/projected/4c6ece11-f761-4493-ba21-fa07150f8ec3-kube-api-access-b2kst\") pod \"aws-ebs-csi-driver-node-plbjn\" (UID: \"4c6ece11-f761-4493-ba21-fa07150f8ec3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plbjn"
Sep 11 18:48:46.275032 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.274135    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0905525e-5ad4-4f0a-882d-d9d55c922e0b-host-slash\") pod \"iptables-alerter-hh9vt\" (UID: \"0905525e-5ad4-4f0a-882d-d9d55c922e0b\") " pod="openshift-network-operator/iptables-alerter-hh9vt"
Sep 11 18:48:46.304460 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.304437    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-zgcbr"
Sep 11 18:48:46.314266 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.314245    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-zgcbr"
Sep 11 18:48:46.361701 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.361675    2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 11 18:48:46.375304 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.375280    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/16f7a243-b894-446f-80f7-fe1ae5a8d66c-cnibin\") pod \"multus-dcbq8\" (UID: \"16f7a243-b894-446f-80f7-fe1ae5a8d66c\") " pod="openshift-multus/multus-dcbq8"
Sep 11 18:48:46.375422 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.375308    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/16f7a243-b894-446f-80f7-fe1ae5a8d66c-multus-socket-dir-parent\") pod \"multus-dcbq8\" (UID: \"16f7a243-b894-446f-80f7-fe1ae5a8d66c\") " pod="openshift-multus/multus-dcbq8"
Sep 11 18:48:46.375422 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.375341    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/16f7a243-b894-446f-80f7-fe1ae5a8d66c-host-var-lib-cni-bin\") pod \"multus-dcbq8\" (UID: \"16f7a243-b894-446f-80f7-fe1ae5a8d66c\") " pod="openshift-multus/multus-dcbq8"
Sep 11 18:48:46.375422 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.375382    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/16f7a243-b894-446f-80f7-fe1ae5a8d66c-hostroot\") pod \"multus-dcbq8\" (UID: \"16f7a243-b894-446f-80f7-fe1ae5a8d66c\") " pod="openshift-multus/multus-dcbq8"
Sep 11 18:48:46.375422 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.375418    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c088609c-51c3-4070-88eb-a329ef4204a7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dc7p2\" (UID: \"c088609c-51c3-4070-88eb-a329ef4204a7\") " pod="openshift-multus/multus-additional-cni-plugins-dc7p2"
Sep 11 18:48:46.375614 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.375447    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/240ee9ec-9e58-435f-938f-62cf71acfb5b-var-lib-kubelet\") pod \"tuned-fqd6d\" (UID: \"240ee9ec-9e58-435f-938f-62cf71acfb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-fqd6d"
Sep 11 18:48:46.375614 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.375480    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4c6ece11-f761-4493-ba21-fa07150f8ec3-socket-dir\") pod \"aws-ebs-csi-driver-node-plbjn\" (UID: \"4c6ece11-f761-4493-ba21-fa07150f8ec3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plbjn"
Sep 11 18:48:46.375614 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.375509    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/16f7a243-b894-446f-80f7-fe1ae5a8d66c-system-cni-dir\") pod \"multus-dcbq8\" (UID: \"16f7a243-b894-446f-80f7-fe1ae5a8d66c\") " pod="openshift-multus/multus-dcbq8"
Sep 11 18:48:46.375614 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.375513    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/240ee9ec-9e58-435f-938f-62cf71acfb5b-var-lib-kubelet\") pod \"tuned-fqd6d\" (UID: \"240ee9ec-9e58-435f-938f-62cf71acfb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-fqd6d"
Sep 11 18:48:46.375614 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.375536    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/240ee9ec-9e58-435f-938f-62cf71acfb5b-etc-kubernetes\") pod \"tuned-fqd6d\" (UID: \"240ee9ec-9e58-435f-938f-62cf71acfb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-fqd6d"
Sep 11 18:48:46.375614 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.375557    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0905525e-5ad4-4f0a-882d-d9d55c922e0b-host-slash\") pod \"iptables-alerter-hh9vt\" (UID: \"0905525e-5ad4-4f0a-882d-d9d55c922e0b\") " pod="openshift-network-operator/iptables-alerter-hh9vt"
Sep 11 18:48:46.375614 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.375578    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/679f1bfe-400c-46c5-837e-39d1dccccf5f-agent-certs\") pod \"konnectivity-agent-8lq2v\" (UID: \"679f1bfe-400c-46c5-837e-39d1dccccf5f\") " pod="kube-system/konnectivity-agent-8lq2v"
Sep 11 18:48:46.375614 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.375620    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4c6ece11-f761-4493-ba21-fa07150f8ec3-sys-fs\") pod \"aws-ebs-csi-driver-node-plbjn\" (UID: \"4c6ece11-f761-4493-ba21-fa07150f8ec3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plbjn"
Sep 11 18:48:46.376004 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.375645    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/23c2932c-66b4-4b05-825e-afbbc27f110e-host-cni-netd\") pod \"ovnkube-node-7xkpg\" (UID: \"23c2932c-66b4-4b05-825e-afbbc27f110e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xkpg"
Sep 11 18:48:46.376004 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.375654    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0905525e-5ad4-4f0a-882d-d9d55c922e0b-host-slash\") pod \"iptables-alerter-hh9vt\" (UID: \"0905525e-5ad4-4f0a-882d-d9d55c922e0b\") " pod="openshift-network-operator/iptables-alerter-hh9vt"
Sep 11 18:48:46.376004 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.375671    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/23c2932c-66b4-4b05-825e-afbbc27f110e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7xkpg\" (UID: \"23c2932c-66b4-4b05-825e-afbbc27f110e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xkpg"
Sep 11 18:48:46.376004 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.375686    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/240ee9ec-9e58-435f-938f-62cf71acfb5b-etc-kubernetes\") pod \"tuned-fqd6d\" (UID: \"240ee9ec-9e58-435f-938f-62cf71acfb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-fqd6d"
Sep 11 18:48:46.376004 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.375700    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/16f7a243-b894-446f-80f7-fe1ae5a8d66c-host-run-multus-certs\") pod \"multus-dcbq8\" (UID: \"16f7a243-b894-446f-80f7-fe1ae5a8d66c\") " pod="openshift-multus/multus-dcbq8"
Sep 11 18:48:46.376004 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.375691    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4c6ece11-f761-4493-ba21-fa07150f8ec3-sys-fs\") pod \"aws-ebs-csi-driver-node-plbjn\" (UID: \"4c6ece11-f761-4493-ba21-fa07150f8ec3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plbjn"
Sep 11 18:48:46.376004 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.375721    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckd8m\" (UniqueName: \"kubernetes.io/projected/16f7a243-b894-446f-80f7-fe1ae5a8d66c-kube-api-access-ckd8m\") pod \"multus-dcbq8\" (UID: \"16f7a243-b894-446f-80f7-fe1ae5a8d66c\") " pod="openshift-multus/multus-dcbq8"
Sep 11 18:48:46.376004 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.375747    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78kr7\" (UniqueName: \"kubernetes.io/projected/0905525e-5ad4-4f0a-882d-d9d55c922e0b-kube-api-access-78kr7\") pod \"iptables-alerter-hh9vt\" (UID: \"0905525e-5ad4-4f0a-882d-d9d55c922e0b\") " pod="openshift-network-operator/iptables-alerter-hh9vt"
Sep 11 18:48:46.376004 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.375774    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/107db790-53d5-4969-8bf9-2ee3b9f9df09-serviceca\") pod \"node-ca-wmk8r\" (UID: \"107db790-53d5-4969-8bf9-2ee3b9f9df09\") " pod="openshift-image-registry/node-ca-wmk8r"
Sep 11 18:48:46.376004 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.375794    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4c6ece11-f761-4493-ba21-fa07150f8ec3-socket-dir\") pod \"aws-ebs-csi-driver-node-plbjn\" (UID: \"4c6ece11-f761-4493-ba21-fa07150f8ec3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plbjn"
Sep 11 18:48:46.376004 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.375799    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl8kx\" (UniqueName: \"kubernetes.io/projected/107db790-53d5-4969-8bf9-2ee3b9f9df09-kube-api-access-tl8kx\") pod \"node-ca-wmk8r\" (UID: \"107db790-53d5-4969-8bf9-2ee3b9f9df09\") " pod="openshift-image-registry/node-ca-wmk8r"
Sep 11 18:48:46.376004 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.375841    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4c6ece11-f761-4493-ba21-fa07150f8ec3-device-dir\") pod \"aws-ebs-csi-driver-node-plbjn\" (UID: \"4c6ece11-f761-4493-ba21-fa07150f8ec3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plbjn"
Sep 11 18:48:46.376004 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.375869    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/90acf5d2-d7df-46bb-a380-73f3ae079b75-kubelet-config\") pod \"global-pull-secret-syncer-vnkmt\" (UID: \"90acf5d2-d7df-46bb-a380-73f3ae079b75\") " pod="kube-system/global-pull-secret-syncer-vnkmt"
Sep 11 18:48:46.376004 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.375895    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/16f7a243-b894-446f-80f7-fe1ae5a8d66c-multus-conf-dir\") pod \"multus-dcbq8\" (UID: \"16f7a243-b894-446f-80f7-fe1ae5a8d66c\") " pod="openshift-multus/multus-dcbq8"
Sep 11 18:48:46.376004 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.375924    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkk6g\" (UniqueName: \"kubernetes.io/projected/c088609c-51c3-4070-88eb-a329ef4204a7-kube-api-access-rkk6g\") pod \"multus-additional-cni-plugins-dc7p2\" (UID: \"c088609c-51c3-4070-88eb-a329ef4204a7\") " pod="openshift-multus/multus-additional-cni-plugins-dc7p2"
Sep 11 18:48:46.376004 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.375950    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nphws\" (UniqueName: \"kubernetes.io/projected/43683762-3da6-4f69-a5ad-d685d63bbe4f-kube-api-access-nphws\") pod \"node-resolver-dnzn9\" (UID: \"43683762-3da6-4f69-a5ad-d685d63bbe4f\") " pod="openshift-dns/node-resolver-dnzn9"
Sep 11 18:48:46.376004 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.375977    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4c6ece11-f761-4493-ba21-fa07150f8ec3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-plbjn\" (UID: \"4c6ece11-f761-4493-ba21-fa07150f8ec3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plbjn"
Sep 11 18:48:46.376854 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.376003    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/23c2932c-66b4-4b05-825e-afbbc27f110e-host-kubelet\") pod \"ovnkube-node-7xkpg\" (UID: \"23c2932c-66b4-4b05-825e-afbbc27f110e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xkpg"
Sep 11 18:48:46.376854 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.376010    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4c6ece11-f761-4493-ba21-fa07150f8ec3-device-dir\") pod \"aws-ebs-csi-driver-node-plbjn\" (UID: \"4c6ece11-f761-4493-ba21-fa07150f8ec3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plbjn"
Sep 11 18:48:46.376854 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.376028    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/23c2932c-66b4-4b05-825e-afbbc27f110e-var-lib-openvswitch\") pod \"ovnkube-node-7xkpg\" (UID: \"23c2932c-66b4-4b05-825e-afbbc27f110e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xkpg"
Sep 11 18:48:46.376854 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.376058    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/16f7a243-b894-446f-80f7-fe1ae5a8d66c-cni-binary-copy\") pod \"multus-dcbq8\" (UID: \"16f7a243-b894-446f-80f7-fe1ae5a8d66c\") " pod="openshift-multus/multus-dcbq8"
Sep 11 18:48:46.376854 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.376086    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/240ee9ec-9e58-435f-938f-62cf71acfb5b-lib-modules\") pod \"tuned-fqd6d\" (UID: \"240ee9ec-9e58-435f-938f-62cf71acfb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-fqd6d"
Sep 11 18:48:46.376854 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.376109    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c088609c-51c3-4070-88eb-a329ef4204a7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dc7p2\" (UID: \"c088609c-51c3-4070-88eb-a329ef4204a7\") " pod="openshift-multus/multus-additional-cni-plugins-dc7p2"
Sep 11 18:48:46.376854 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.376121    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hrvd\" (UniqueName: \"kubernetes.io/projected/240ee9ec-9e58-435f-938f-62cf71acfb5b-kube-api-access-9hrvd\") pod \"tuned-fqd6d\" (UID: \"240ee9ec-9e58-435f-938f-62cf71acfb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-fqd6d"
Sep 11 18:48:46.376854 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.376130    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4c6ece11-f761-4493-ba21-fa07150f8ec3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-plbjn\" (UID: \"4c6ece11-f761-4493-ba21-fa07150f8ec3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plbjn"
Sep 11 18:48:46.376854 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.376150    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4c6ece11-f761-4493-ba21-fa07150f8ec3-registration-dir\") pod \"aws-ebs-csi-driver-node-plbjn\" (UID: \"4c6ece11-f761-4493-ba21-fa07150f8ec3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plbjn"
Sep 11 18:48:46.376854 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.376184    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/23c2932c-66b4-4b05-825e-afbbc27f110e-systemd-units\") pod \"ovnkube-node-7xkpg\" (UID: \"23c2932c-66b4-4b05-825e-afbbc27f110e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xkpg"
Sep 11 18:48:46.376854 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.376217    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c088609c-51c3-4070-88eb-a329ef4204a7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dc7p2\" (UID: \"c088609c-51c3-4070-88eb-a329ef4204a7\") " pod="openshift-multus/multus-additional-cni-plugins-dc7p2"
Sep 11 18:48:46.376854 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.376228    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4c6ece11-f761-4493-ba21-fa07150f8ec3-registration-dir\") pod \"aws-ebs-csi-driver-node-plbjn\" (UID: \"4c6ece11-f761-4493-ba21-fa07150f8ec3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plbjn"
Sep 11 18:48:46.376854 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.376243    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/43683762-3da6-4f69-a5ad-d685d63bbe4f-hosts-file\") pod \"node-resolver-dnzn9\" (UID: \"43683762-3da6-4f69-a5ad-d685d63bbe4f\") " pod="openshift-dns/node-resolver-dnzn9"
Sep 11 18:48:46.376854 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.376280    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/107db790-53d5-4969-8bf9-2ee3b9f9df09-serviceca\") pod \"node-ca-wmk8r\" (UID: \"107db790-53d5-4969-8bf9-2ee3b9f9df09\") " pod="openshift-image-registry/node-ca-wmk8r"
Sep 11 18:48:46.376854 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.376431    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/43683762-3da6-4f69-a5ad-d685d63bbe4f-hosts-file\") pod \"node-resolver-dnzn9\" (UID: \"43683762-3da6-4f69-a5ad-d685d63bbe4f\") " pod="openshift-dns/node-resolver-dnzn9"
Sep 11 18:48:46.376854 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.376470    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/23c2932c-66b4-4b05-825e-afbbc27f110e-run-systemd\") pod \"ovnkube-node-7xkpg\" (UID: \"23c2932c-66b4-4b05-825e-afbbc27f110e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xkpg"
Sep 11 18:48:46.376854 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.376499    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/23c2932c-66b4-4b05-825e-afbbc27f110e-host-run-ovn-kubernetes\") pod \"ovnkube-node-7xkpg\" (UID: \"23c2932c-66b4-4b05-825e-afbbc27f110e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xkpg"
Sep 11 18:48:46.377492 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.376504    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/240ee9ec-9e58-435f-938f-62cf71acfb5b-lib-modules\") pod \"tuned-fqd6d\" (UID: \"240ee9ec-9e58-435f-938f-62cf71acfb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-fqd6d"
Sep 11 18:48:46.377492 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.376541    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/23c2932c-66b4-4b05-825e-afbbc27f110e-host-cni-bin\") pod \"ovnkube-node-7xkpg\" (UID: \"23c2932c-66b4-4b05-825e-afbbc27f110e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xkpg"
Sep 11 18:48:46.377492 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.376623    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/16f7a243-b894-446f-80f7-fe1ae5a8d66c-etc-kubernetes\") pod \"multus-dcbq8\" (UID: \"16f7a243-b894-446f-80f7-fe1ae5a8d66c\") " pod="openshift-multus/multus-dcbq8"
Sep 11 18:48:46.377492 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.376662    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/23c2932c-66b4-4b05-825e-afbbc27f110e-run-ovn\") pod \"ovnkube-node-7xkpg\" (UID: \"23c2932c-66b4-4b05-825e-afbbc27f110e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xkpg"
Sep 11 18:48:46.377492 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.376702    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c088609c-51c3-4070-88eb-a329ef4204a7-system-cni-dir\") pod \"multus-additional-cni-plugins-dc7p2\" (UID: \"c088609c-51c3-4070-88eb-a329ef4204a7\") " pod="openshift-multus/multus-additional-cni-plugins-dc7p2"
Sep 11 18:48:46.377492 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.376711    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c088609c-51c3-4070-88eb-a329ef4204a7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dc7p2\" (UID: \"c088609c-51c3-4070-88eb-a329ef4204a7\") " pod="openshift-multus/multus-additional-cni-plugins-dc7p2"
Sep 11 18:48:46.377492 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.376731    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/240ee9ec-9e58-435f-938f-62cf71acfb5b-etc-sysctl-conf\") pod \"tuned-fqd6d\" (UID: \"240ee9ec-9e58-435f-938f-62cf71acfb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-fqd6d"
Sep 11 18:48:46.377492 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.376758    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/240ee9ec-9e58-435f-938f-62cf71acfb5b-etc-tuned\") pod \"tuned-fqd6d\" (UID: \"240ee9ec-9e58-435f-938f-62cf71acfb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-fqd6d"
Sep 11 18:48:46.377492 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.376844    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c088609c-51c3-4070-88eb-a329ef4204a7-system-cni-dir\") pod \"multus-additional-cni-plugins-dc7p2\" (UID: \"c088609c-51c3-4070-88eb-a329ef4204a7\") " pod="openshift-multus/multus-additional-cni-plugins-dc7p2"
Sep 11 18:48:46.377492 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.376966    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/240ee9ec-9e58-435f-938f-62cf71acfb5b-etc-sysctl-conf\") pod \"tuned-fqd6d\" (UID: \"240ee9ec-9e58-435f-938f-62cf71acfb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-fqd6d"
Sep 11 18:48:46.377492 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.377008    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4c6ece11-f761-4493-ba21-fa07150f8ec3-etc-selinux\") pod \"aws-ebs-csi-driver-node-plbjn\" (UID: \"4c6ece11-f761-4493-ba21-fa07150f8ec3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plbjn"
Sep 11 18:48:46.377492 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.377040    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/23c2932c-66b4-4b05-825e-afbbc27f110e-run-openvswitch\") pod \"ovnkube-node-7xkpg\" (UID: \"23c2932c-66b4-4b05-825e-afbbc27f110e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xkpg"
Sep 11 18:48:46.377492 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.377061    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/16f7a243-b894-446f-80f7-fe1ae5a8d66c-multus-cni-dir\") pod \"multus-dcbq8\" (UID: \"16f7a243-b894-446f-80f7-fe1ae5a8d66c\") " pod="openshift-multus/multus-dcbq8"
Sep 11 18:48:46.377492 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.377091    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/16f7a243-b894-446f-80f7-fe1ae5a8d66c-host-var-lib-kubelet\") pod \"multus-dcbq8\" (UID: \"16f7a243-b894-446f-80f7-fe1ae5a8d66c\") " pod="openshift-multus/multus-dcbq8"
Sep 11 18:48:46.377492 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.377113    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/240ee9ec-9e58-435f-938f-62cf71acfb5b-etc-modprobe-d\") pod \"tuned-fqd6d\" (UID: \"240ee9ec-9e58-435f-938f-62cf71acfb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-fqd6d"
Sep 11 18:48:46.377492 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.377133    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/240ee9ec-9e58-435f-938f-62cf71acfb5b-sys\") pod \"tuned-fqd6d\" (UID: \"240ee9ec-9e58-435f-938f-62cf71acfb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-fqd6d"
Sep 11 18:48:46.377492 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.377220    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/240ee9ec-9e58-435f-938f-62cf71acfb5b-etc-modprobe-d\") pod \"tuned-fqd6d\" (UID: \"240ee9ec-9e58-435f-938f-62cf71acfb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-fqd6d"
Sep 11 18:48:46.378282 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.377246    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/23c2932c-66b4-4b05-825e-afbbc27f110e-node-log\") pod \"ovnkube-node-7xkpg\" (UID: \"23c2932c-66b4-4b05-825e-afbbc27f110e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xkpg"
Sep 11 18:48:46.378282 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.377273    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/16f7a243-b894-446f-80f7-fe1ae5a8d66c-host-var-lib-cni-multus\") pod \"multus-dcbq8\" (UID: \"16f7a243-b894-446f-80f7-fe1ae5a8d66c\") " pod="openshift-multus/multus-dcbq8"
Sep 11 18:48:46.378282 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.377297    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/107db790-53d5-4969-8bf9-2ee3b9f9df09-host\") pod \"node-ca-wmk8r\" (UID: \"107db790-53d5-4969-8bf9-2ee3b9f9df09\") " pod="openshift-image-registry/node-ca-wmk8r"
Sep 11 18:48:46.378282 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.377318    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/240ee9ec-9e58-435f-938f-62cf71acfb5b-host\") pod \"tuned-fqd6d\" (UID: \"240ee9ec-9e58-435f-938f-62cf71acfb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-fqd6d"
Sep 11 18:48:46.378282 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.377338    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/240ee9ec-9e58-435f-938f-62cf71acfb5b-tmp\") pod \"tuned-fqd6d\" (UID: \"240ee9ec-9e58-435f-938f-62cf71acfb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-fqd6d"
Sep 11 18:48:46.378282 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.377421    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2kst\" (UniqueName: \"kubernetes.io/projected/4c6ece11-f761-4493-ba21-fa07150f8ec3-kube-api-access-b2kst\") pod \"aws-ebs-csi-driver-node-plbjn\" (UID: \"4c6ece11-f761-4493-ba21-fa07150f8ec3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plbjn"
Sep 11 18:48:46.378282 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.377422    2578 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 11 18:48:46.378282 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.377451    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/23c2932c-66b4-4b05-825e-afbbc27f110e-etc-openvswitch\") pod \"ovnkube-node-7xkpg\" (UID: \"23c2932c-66b4-4b05-825e-afbbc27f110e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xkpg"
Sep 11 18:48:46.378282 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.377479    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/23c2932c-66b4-4b05-825e-afbbc27f110e-ovnkube-config\") pod \"ovnkube-node-7xkpg\" (UID: \"23c2932c-66b4-4b05-825e-afbbc27f110e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xkpg"
Sep 11 18:48:46.378282 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.377498    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c088609c-51c3-4070-88eb-a329ef4204a7-cni-binary-copy\") pod \"multus-additional-cni-plugins-dc7p2\" (UID: \"c088609c-51c3-4070-88eb-a329ef4204a7\") " pod="openshift-multus/multus-additional-cni-plugins-dc7p2"
Sep 11 18:48:46.378282 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.377497    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4c6ece11-f761-4493-ba21-fa07150f8ec3-etc-selinux\") pod \"aws-ebs-csi-driver-node-plbjn\" (UID: \"4c6ece11-f761-4493-ba21-fa07150f8ec3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plbjn"
Sep 11 18:48:46.378282 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.377526    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/23c2932c-66b4-4b05-825e-afbbc27f110e-log-socket\") pod \"ovnkube-node-7xkpg\" (UID: \"23c2932c-66b4-4b05-825e-afbbc27f110e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xkpg"
Sep 11 18:48:46.378282 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.377553    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/23c2932c-66b4-4b05-825e-afbbc27f110e-ovn-node-metrics-cert\") pod \"ovnkube-node-7xkpg\" (UID: \"23c2932c-66b4-4b05-825e-afbbc27f110e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xkpg"
Sep 11 18:48:46.378282 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.377576    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/16f7a243-b894-446f-80f7-fe1ae5a8d66c-host-run-k8s-cni-cncf-io\") pod \"multus-dcbq8\" (UID: \"16f7a243-b894-446f-80f7-fe1ae5a8d66c\") " pod="openshift-multus/multus-dcbq8"
Sep 11 18:48:46.378282 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.377632    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8h4f\" (UniqueName: \"kubernetes.io/projected/c7382740-2bba-4368-8e7f-508c64d170a7-kube-api-access-f8h4f\") pod \"network-check-target-7c8ns\" (UID: \"c7382740-2bba-4368-8e7f-508c64d170a7\") " pod="openshift-network-diagnostics/network-check-target-7c8ns"
Sep 11 18:48:46.378282 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.377686    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/23c2932c-66b4-4b05-825e-afbbc27f110e-host-slash\") pod \"ovnkube-node-7xkpg\" (UID: \"23c2932c-66b4-4b05-825e-afbbc27f110e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xkpg"
Sep 11 18:48:46.378282 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.377712    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/16f7a243-b894-446f-80f7-fe1ae5a8d66c-multus-daemon-config\") pod \"multus-dcbq8\" (UID: \"16f7a243-b894-446f-80f7-fe1ae5a8d66c\") " pod="openshift-multus/multus-dcbq8"
Sep 11 18:48:46.379013 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.377743    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c088609c-51c3-4070-88eb-a329ef4204a7-cnibin\") pod \"multus-additional-cni-plugins-dc7p2\" (UID: \"c088609c-51c3-4070-88eb-a329ef4204a7\") " pod="openshift-multus/multus-additional-cni-plugins-dc7p2"
Sep 11 18:48:46.379013 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.377760    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/107db790-53d5-4969-8bf9-2ee3b9f9df09-host\") pod \"node-ca-wmk8r\" (UID: \"107db790-53d5-4969-8bf9-2ee3b9f9df09\") " pod="openshift-image-registry/node-ca-wmk8r"
Sep 11 18:48:46.379013 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.377770    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c088609c-51c3-4070-88eb-a329ef4204a7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dc7p2\" (UID: \"c088609c-51c3-4070-88eb-a329ef4204a7\") " pod="openshift-multus/multus-additional-cni-plugins-dc7p2"
Sep 11 18:48:46.379013 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.377423    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/240ee9ec-9e58-435f-938f-62cf71acfb5b-host\") pod \"tuned-fqd6d\" (UID: \"240ee9ec-9e58-435f-938f-62cf71acfb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-fqd6d"
Sep 11 18:48:46.379013 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.377798    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/240ee9ec-9e58-435f-938f-62cf71acfb5b-etc-systemd\") pod \"tuned-fqd6d\" (UID: \"240ee9ec-9e58-435f-938f-62cf71acfb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-fqd6d"
Sep 11 18:48:46.379013 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.378030    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/240ee9ec-9e58-435f-938f-62cf71acfb5b-etc-systemd\") pod \"tuned-fqd6d\" (UID: \"240ee9ec-9e58-435f-938f-62cf71acfb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-fqd6d"
Sep 11 18:48:46.379013 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.378030    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/240ee9ec-9e58-435f-938f-62cf71acfb5b-sys\") pod \"tuned-fqd6d\" (UID: \"240ee9ec-9e58-435f-938f-62cf71acfb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-fqd6d"
Sep 11 18:48:46.379013 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.378065    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/240ee9ec-9e58-435f-938f-62cf71acfb5b-run\") pod \"tuned-fqd6d\" (UID: \"240ee9ec-9e58-435f-938f-62cf71acfb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-fqd6d"
Sep 11 18:48:46.379013 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.378094    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/679f1bfe-400c-46c5-837e-39d1dccccf5f-konnectivity-ca\") pod \"konnectivity-agent-8lq2v\" (UID: \"679f1bfe-400c-46c5-837e-39d1dccccf5f\") " pod="kube-system/konnectivity-agent-8lq2v"
Sep 11 18:48:46.379013 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.378121    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n94jg\" (UniqueName: \"kubernetes.io/projected/23c2932c-66b4-4b05-825e-afbbc27f110e-kube-api-access-n94jg\") pod \"ovnkube-node-7xkpg\" (UID: \"23c2932c-66b4-4b05-825e-afbbc27f110e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xkpg"
Sep 11 18:48:46.379013 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.378189    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c088609c-51c3-4070-88eb-a329ef4204a7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dc7p2\" (UID: \"c088609c-51c3-4070-88eb-a329ef4204a7\") " pod="openshift-multus/multus-additional-cni-plugins-dc7p2"
Sep 11 18:48:46.379013 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.378267    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7nng\" (UniqueName: \"kubernetes.io/projected/8498e9d1-1eeb-4ce9-b225-d18452e49150-kube-api-access-r7nng\") pod \"network-metrics-daemon-bjcnp\" (UID: \"8498e9d1-1eeb-4ce9-b225-d18452e49150\") " pod="openshift-multus/network-metrics-daemon-bjcnp"
Sep 11 18:48:46.379013 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.378277    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c088609c-51c3-4070-88eb-a329ef4204a7-cni-binary-copy\") pod \"multus-additional-cni-plugins-dc7p2\" (UID: \"c088609c-51c3-4070-88eb-a329ef4204a7\") " pod="openshift-multus/multus-additional-cni-plugins-dc7p2"
Sep 11 18:48:46.379013 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.378299    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/90acf5d2-d7df-46bb-a380-73f3ae079b75-original-pull-secret\") pod \"global-pull-secret-syncer-vnkmt\" (UID: \"90acf5d2-d7df-46bb-a380-73f3ae079b75\") " pod="kube-system/global-pull-secret-syncer-vnkmt"
Sep 11 18:48:46.379013 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.378331    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0905525e-5ad4-4f0a-882d-d9d55c922e0b-iptables-alerter-script\") pod \"iptables-alerter-hh9vt\" (UID: \"0905525e-5ad4-4f0a-882d-d9d55c922e0b\") " pod="openshift-network-operator/iptables-alerter-hh9vt"
Sep 11 18:48:46.379013 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.378358    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8498e9d1-1eeb-4ce9-b225-d18452e49150-metrics-certs\") pod \"network-metrics-daemon-bjcnp\" (UID: \"8498e9d1-1eeb-4ce9-b225-d18452e49150\") " pod="openshift-multus/network-metrics-daemon-bjcnp"
Sep 11 18:48:46.379013 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.378411    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/240ee9ec-9e58-435f-938f-62cf71acfb5b-etc-sysconfig\") pod \"tuned-fqd6d\" (UID: \"240ee9ec-9e58-435f-938f-62cf71acfb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-fqd6d"
Sep 11 18:48:46.380025 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.378437    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/240ee9ec-9e58-435f-938f-62cf71acfb5b-etc-sysctl-d\") pod \"tuned-fqd6d\" (UID: \"240ee9ec-9e58-435f-938f-62cf71acfb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-fqd6d"
Sep 11 18:48:46.380025 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.378462    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43683762-3da6-4f69-a5ad-d685d63bbe4f-tmp-dir\") pod \"node-resolver-dnzn9\" (UID: \"43683762-3da6-4f69-a5ad-d685d63bbe4f\") " pod="openshift-dns/node-resolver-dnzn9"
Sep 11 18:48:46.380025 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.378492    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/23c2932c-66b4-4b05-825e-afbbc27f110e-host-run-netns\") pod \"ovnkube-node-7xkpg\" (UID: \"23c2932c-66b4-4b05-825e-afbbc27f110e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xkpg"
Sep 11 18:48:46.380025 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.378517    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/16f7a243-b894-446f-80f7-fe1ae5a8d66c-host-run-netns\") pod \"multus-dcbq8\" (UID: \"16f7a243-b894-446f-80f7-fe1ae5a8d66c\") " pod="openshift-multus/multus-dcbq8"
Sep 11 18:48:46.380025 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.378545    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/23c2932c-66b4-4b05-825e-afbbc27f110e-env-overrides\") pod \"ovnkube-node-7xkpg\" (UID: \"23c2932c-66b4-4b05-825e-afbbc27f110e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xkpg"
Sep 11 18:48:46.380025 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.378571    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/23c2932c-66b4-4b05-825e-afbbc27f110e-ovnkube-script-lib\") pod \"ovnkube-node-7xkpg\" (UID: \"23c2932c-66b4-4b05-825e-afbbc27f110e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xkpg"
Sep 11 18:48:46.380025 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.378618    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c088609c-51c3-4070-88eb-a329ef4204a7-cnibin\") pod \"multus-additional-cni-plugins-dc7p2\" (UID: \"c088609c-51c3-4070-88eb-a329ef4204a7\") " pod="openshift-multus/multus-additional-cni-plugins-dc7p2"
Sep 11 18:48:46.380025 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.378644    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/90acf5d2-d7df-46bb-a380-73f3ae079b75-dbus\") pod \"global-pull-secret-syncer-vnkmt\" (UID: \"90acf5d2-d7df-46bb-a380-73f3ae079b75\") " pod="kube-system/global-pull-secret-syncer-vnkmt"
Sep 11 18:48:46.380025 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.378753    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/240ee9ec-9e58-435f-938f-62cf71acfb5b-run\") pod \"tuned-fqd6d\" (UID: \"240ee9ec-9e58-435f-938f-62cf71acfb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-fqd6d"
Sep 11 18:48:46.380025 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.378773    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/240ee9ec-9e58-435f-938f-62cf71acfb5b-etc-sysconfig\") pod \"tuned-fqd6d\" (UID: \"240ee9ec-9e58-435f-938f-62cf71acfb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-fqd6d"
Sep 11 18:48:46.380025 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:48:46.378843    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 18:48:46.380025 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.378846    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/679f1bfe-400c-46c5-837e-39d1dccccf5f-konnectivity-ca\") pod \"konnectivity-agent-8lq2v\" (UID: \"679f1bfe-400c-46c5-837e-39d1dccccf5f\") " pod="kube-system/konnectivity-agent-8lq2v"
Sep 11 18:48:46.380025 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:48:46.379002    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8498e9d1-1eeb-4ce9-b225-d18452e49150-metrics-certs podName:8498e9d1-1eeb-4ce9-b225-d18452e49150 nodeName:}" failed. No retries permitted until 2026-09-11 18:48:46.87897971 +0000 UTC m=+2.956570268 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8498e9d1-1eeb-4ce9-b225-d18452e49150-metrics-certs") pod "network-metrics-daemon-bjcnp" (UID: "8498e9d1-1eeb-4ce9-b225-d18452e49150") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 18:48:46.380025 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.379137    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43683762-3da6-4f69-a5ad-d685d63bbe4f-tmp-dir\") pod \"node-resolver-dnzn9\" (UID: \"43683762-3da6-4f69-a5ad-d685d63bbe4f\") " pod="openshift-dns/node-resolver-dnzn9"
Sep 11 18:48:46.380025 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.379279    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/240ee9ec-9e58-435f-938f-62cf71acfb5b-etc-sysctl-d\") pod \"tuned-fqd6d\" (UID: \"240ee9ec-9e58-435f-938f-62cf71acfb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-fqd6d"
Sep 11 18:48:46.380025 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.379393    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0905525e-5ad4-4f0a-882d-d9d55c922e0b-iptables-alerter-script\") pod \"iptables-alerter-hh9vt\" (UID: \"0905525e-5ad4-4f0a-882d-d9d55c922e0b\") " pod="openshift-network-operator/iptables-alerter-hh9vt"
Sep 11 18:48:46.382500 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.382334    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/679f1bfe-400c-46c5-837e-39d1dccccf5f-agent-certs\") pod \"konnectivity-agent-8lq2v\" (UID: \"679f1bfe-400c-46c5-837e-39d1dccccf5f\") " pod="kube-system/konnectivity-agent-8lq2v"
Sep 11 18:48:46.382636 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.382382    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/240ee9ec-9e58-435f-938f-62cf71acfb5b-etc-tuned\") pod \"tuned-fqd6d\" (UID: \"240ee9ec-9e58-435f-938f-62cf71acfb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-fqd6d"
Sep 11 18:48:46.382636 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.382422    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/240ee9ec-9e58-435f-938f-62cf71acfb5b-tmp\") pod \"tuned-fqd6d\" (UID: \"240ee9ec-9e58-435f-938f-62cf71acfb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-fqd6d"
Sep 11 18:48:46.390657 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:48:46.390325    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 18:48:46.390657 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:48:46.390348    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 18:48:46.390657 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:48:46.390362    2578 projected.go:196] Error preparing data for projected volume kube-api-access-f8h4f for pod openshift-network-diagnostics/network-check-target-7c8ns: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 18:48:46.390657 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:48:46.390421    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c7382740-2bba-4368-8e7f-508c64d170a7-kube-api-access-f8h4f podName:c7382740-2bba-4368-8e7f-508c64d170a7 nodeName:}" failed. No retries permitted until 2026-09-11 18:48:46.890401097 +0000 UTC m=+2.967991650 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-f8h4f" (UniqueName: "kubernetes.io/projected/c7382740-2bba-4368-8e7f-508c64d170a7-kube-api-access-f8h4f") pod "network-check-target-7c8ns" (UID: "c7382740-2bba-4368-8e7f-508c64d170a7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 18:48:46.392414 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.392363    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-78kr7\" (UniqueName: \"kubernetes.io/projected/0905525e-5ad4-4f0a-882d-d9d55c922e0b-kube-api-access-78kr7\") pod \"iptables-alerter-hh9vt\" (UID: \"0905525e-5ad4-4f0a-882d-d9d55c922e0b\") " pod="openshift-network-operator/iptables-alerter-hh9vt"
Sep 11 18:48:46.392934 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.392888    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2kst\" (UniqueName: \"kubernetes.io/projected/4c6ece11-f761-4493-ba21-fa07150f8ec3-kube-api-access-b2kst\") pod \"aws-ebs-csi-driver-node-plbjn\" (UID: \"4c6ece11-f761-4493-ba21-fa07150f8ec3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plbjn"
Sep 11 18:48:46.393606 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.393576    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7nng\" (UniqueName: \"kubernetes.io/projected/8498e9d1-1eeb-4ce9-b225-d18452e49150-kube-api-access-r7nng\") pod \"network-metrics-daemon-bjcnp\" (UID: \"8498e9d1-1eeb-4ce9-b225-d18452e49150\") " pod="openshift-multus/network-metrics-daemon-bjcnp"
Sep 11 18:48:46.394844 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.394821    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl8kx\" (UniqueName: \"kubernetes.io/projected/107db790-53d5-4969-8bf9-2ee3b9f9df09-kube-api-access-tl8kx\") pod \"node-ca-wmk8r\" (UID: \"107db790-53d5-4969-8bf9-2ee3b9f9df09\") " pod="openshift-image-registry/node-ca-wmk8r"
Sep 11 18:48:46.394940 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.394885    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkk6g\" (UniqueName: \"kubernetes.io/projected/c088609c-51c3-4070-88eb-a329ef4204a7-kube-api-access-rkk6g\") pod \"multus-additional-cni-plugins-dc7p2\" (UID: \"c088609c-51c3-4070-88eb-a329ef4204a7\") " pod="openshift-multus/multus-additional-cni-plugins-dc7p2"
Sep 11 18:48:46.394940 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.394893    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hrvd\" (UniqueName: \"kubernetes.io/projected/240ee9ec-9e58-435f-938f-62cf71acfb5b-kube-api-access-9hrvd\") pod \"tuned-fqd6d\" (UID: \"240ee9ec-9e58-435f-938f-62cf71acfb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-fqd6d"
Sep 11 18:48:46.398251 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.398229    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nphws\" (UniqueName: \"kubernetes.io/projected/43683762-3da6-4f69-a5ad-d685d63bbe4f-kube-api-access-nphws\") pod \"node-resolver-dnzn9\" (UID: \"43683762-3da6-4f69-a5ad-d685d63bbe4f\") " pod="openshift-dns/node-resolver-dnzn9"
Sep 11 18:48:46.479994 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.479956    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/23c2932c-66b4-4b05-825e-afbbc27f110e-node-log\") pod \"ovnkube-node-7xkpg\" (UID: \"23c2932c-66b4-4b05-825e-afbbc27f110e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xkpg"
Sep 11 18:48:46.480208 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.480030    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/16f7a243-b894-446f-80f7-fe1ae5a8d66c-host-var-lib-cni-multus\") pod \"multus-dcbq8\" (UID: \"16f7a243-b894-446f-80f7-fe1ae5a8d66c\") " pod="openshift-multus/multus-dcbq8"
Sep 11 18:48:46.480208 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.480036    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/23c2932c-66b4-4b05-825e-afbbc27f110e-node-log\") pod \"ovnkube-node-7xkpg\" (UID: \"23c2932c-66b4-4b05-825e-afbbc27f110e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xkpg"
Sep 11 18:48:46.480208 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.480063    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/23c2932c-66b4-4b05-825e-afbbc27f110e-etc-openvswitch\") pod \"ovnkube-node-7xkpg\" (UID: \"23c2932c-66b4-4b05-825e-afbbc27f110e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xkpg"
Sep 11 18:48:46.480208 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.480086    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/23c2932c-66b4-4b05-825e-afbbc27f110e-ovnkube-config\") pod \"ovnkube-node-7xkpg\" (UID: \"23c2932c-66b4-4b05-825e-afbbc27f110e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xkpg"
Sep 11 18:48:46.480208 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.480091    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/16f7a243-b894-446f-80f7-fe1ae5a8d66c-host-var-lib-cni-multus\") pod \"multus-dcbq8\" (UID: \"16f7a243-b894-446f-80f7-fe1ae5a8d66c\") " pod="openshift-multus/multus-dcbq8"
Sep 11 18:48:46.480208 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.480104    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/23c2932c-66b4-4b05-825e-afbbc27f110e-log-socket\") pod \"ovnkube-node-7xkpg\" (UID: \"23c2932c-66b4-4b05-825e-afbbc27f110e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xkpg"
Sep 11 18:48:46.480208 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.480127    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/23c2932c-66b4-4b05-825e-afbbc27f110e-ovn-node-metrics-cert\") pod \"ovnkube-node-7xkpg\" (UID: \"23c2932c-66b4-4b05-825e-afbbc27f110e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xkpg"
Sep 11 18:48:46.480208 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.480142    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/16f7a243-b894-446f-80f7-fe1ae5a8d66c-host-run-k8s-cni-cncf-io\") pod \"multus-dcbq8\" (UID: \"16f7a243-b894-446f-80f7-fe1ae5a8d66c\") " pod="openshift-multus/multus-dcbq8"
Sep 11 18:48:46.480208 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.480145    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/23c2932c-66b4-4b05-825e-afbbc27f110e-etc-openvswitch\") pod \"ovnkube-node-7xkpg\" (UID: \"23c2932c-66b4-4b05-825e-afbbc27f110e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xkpg"
Sep 11 18:48:46.480208 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.480169    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/23c2932c-66b4-4b05-825e-afbbc27f110e-host-slash\") pod \"ovnkube-node-7xkpg\" (UID: \"23c2932c-66b4-4b05-825e-afbbc27f110e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xkpg"
Sep 11 18:48:46.480208 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.480193    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/16f7a243-b894-446f-80f7-fe1ae5a8d66c-multus-daemon-config\") pod \"multus-dcbq8\" (UID: \"16f7a243-b894-446f-80f7-fe1ae5a8d66c\") " pod="openshift-multus/multus-dcbq8"
Sep 11 18:48:46.480806 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.480223    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n94jg\" (UniqueName: \"kubernetes.io/projected/23c2932c-66b4-4b05-825e-afbbc27f110e-kube-api-access-n94jg\") pod \"ovnkube-node-7xkpg\" (UID: \"23c2932c-66b4-4b05-825e-afbbc27f110e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xkpg"
Sep 11 18:48:46.480806 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.480242    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/16f7a243-b894-446f-80f7-fe1ae5a8d66c-host-run-k8s-cni-cncf-io\") pod \"multus-dcbq8\" (UID: \"16f7a243-b894-446f-80f7-fe1ae5a8d66c\") " pod="openshift-multus/multus-dcbq8"
Sep 11 18:48:46.480806 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.480260    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/23c2932c-66b4-4b05-825e-afbbc27f110e-host-slash\") pod \"ovnkube-node-7xkpg\" (UID: \"23c2932c-66b4-4b05-825e-afbbc27f110e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xkpg"
Sep 11 18:48:46.480806 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.480249    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/90acf5d2-d7df-46bb-a380-73f3ae079b75-original-pull-secret\") pod \"global-pull-secret-syncer-vnkmt\" (UID: \"90acf5d2-d7df-46bb-a380-73f3ae079b75\") " pod="kube-system/global-pull-secret-syncer-vnkmt"
Sep 11 18:48:46.480806 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:48:46.480329    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 18:48:46.480806 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.480341    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/23c2932c-66b4-4b05-825e-afbbc27f110e-host-run-netns\") pod \"ovnkube-node-7xkpg\" (UID: \"23c2932c-66b4-4b05-825e-afbbc27f110e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xkpg"
Sep 11 18:48:46.480806 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.480373    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/16f7a243-b894-446f-80f7-fe1ae5a8d66c-host-run-netns\") pod \"multus-dcbq8\" (UID: \"16f7a243-b894-446f-80f7-fe1ae5a8d66c\") " pod="openshift-multus/multus-dcbq8"
Sep 11 18:48:46.480806 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:48:46.480385    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90acf5d2-d7df-46bb-a380-73f3ae079b75-original-pull-secret podName:90acf5d2-d7df-46bb-a380-73f3ae079b75 nodeName:}" failed. No retries permitted until 2026-09-11 18:48:46.980367705 +0000 UTC m=+3.057958239 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/90acf5d2-d7df-46bb-a380-73f3ae079b75-original-pull-secret") pod "global-pull-secret-syncer-vnkmt" (UID: "90acf5d2-d7df-46bb-a380-73f3ae079b75") : object "kube-system"/"original-pull-secret" not registered
Sep 11 18:48:46.480806 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.480405    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/16f7a243-b894-446f-80f7-fe1ae5a8d66c-host-run-netns\") pod \"multus-dcbq8\" (UID: \"16f7a243-b894-446f-80f7-fe1ae5a8d66c\") " pod="openshift-multus/multus-dcbq8"
Sep 11 18:48:46.480806 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.480413    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/23c2932c-66b4-4b05-825e-afbbc27f110e-env-overrides\") pod \"ovnkube-node-7xkpg\" (UID: \"23c2932c-66b4-4b05-825e-afbbc27f110e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xkpg"
Sep 11 18:48:46.480806 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.480442    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/23c2932c-66b4-4b05-825e-afbbc27f110e-host-run-netns\") pod \"ovnkube-node-7xkpg\" (UID: \"23c2932c-66b4-4b05-825e-afbbc27f110e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xkpg"
Sep 11 18:48:46.480806 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.480440    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/23c2932c-66b4-4b05-825e-afbbc27f110e-ovnkube-script-lib\") pod \"ovnkube-node-7xkpg\" (UID: \"23c2932c-66b4-4b05-825e-afbbc27f110e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xkpg"
Sep 11 18:48:46.480806 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.480479    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/90acf5d2-d7df-46bb-a380-73f3ae079b75-dbus\") pod \"global-pull-secret-syncer-vnkmt\" (UID: \"90acf5d2-d7df-46bb-a380-73f3ae079b75\") " pod="kube-system/global-pull-secret-syncer-vnkmt"
Sep 11 18:48:46.480806 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.480523    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/23c2932c-66b4-4b05-825e-afbbc27f110e-log-socket\") pod \"ovnkube-node-7xkpg\" (UID: \"23c2932c-66b4-4b05-825e-afbbc27f110e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xkpg"
Sep 11 18:48:46.480806 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.480617    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/90acf5d2-d7df-46bb-a380-73f3ae079b75-dbus\") pod \"global-pull-secret-syncer-vnkmt\" (UID: \"90acf5d2-d7df-46bb-a380-73f3ae079b75\") " pod="kube-system/global-pull-secret-syncer-vnkmt"
Sep 11 18:48:46.480806 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.480679    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/16f7a243-b894-446f-80f7-fe1ae5a8d66c-cnibin\") pod \"multus-dcbq8\" (UID: \"16f7a243-b894-446f-80f7-fe1ae5a8d66c\") " pod="openshift-multus/multus-dcbq8"
Sep 11 18:48:46.480806 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.480706    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/16f7a243-b894-446f-80f7-fe1ae5a8d66c-multus-socket-dir-parent\") pod \"multus-dcbq8\" (UID: \"16f7a243-b894-446f-80f7-fe1ae5a8d66c\") " pod="openshift-multus/multus-dcbq8"
Sep 11 18:48:46.481530 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.480730    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/16f7a243-b894-446f-80f7-fe1ae5a8d66c-host-var-lib-cni-bin\") pod \"multus-dcbq8\" (UID: \"16f7a243-b894-446f-80f7-fe1ae5a8d66c\") " pod="openshift-multus/multus-dcbq8"
Sep 11 18:48:46.481530 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.480754    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/16f7a243-b894-446f-80f7-fe1ae5a8d66c-hostroot\") pod \"multus-dcbq8\" (UID: \"16f7a243-b894-446f-80f7-fe1ae5a8d66c\") " pod="openshift-multus/multus-dcbq8"
Sep 11 18:48:46.481530 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.480780    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/16f7a243-b894-446f-80f7-fe1ae5a8d66c-cnibin\") pod \"multus-dcbq8\" (UID: \"16f7a243-b894-446f-80f7-fe1ae5a8d66c\") " pod="openshift-multus/multus-dcbq8"
Sep 11 18:48:46.481530 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.480809    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/23c2932c-66b4-4b05-825e-afbbc27f110e-ovnkube-config\") pod \"ovnkube-node-7xkpg\" (UID: \"23c2932c-66b4-4b05-825e-afbbc27f110e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xkpg"
Sep 11 18:48:46.481530 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.480830    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/16f7a243-b894-446f-80f7-fe1ae5a8d66c-multus-socket-dir-parent\") pod \"multus-dcbq8\" (UID: \"16f7a243-b894-446f-80f7-fe1ae5a8d66c\") " pod="openshift-multus/multus-dcbq8"
Sep 11 18:48:46.481530 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.480834    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/16f7a243-b894-446f-80f7-fe1ae5a8d66c-system-cni-dir\") pod \"multus-dcbq8\" (UID: \"16f7a243-b894-446f-80f7-fe1ae5a8d66c\") " pod="openshift-multus/multus-dcbq8"
Sep 11 18:48:46.481530 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.480783    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/16f7a243-b894-446f-80f7-fe1ae5a8d66c-system-cni-dir\") pod \"multus-dcbq8\" (UID: \"16f7a243-b894-446f-80f7-fe1ae5a8d66c\") " pod="openshift-multus/multus-dcbq8"
Sep 11 18:48:46.481530 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.480866    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/16f7a243-b894-446f-80f7-fe1ae5a8d66c-host-var-lib-cni-bin\") pod \"multus-dcbq8\" (UID: \"16f7a243-b894-446f-80f7-fe1ae5a8d66c\") " pod="openshift-multus/multus-dcbq8"
Sep 11 18:48:46.481530 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.480874    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/23c2932c-66b4-4b05-825e-afbbc27f110e-host-cni-netd\") pod \"ovnkube-node-7xkpg\" (UID: \"23c2932c-66b4-4b05-825e-afbbc27f110e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xkpg"
Sep 11 18:48:46.481530 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.480879    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/16f7a243-b894-446f-80f7-fe1ae5a8d66c-hostroot\") pod \"multus-dcbq8\" (UID: \"16f7a243-b894-446f-80f7-fe1ae5a8d66c\") " pod="openshift-multus/multus-dcbq8"
Sep 11 18:48:46.481530 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.480903    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/23c2932c-66b4-4b05-825e-afbbc27f110e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7xkpg\" (UID: \"23c2932c-66b4-4b05-825e-afbbc27f110e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xkpg"
Sep 11 18:48:46.481530 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.480917    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/16f7a243-b894-446f-80f7-fe1ae5a8d66c-multus-daemon-config\") pod \"multus-dcbq8\" (UID: \"16f7a243-b894-446f-80f7-fe1ae5a8d66c\") " pod="openshift-multus/multus-dcbq8"
Sep 11 18:48:46.481530 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.480926    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/16f7a243-b894-446f-80f7-fe1ae5a8d66c-host-run-multus-certs\") pod \"multus-dcbq8\" (UID: \"16f7a243-b894-446f-80f7-fe1ae5a8d66c\") " pod="openshift-multus/multus-dcbq8"
Sep 11 18:48:46.481530 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.480950    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/16f7a243-b894-446f-80f7-fe1ae5a8d66c-host-run-multus-certs\") pod \"multus-dcbq8\" (UID: \"16f7a243-b894-446f-80f7-fe1ae5a8d66c\") " pod="openshift-multus/multus-dcbq8"
Sep 11 18:48:46.481530 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.480958    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/23c2932c-66b4-4b05-825e-afbbc27f110e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7xkpg\" (UID: \"23c2932c-66b4-4b05-825e-afbbc27f110e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xkpg"
Sep 11 18:48:46.481530 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.480927    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/23c2932c-66b4-4b05-825e-afbbc27f110e-host-cni-netd\") pod \"ovnkube-node-7xkpg\" (UID: \"23c2932c-66b4-4b05-825e-afbbc27f110e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xkpg"
Sep 11 18:48:46.481530 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.480981    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckd8m\" (UniqueName: \"kubernetes.io/projected/16f7a243-b894-446f-80f7-fe1ae5a8d66c-kube-api-access-ckd8m\") pod \"multus-dcbq8\" (UID: \"16f7a243-b894-446f-80f7-fe1ae5a8d66c\") " pod="openshift-multus/multus-dcbq8"
Sep 11 18:48:46.481530 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.481013    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/90acf5d2-d7df-46bb-a380-73f3ae079b75-kubelet-config\") pod \"global-pull-secret-syncer-vnkmt\" (UID: \"90acf5d2-d7df-46bb-a380-73f3ae079b75\") " pod="kube-system/global-pull-secret-syncer-vnkmt"
Sep 11 18:48:46.482243 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.480980    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/23c2932c-66b4-4b05-825e-afbbc27f110e-env-overrides\") pod \"ovnkube-node-7xkpg\" (UID: \"23c2932c-66b4-4b05-825e-afbbc27f110e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xkpg"
Sep 11 18:48:46.482243 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.481037    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/16f7a243-b894-446f-80f7-fe1ae5a8d66c-multus-conf-dir\") pod \"multus-dcbq8\" (UID: \"16f7a243-b894-446f-80f7-fe1ae5a8d66c\") " pod="openshift-multus/multus-dcbq8"
Sep 11 18:48:46.482243 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.481035    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/23c2932c-66b4-4b05-825e-afbbc27f110e-ovnkube-script-lib\") pod \"ovnkube-node-7xkpg\" (UID: \"23c2932c-66b4-4b05-825e-afbbc27f110e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xkpg"
Sep 11 18:48:46.482243 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.481067    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/23c2932c-66b4-4b05-825e-afbbc27f110e-host-kubelet\") pod \"ovnkube-node-7xkpg\" (UID: \"23c2932c-66b4-4b05-825e-afbbc27f110e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xkpg"
Sep 11 18:48:46.482243 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.481076    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/90acf5d2-d7df-46bb-a380-73f3ae079b75-kubelet-config\") pod \"global-pull-secret-syncer-vnkmt\" (UID: \"90acf5d2-d7df-46bb-a380-73f3ae079b75\") " pod="kube-system/global-pull-secret-syncer-vnkmt"
Sep 11 18:48:46.482243 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.481091    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/23c2932c-66b4-4b05-825e-afbbc27f110e-var-lib-openvswitch\") pod \"ovnkube-node-7xkpg\" (UID: \"23c2932c-66b4-4b05-825e-afbbc27f110e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xkpg"
Sep 11 18:48:46.482243 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.481117    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/16f7a243-b894-446f-80f7-fe1ae5a8d66c-cni-binary-copy\") pod \"multus-dcbq8\" (UID: \"16f7a243-b894-446f-80f7-fe1ae5a8d66c\") " pod="openshift-multus/multus-dcbq8"
Sep 11 18:48:46.482243 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.481119    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/23c2932c-66b4-4b05-825e-afbbc27f110e-host-kubelet\") pod \"ovnkube-node-7xkpg\" (UID: \"23c2932c-66b4-4b05-825e-afbbc27f110e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xkpg"
Sep 11 18:48:46.482243 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.481126    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/23c2932c-66b4-4b05-825e-afbbc27f110e-var-lib-openvswitch\") pod \"ovnkube-node-7xkpg\" (UID: \"23c2932c-66b4-4b05-825e-afbbc27f110e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xkpg"
Sep 11 18:48:46.482243 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.481094    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/16f7a243-b894-446f-80f7-fe1ae5a8d66c-multus-conf-dir\") pod \"multus-dcbq8\" (UID: \"16f7a243-b894-446f-80f7-fe1ae5a8d66c\") " pod="openshift-multus/multus-dcbq8"
Sep 11 18:48:46.482243 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.481144    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/23c2932c-66b4-4b05-825e-afbbc27f110e-systemd-units\") pod \"ovnkube-node-7xkpg\" (UID: \"23c2932c-66b4-4b05-825e-afbbc27f110e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xkpg"
Sep 11 18:48:46.482243 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.481169    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/23c2932c-66b4-4b05-825e-afbbc27f110e-run-systemd\") pod \"ovnkube-node-7xkpg\" (UID: \"23c2932c-66b4-4b05-825e-afbbc27f110e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xkpg"
Sep 11 18:48:46.482243 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.481194    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/23c2932c-66b4-4b05-825e-afbbc27f110e-host-run-ovn-kubernetes\") pod \"ovnkube-node-7xkpg\" (UID: \"23c2932c-66b4-4b05-825e-afbbc27f110e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xkpg"
Sep 11 18:48:46.482243 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.481217    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/23c2932c-66b4-4b05-825e-afbbc27f110e-systemd-units\") pod \"ovnkube-node-7xkpg\" (UID: \"23c2932c-66b4-4b05-825e-afbbc27f110e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xkpg"
Sep 11 18:48:46.482243 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.481220    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/23c2932c-66b4-4b05-825e-afbbc27f110e-host-cni-bin\") pod \"ovnkube-node-7xkpg\" (UID: \"23c2932c-66b4-4b05-825e-afbbc27f110e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xkpg"
Sep 11 18:48:46.482243 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.481228    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/23c2932c-66b4-4b05-825e-afbbc27f110e-run-systemd\") pod \"ovnkube-node-7xkpg\" (UID: \"23c2932c-66b4-4b05-825e-afbbc27f110e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xkpg"
Sep 11 18:48:46.482243 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.481253    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/23c2932c-66b4-4b05-825e-afbbc27f110e-host-cni-bin\") pod \"ovnkube-node-7xkpg\" (UID: \"23c2932c-66b4-4b05-825e-afbbc27f110e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xkpg"
Sep 11 18:48:46.482243 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.481256    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/23c2932c-66b4-4b05-825e-afbbc27f110e-host-run-ovn-kubernetes\") pod \"ovnkube-node-7xkpg\" (UID: \"23c2932c-66b4-4b05-825e-afbbc27f110e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xkpg"
Sep 11 18:48:46.482951 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.481252    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/16f7a243-b894-446f-80f7-fe1ae5a8d66c-etc-kubernetes\") pod \"multus-dcbq8\" (UID: \"16f7a243-b894-446f-80f7-fe1ae5a8d66c\") " pod="openshift-multus/multus-dcbq8"
Sep 11 18:48:46.482951 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.481286    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/16f7a243-b894-446f-80f7-fe1ae5a8d66c-etc-kubernetes\") pod \"multus-dcbq8\" (UID: \"16f7a243-b894-446f-80f7-fe1ae5a8d66c\") " pod="openshift-multus/multus-dcbq8"
Sep 11 18:48:46.482951 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.481293    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/23c2932c-66b4-4b05-825e-afbbc27f110e-run-ovn\") pod \"ovnkube-node-7xkpg\" (UID: \"23c2932c-66b4-4b05-825e-afbbc27f110e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xkpg"
Sep 11 18:48:46.482951 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.481321    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/23c2932c-66b4-4b05-825e-afbbc27f110e-run-openvswitch\") pod \"ovnkube-node-7xkpg\" (UID: \"23c2932c-66b4-4b05-825e-afbbc27f110e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xkpg"
Sep 11 18:48:46.482951 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.481338    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/16f7a243-b894-446f-80f7-fe1ae5a8d66c-multus-cni-dir\") pod \"multus-dcbq8\" (UID: \"16f7a243-b894-446f-80f7-fe1ae5a8d66c\") " pod="openshift-multus/multus-dcbq8"
Sep 11 18:48:46.482951 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.481328    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/23c2932c-66b4-4b05-825e-afbbc27f110e-run-ovn\") pod \"ovnkube-node-7xkpg\" (UID: \"23c2932c-66b4-4b05-825e-afbbc27f110e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xkpg"
Sep 11 18:48:46.482951 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.481358    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/16f7a243-b894-446f-80f7-fe1ae5a8d66c-host-var-lib-kubelet\") pod \"multus-dcbq8\" (UID: \"16f7a243-b894-446f-80f7-fe1ae5a8d66c\") " pod="openshift-multus/multus-dcbq8"
Sep 11 18:48:46.482951 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.481394    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/23c2932c-66b4-4b05-825e-afbbc27f110e-run-openvswitch\") pod \"ovnkube-node-7xkpg\" (UID: \"23c2932c-66b4-4b05-825e-afbbc27f110e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xkpg"
Sep 11 18:48:46.482951 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.481420    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/16f7a243-b894-446f-80f7-fe1ae5a8d66c-host-var-lib-kubelet\") pod \"multus-dcbq8\" (UID: \"16f7a243-b894-446f-80f7-fe1ae5a8d66c\") " pod="openshift-multus/multus-dcbq8"
Sep 11 18:48:46.482951 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.481461    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/16f7a243-b894-446f-80f7-fe1ae5a8d66c-multus-cni-dir\") pod \"multus-dcbq8\" (UID: \"16f7a243-b894-446f-80f7-fe1ae5a8d66c\") " pod="openshift-multus/multus-dcbq8"
Sep 11 18:48:46.482951 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.481568    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/16f7a243-b894-446f-80f7-fe1ae5a8d66c-cni-binary-copy\") pod \"multus-dcbq8\" (UID: \"16f7a243-b894-446f-80f7-fe1ae5a8d66c\") " pod="openshift-multus/multus-dcbq8"
Sep 11 18:48:46.483882 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.483858    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/23c2932c-66b4-4b05-825e-afbbc27f110e-ovn-node-metrics-cert\") pod \"ovnkube-node-7xkpg\" (UID: \"23c2932c-66b4-4b05-825e-afbbc27f110e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xkpg"
Sep 11 18:48:46.490006 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.489981    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckd8m\" (UniqueName: \"kubernetes.io/projected/16f7a243-b894-446f-80f7-fe1ae5a8d66c-kube-api-access-ckd8m\") pod \"multus-dcbq8\" (UID: \"16f7a243-b894-446f-80f7-fe1ae5a8d66c\") " pod="openshift-multus/multus-dcbq8"
Sep 11 18:48:46.490214 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.490196    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n94jg\" (UniqueName: \"kubernetes.io/projected/23c2932c-66b4-4b05-825e-afbbc27f110e-kube-api-access-n94jg\") pod \"ovnkube-node-7xkpg\" (UID: \"23c2932c-66b4-4b05-825e-afbbc27f110e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xkpg"
Sep 11 18:48:46.569205 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.569124    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hh9vt"
Sep 11 18:48:46.576628 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.576583    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-wmk8r"
Sep 11 18:48:46.587721 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.587693    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dc7p2"
Sep 11 18:48:46.598795 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.598772    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fqd6d"
Sep 11 18:48:46.606541 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.606518    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dnzn9"
Sep 11 18:48:46.614500 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.614475    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8lq2v"
Sep 11 18:48:46.623724 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.623509    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plbjn"
Sep 11 18:48:46.631400 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.631375    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7xkpg"
Sep 11 18:48:46.638278 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.638258    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dcbq8"
Sep 11 18:48:46.784737 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.784709    2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 18:48:46.884267 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.884175    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8498e9d1-1eeb-4ce9-b225-d18452e49150-metrics-certs\") pod \"network-metrics-daemon-bjcnp\" (UID: \"8498e9d1-1eeb-4ce9-b225-d18452e49150\") " pod="openshift-multus/network-metrics-daemon-bjcnp"
Sep 11 18:48:46.884449 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:48:46.884299    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 18:48:46.884449 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:48:46.884355    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8498e9d1-1eeb-4ce9-b225-d18452e49150-metrics-certs podName:8498e9d1-1eeb-4ce9-b225-d18452e49150 nodeName:}" failed. No retries permitted until 2026-09-11 18:48:47.884340356 +0000 UTC m=+3.961930891 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8498e9d1-1eeb-4ce9-b225-d18452e49150-metrics-certs") pod "network-metrics-daemon-bjcnp" (UID: "8498e9d1-1eeb-4ce9-b225-d18452e49150") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 18:48:46.985497 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.985444    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/90acf5d2-d7df-46bb-a380-73f3ae079b75-original-pull-secret\") pod \"global-pull-secret-syncer-vnkmt\" (UID: \"90acf5d2-d7df-46bb-a380-73f3ae079b75\") " pod="kube-system/global-pull-secret-syncer-vnkmt"
Sep 11 18:48:46.985702 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:46.985540    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8h4f\" (UniqueName: \"kubernetes.io/projected/c7382740-2bba-4368-8e7f-508c64d170a7-kube-api-access-f8h4f\") pod \"network-check-target-7c8ns\" (UID: \"c7382740-2bba-4368-8e7f-508c64d170a7\") " pod="openshift-network-diagnostics/network-check-target-7c8ns"
Sep 11 18:48:46.985702 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:48:46.985624    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 18:48:46.985702 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:48:46.985652    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 18:48:46.985702 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:48:46.985668    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 18:48:46.985702 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:48:46.985678    2578 projected.go:196] Error preparing data for projected volume kube-api-access-f8h4f for pod openshift-network-diagnostics/network-check-target-7c8ns: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 18:48:46.985977 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:48:46.985713    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90acf5d2-d7df-46bb-a380-73f3ae079b75-original-pull-secret podName:90acf5d2-d7df-46bb-a380-73f3ae079b75 nodeName:}" failed. No retries permitted until 2026-09-11 18:48:47.985693519 +0000 UTC m=+4.063284063 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/90acf5d2-d7df-46bb-a380-73f3ae079b75-original-pull-secret") pod "global-pull-secret-syncer-vnkmt" (UID: "90acf5d2-d7df-46bb-a380-73f3ae079b75") : object "kube-system"/"original-pull-secret" not registered
Sep 11 18:48:46.985977 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:48:46.985730    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c7382740-2bba-4368-8e7f-508c64d170a7-kube-api-access-f8h4f podName:c7382740-2bba-4368-8e7f-508c64d170a7 nodeName:}" failed. No retries permitted until 2026-09-11 18:48:47.985723011 +0000 UTC m=+4.063313546 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-f8h4f" (UniqueName: "kubernetes.io/projected/c7382740-2bba-4368-8e7f-508c64d170a7-kube-api-access-f8h4f") pod "network-check-target-7c8ns" (UID: "c7382740-2bba-4368-8e7f-508c64d170a7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 18:48:47.317557 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:47.317482    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 18:43:46 +0000 UTC" deadline="2028-03-07 17:43:25.150667652 +0000 UTC"
Sep 11 18:48:47.317557 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:47.317527    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13030h54m37.833146719s"
Sep 11 18:48:47.323459 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:47.323439    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod240ee9ec_9e58_435f_938f_62cf71acfb5b.slice/crio-5d468fdf96ed4b663a4589d227da43ee8dd9f0be6cbb07fdf2471447eb4500d4 WatchSource:0}: Error finding container 5d468fdf96ed4b663a4589d227da43ee8dd9f0be6cbb07fdf2471447eb4500d4: Status 404 returned error can't find the container with id 5d468fdf96ed4b663a4589d227da43ee8dd9f0be6cbb07fdf2471447eb4500d4
Sep 11 18:48:47.325025 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:47.324956    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0905525e_5ad4_4f0a_882d_d9d55c922e0b.slice/crio-9795f6bc6e47a75c6098441a06eb73b3a5c54275af4e3c7752791ebc41788de6 WatchSource:0}: Error finding container 9795f6bc6e47a75c6098441a06eb73b3a5c54275af4e3c7752791ebc41788de6: Status 404 returned error can't find the container with id 9795f6bc6e47a75c6098441a06eb73b3a5c54275af4e3c7752791ebc41788de6
Sep 11 18:48:47.327109 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:47.327082    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16f7a243_b894_446f_80f7_fe1ae5a8d66c.slice/crio-66ede7c3832f1280f3bbd656e7278a92c6a790fd1bd963b98d7d248485914f50 WatchSource:0}: Error finding container 66ede7c3832f1280f3bbd656e7278a92c6a790fd1bd963b98d7d248485914f50: Status 404 returned error can't find the container with id 66ede7c3832f1280f3bbd656e7278a92c6a790fd1bd963b98d7d248485914f50
Sep 11 18:48:47.329141 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:47.329119    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c6ece11_f761_4493_ba21_fa07150f8ec3.slice/crio-6eb8503b8f111a15d75b1ad63f35618309b3ad205dc49685434a54737b6dcf23 WatchSource:0}: Error finding container 6eb8503b8f111a15d75b1ad63f35618309b3ad205dc49685434a54737b6dcf23: Status 404 returned error can't find the container with id 6eb8503b8f111a15d75b1ad63f35618309b3ad205dc49685434a54737b6dcf23
Sep 11 18:48:47.330339 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:47.330310    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc088609c_51c3_4070_88eb_a329ef4204a7.slice/crio-2b4a830c1ceed7cb308f1b1b620853267ea74377f559771011f6d215dc76d603 WatchSource:0}: Error finding container 2b4a830c1ceed7cb308f1b1b620853267ea74377f559771011f6d215dc76d603: Status 404 returned error can't find the container with id 2b4a830c1ceed7cb308f1b1b620853267ea74377f559771011f6d215dc76d603
Sep 11 18:48:47.331371 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:47.331349    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23c2932c_66b4_4b05_825e_afbbc27f110e.slice/crio-64955f05d85371adbe1cb5544615b99ac0eb731c90288c34f35f8ca8dce68643 WatchSource:0}: Error finding container 64955f05d85371adbe1cb5544615b99ac0eb731c90288c34f35f8ca8dce68643: Status 404 returned error can't find the container with id 64955f05d85371adbe1cb5544615b99ac0eb731c90288c34f35f8ca8dce68643
Sep 11 18:48:47.332631 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:47.332508    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43683762_3da6_4f69_a5ad_d685d63bbe4f.slice/crio-2b8445141fe370a081cecb613ddfb6e4ff50a711b947ffa415c38cfa0932a9b6 WatchSource:0}: Error finding container 2b8445141fe370a081cecb613ddfb6e4ff50a711b947ffa415c38cfa0932a9b6: Status 404 returned error can't find the container with id 2b8445141fe370a081cecb613ddfb6e4ff50a711b947ffa415c38cfa0932a9b6
Sep 11 18:48:47.336233 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:48:47.336206    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod107db790_53d5_4969_8bf9_2ee3b9f9df09.slice/crio-54fc59e07a5f9218b73b9c59a98c833c56229e328bc487b713bac804abccaae7 WatchSource:0}: Error finding container 54fc59e07a5f9218b73b9c59a98c833c56229e328bc487b713bac804abccaae7: Status 404 returned error can't find the container with id 54fc59e07a5f9218b73b9c59a98c833c56229e328bc487b713bac804abccaae7
Sep 11 18:48:47.398598 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:47.398561    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dc7p2" event={"ID":"c088609c-51c3-4070-88eb-a329ef4204a7","Type":"ContainerStarted","Data":"2b4a830c1ceed7cb308f1b1b620853267ea74377f559771011f6d215dc76d603"}
Sep 11 18:48:47.399805 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:47.399774    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plbjn" event={"ID":"4c6ece11-f761-4493-ba21-fa07150f8ec3","Type":"ContainerStarted","Data":"6eb8503b8f111a15d75b1ad63f35618309b3ad205dc49685434a54737b6dcf23"}
Sep 11 18:48:47.401124 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:47.401089    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8lq2v" event={"ID":"679f1bfe-400c-46c5-837e-39d1dccccf5f","Type":"ContainerStarted","Data":"41990a92382b5f2743a12c1235f6d49fdb524d9a48c402ecd6538e8ebadf3b10"}
Sep 11 18:48:47.402323 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:47.402301    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-wmk8r" event={"ID":"107db790-53d5-4969-8bf9-2ee3b9f9df09","Type":"ContainerStarted","Data":"54fc59e07a5f9218b73b9c59a98c833c56229e328bc487b713bac804abccaae7"}
Sep 11 18:48:47.403616 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:47.403580    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dnzn9" event={"ID":"43683762-3da6-4f69-a5ad-d685d63bbe4f","Type":"ContainerStarted","Data":"2b8445141fe370a081cecb613ddfb6e4ff50a711b947ffa415c38cfa0932a9b6"}
Sep 11 18:48:47.404906 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:47.404880    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7xkpg" event={"ID":"23c2932c-66b4-4b05-825e-afbbc27f110e","Type":"ContainerStarted","Data":"64955f05d85371adbe1cb5544615b99ac0eb731c90288c34f35f8ca8dce68643"}
Sep 11 18:48:47.406194 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:47.406170    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dcbq8" event={"ID":"16f7a243-b894-446f-80f7-fe1ae5a8d66c","Type":"ContainerStarted","Data":"66ede7c3832f1280f3bbd656e7278a92c6a790fd1bd963b98d7d248485914f50"}
Sep 11 18:48:47.407495 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:47.407472    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hh9vt" event={"ID":"0905525e-5ad4-4f0a-882d-d9d55c922e0b","Type":"ContainerStarted","Data":"9795f6bc6e47a75c6098441a06eb73b3a5c54275af4e3c7752791ebc41788de6"}
Sep 11 18:48:47.408677 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:47.408650    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fqd6d" event={"ID":"240ee9ec-9e58-435f-938f-62cf71acfb5b","Type":"ContainerStarted","Data":"5d468fdf96ed4b663a4589d227da43ee8dd9f0be6cbb07fdf2471447eb4500d4"}
Sep 11 18:48:47.892922 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:47.892740    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8498e9d1-1eeb-4ce9-b225-d18452e49150-metrics-certs\") pod \"network-metrics-daemon-bjcnp\" (UID: \"8498e9d1-1eeb-4ce9-b225-d18452e49150\") " pod="openshift-multus/network-metrics-daemon-bjcnp"
Sep 11 18:48:47.892922 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:48:47.892892    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 18:48:47.893145 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:48:47.892959    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8498e9d1-1eeb-4ce9-b225-d18452e49150-metrics-certs podName:8498e9d1-1eeb-4ce9-b225-d18452e49150 nodeName:}" failed. No retries permitted until 2026-09-11 18:48:49.892938054 +0000 UTC m=+5.970528592 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8498e9d1-1eeb-4ce9-b225-d18452e49150-metrics-certs") pod "network-metrics-daemon-bjcnp" (UID: "8498e9d1-1eeb-4ce9-b225-d18452e49150") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 18:48:47.994414 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:47.993588    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8h4f\" (UniqueName: \"kubernetes.io/projected/c7382740-2bba-4368-8e7f-508c64d170a7-kube-api-access-f8h4f\") pod \"network-check-target-7c8ns\" (UID: \"c7382740-2bba-4368-8e7f-508c64d170a7\") " pod="openshift-network-diagnostics/network-check-target-7c8ns"
Sep 11 18:48:47.994414 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:47.993660    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/90acf5d2-d7df-46bb-a380-73f3ae079b75-original-pull-secret\") pod \"global-pull-secret-syncer-vnkmt\" (UID: \"90acf5d2-d7df-46bb-a380-73f3ae079b75\") " pod="kube-system/global-pull-secret-syncer-vnkmt"
Sep 11 18:48:47.994414 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:48:47.993794    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 18:48:47.994414 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:48:47.993857    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90acf5d2-d7df-46bb-a380-73f3ae079b75-original-pull-secret podName:90acf5d2-d7df-46bb-a380-73f3ae079b75 nodeName:}" failed. No retries permitted until 2026-09-11 18:48:49.993838806 +0000 UTC m=+6.071429356 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/90acf5d2-d7df-46bb-a380-73f3ae079b75-original-pull-secret") pod "global-pull-secret-syncer-vnkmt" (UID: "90acf5d2-d7df-46bb-a380-73f3ae079b75") : object "kube-system"/"original-pull-secret" not registered
Sep 11 18:48:47.994414 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:48:47.994293    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 18:48:47.994414 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:48:47.994312    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 18:48:47.994414 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:48:47.994326    2578 projected.go:196] Error preparing data for projected volume kube-api-access-f8h4f for pod openshift-network-diagnostics/network-check-target-7c8ns: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 18:48:47.994414 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:48:47.994374    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c7382740-2bba-4368-8e7f-508c64d170a7-kube-api-access-f8h4f podName:c7382740-2bba-4368-8e7f-508c64d170a7 nodeName:}" failed. No retries permitted until 2026-09-11 18:48:49.994357655 +0000 UTC m=+6.071948194 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-f8h4f" (UniqueName: "kubernetes.io/projected/c7382740-2bba-4368-8e7f-508c64d170a7-kube-api-access-f8h4f") pod "network-check-target-7c8ns" (UID: "c7382740-2bba-4368-8e7f-508c64d170a7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 18:48:48.317771 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:48.317707    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 18:43:46 +0000 UTC" deadline="2028-05-11 10:20:11.240699601 +0000 UTC"
Sep 11 18:48:48.317771 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:48.317767    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14583h31m22.922936388s"
Sep 11 18:48:48.392098 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:48.391305    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bjcnp"
Sep 11 18:48:48.392098 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:48:48.391459    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bjcnp" podUID="8498e9d1-1eeb-4ce9-b225-d18452e49150"
Sep 11 18:48:48.392098 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:48.391933    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnkmt"
Sep 11 18:48:48.392098 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:48:48.392039    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vnkmt" podUID="90acf5d2-d7df-46bb-a380-73f3ae079b75"
Sep 11 18:48:48.392735 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:48.392542    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7c8ns"
Sep 11 18:48:48.392735 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:48:48.392675    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7c8ns" podUID="c7382740-2bba-4368-8e7f-508c64d170a7"
Sep 11 18:48:48.418700 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:48.418670    2578 generic.go:356] "Generic (PLEG): container finished" podID="4c97f908167f20c09da5c0e0087b3f35" containerID="f35ba2eeb502135d4034f19f4b48fb33f7bad176a94d6229cd46d12f0c8cfc0b" exitCode=0
Sep 11 18:48:48.418825 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:48.418777    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-189.ec2.internal" event={"ID":"4c97f908167f20c09da5c0e0087b3f35","Type":"ContainerDied","Data":"f35ba2eeb502135d4034f19f4b48fb33f7bad176a94d6229cd46d12f0c8cfc0b"}
Sep 11 18:48:48.432368 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:48.432339    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-189.ec2.internal" event={"ID":"4bf36dcd150da66bbf96efc4d354ac7b","Type":"ContainerStarted","Data":"8a3881150a2ae4fc97f440dd910ab515cbdbe049c6b398550ef586809175541e"}
Sep 11 18:48:49.455293 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:49.454779    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-189.ec2.internal" event={"ID":"4c97f908167f20c09da5c0e0087b3f35","Type":"ContainerStarted","Data":"ca898e79bf2a10d66b53ff53573c924676738a4af32d85ca533a7f24d3e5f351"}
Sep 11 18:48:49.472272 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:49.471345    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-189.ec2.internal" podStartSLOduration=4.471323222 podStartE2EDuration="4.471323222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:48:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:48:48.450955713 +0000 UTC m=+4.528546274" watchObservedRunningTime="2026-09-11 18:48:49.471323222 +0000 UTC m=+5.548913786"
Sep 11 18:48:49.907992 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:49.907898    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8498e9d1-1eeb-4ce9-b225-d18452e49150-metrics-certs\") pod \"network-metrics-daemon-bjcnp\" (UID: \"8498e9d1-1eeb-4ce9-b225-d18452e49150\") " pod="openshift-multus/network-metrics-daemon-bjcnp"
Sep 11 18:48:49.908162 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:48:49.908042    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 18:48:49.908162 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:48:49.908102    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8498e9d1-1eeb-4ce9-b225-d18452e49150-metrics-certs podName:8498e9d1-1eeb-4ce9-b225-d18452e49150 nodeName:}" failed. No retries permitted until 2026-09-11 18:48:53.908088869 +0000 UTC m=+9.985679404 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8498e9d1-1eeb-4ce9-b225-d18452e49150-metrics-certs") pod "network-metrics-daemon-bjcnp" (UID: "8498e9d1-1eeb-4ce9-b225-d18452e49150") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 18:48:50.009099 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:50.009062    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8h4f\" (UniqueName: \"kubernetes.io/projected/c7382740-2bba-4368-8e7f-508c64d170a7-kube-api-access-f8h4f\") pod \"network-check-target-7c8ns\" (UID: \"c7382740-2bba-4368-8e7f-508c64d170a7\") " pod="openshift-network-diagnostics/network-check-target-7c8ns"
Sep 11 18:48:50.009295 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:50.009113    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/90acf5d2-d7df-46bb-a380-73f3ae079b75-original-pull-secret\") pod \"global-pull-secret-syncer-vnkmt\" (UID: \"90acf5d2-d7df-46bb-a380-73f3ae079b75\") " pod="kube-system/global-pull-secret-syncer-vnkmt"
Sep 11 18:48:50.009295 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:48:50.009269    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 18:48:50.009416 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:48:50.009333    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90acf5d2-d7df-46bb-a380-73f3ae079b75-original-pull-secret podName:90acf5d2-d7df-46bb-a380-73f3ae079b75 nodeName:}" failed. No retries permitted until 2026-09-11 18:48:54.009313242 +0000 UTC m=+10.086903781 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/90acf5d2-d7df-46bb-a380-73f3ae079b75-original-pull-secret") pod "global-pull-secret-syncer-vnkmt" (UID: "90acf5d2-d7df-46bb-a380-73f3ae079b75") : object "kube-system"/"original-pull-secret" not registered
Sep 11 18:48:50.009818 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:48:50.009788    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 18:48:50.009818 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:48:50.009814    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 18:48:50.009985 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:48:50.009827    2578 projected.go:196] Error preparing data for projected volume kube-api-access-f8h4f for pod openshift-network-diagnostics/network-check-target-7c8ns: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 18:48:50.009985 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:48:50.009885    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c7382740-2bba-4368-8e7f-508c64d170a7-kube-api-access-f8h4f podName:c7382740-2bba-4368-8e7f-508c64d170a7 nodeName:}" failed. No retries permitted until 2026-09-11 18:48:54.009869082 +0000 UTC m=+10.087459624 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-f8h4f" (UniqueName: "kubernetes.io/projected/c7382740-2bba-4368-8e7f-508c64d170a7-kube-api-access-f8h4f") pod "network-check-target-7c8ns" (UID: "c7382740-2bba-4368-8e7f-508c64d170a7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 18:48:50.392162 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:50.392125    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7c8ns"
Sep 11 18:48:50.392355 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:50.392180    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bjcnp"
Sep 11 18:48:50.392355 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:48:50.392294    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7c8ns" podUID="c7382740-2bba-4368-8e7f-508c64d170a7"
Sep 11 18:48:50.392758 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:50.392720    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnkmt"
Sep 11 18:48:50.392858 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:48:50.392830    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bjcnp" podUID="8498e9d1-1eeb-4ce9-b225-d18452e49150"
Sep 11 18:48:50.392951 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:48:50.392927    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vnkmt" podUID="90acf5d2-d7df-46bb-a380-73f3ae079b75"
Sep 11 18:48:52.391206 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:52.390391    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnkmt"
Sep 11 18:48:52.391206 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:48:52.390514    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vnkmt" podUID="90acf5d2-d7df-46bb-a380-73f3ae079b75"
Sep 11 18:48:52.391206 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:52.390936    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7c8ns"
Sep 11 18:48:52.391206 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:48:52.391029    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7c8ns" podUID="c7382740-2bba-4368-8e7f-508c64d170a7"
Sep 11 18:48:52.391206 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:52.391092    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bjcnp"
Sep 11 18:48:52.391206 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:48:52.391173    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bjcnp" podUID="8498e9d1-1eeb-4ce9-b225-d18452e49150"
Sep 11 18:48:53.945036 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:53.944994    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8498e9d1-1eeb-4ce9-b225-d18452e49150-metrics-certs\") pod \"network-metrics-daemon-bjcnp\" (UID: \"8498e9d1-1eeb-4ce9-b225-d18452e49150\") " pod="openshift-multus/network-metrics-daemon-bjcnp"
Sep 11 18:48:53.945491 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:48:53.945102    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 18:48:53.945491 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:48:53.945154    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8498e9d1-1eeb-4ce9-b225-d18452e49150-metrics-certs podName:8498e9d1-1eeb-4ce9-b225-d18452e49150 nodeName:}" failed. No retries permitted until 2026-09-11 18:49:01.945140206 +0000 UTC m=+18.022730739 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8498e9d1-1eeb-4ce9-b225-d18452e49150-metrics-certs") pod "network-metrics-daemon-bjcnp" (UID: "8498e9d1-1eeb-4ce9-b225-d18452e49150") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 18:48:54.046229 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:54.046190    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8h4f\" (UniqueName: \"kubernetes.io/projected/c7382740-2bba-4368-8e7f-508c64d170a7-kube-api-access-f8h4f\") pod \"network-check-target-7c8ns\" (UID: \"c7382740-2bba-4368-8e7f-508c64d170a7\") " pod="openshift-network-diagnostics/network-check-target-7c8ns"
Sep 11 18:48:54.046412 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:54.046248    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/90acf5d2-d7df-46bb-a380-73f3ae079b75-original-pull-secret\") pod \"global-pull-secret-syncer-vnkmt\" (UID: \"90acf5d2-d7df-46bb-a380-73f3ae079b75\") " pod="kube-system/global-pull-secret-syncer-vnkmt"
Sep 11 18:48:54.046412 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:48:54.046362    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 18:48:54.046412 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:48:54.046386    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 18:48:54.046412 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:48:54.046395    2578 projected.go:196] Error preparing data for projected volume kube-api-access-f8h4f for pod openshift-network-diagnostics/network-check-target-7c8ns: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 18:48:54.046559 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:48:54.046414    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 18:48:54.046559 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:48:54.046447    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c7382740-2bba-4368-8e7f-508c64d170a7-kube-api-access-f8h4f podName:c7382740-2bba-4368-8e7f-508c64d170a7 nodeName:}" failed. No retries permitted until 2026-09-11 18:49:02.046433676 +0000 UTC m=+18.124024221 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-f8h4f" (UniqueName: "kubernetes.io/projected/c7382740-2bba-4368-8e7f-508c64d170a7-kube-api-access-f8h4f") pod "network-check-target-7c8ns" (UID: "c7382740-2bba-4368-8e7f-508c64d170a7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 18:48:54.046559 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:48:54.046471    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90acf5d2-d7df-46bb-a380-73f3ae079b75-original-pull-secret podName:90acf5d2-d7df-46bb-a380-73f3ae079b75 nodeName:}" failed. No retries permitted until 2026-09-11 18:49:02.046455011 +0000 UTC m=+18.124045546 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/90acf5d2-d7df-46bb-a380-73f3ae079b75-original-pull-secret") pod "global-pull-secret-syncer-vnkmt" (UID: "90acf5d2-d7df-46bb-a380-73f3ae079b75") : object "kube-system"/"original-pull-secret" not registered
Sep 11 18:48:54.394822 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:54.394798    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7c8ns"
Sep 11 18:48:54.394996 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:48:54.394889    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7c8ns" podUID="c7382740-2bba-4368-8e7f-508c64d170a7"
Sep 11 18:48:54.395223 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:54.395208    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bjcnp"
Sep 11 18:48:54.395330 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:48:54.395305    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bjcnp" podUID="8498e9d1-1eeb-4ce9-b225-d18452e49150"
Sep 11 18:48:54.395447 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:54.395354    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnkmt"
Sep 11 18:48:54.395447 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:48:54.395422    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vnkmt" podUID="90acf5d2-d7df-46bb-a380-73f3ae079b75"
Sep 11 18:48:56.395842 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:56.395812    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnkmt"
Sep 11 18:48:56.396318 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:56.395925    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7c8ns"
Sep 11 18:48:56.396318 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:48:56.395933    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vnkmt" podUID="90acf5d2-d7df-46bb-a380-73f3ae079b75"
Sep 11 18:48:56.396318 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:56.395954    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bjcnp"
Sep 11 18:48:56.396318 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:48:56.396034    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7c8ns" podUID="c7382740-2bba-4368-8e7f-508c64d170a7"
Sep 11 18:48:56.396318 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:48:56.396109    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bjcnp" podUID="8498e9d1-1eeb-4ce9-b225-d18452e49150"
Sep 11 18:48:58.390887 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:58.390807    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnkmt"
Sep 11 18:48:58.390887 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:58.390846    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7c8ns"
Sep 11 18:48:58.390887 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:48:58.390814    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bjcnp"
Sep 11 18:48:58.391476 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:48:58.390928    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vnkmt" podUID="90acf5d2-d7df-46bb-a380-73f3ae079b75"
Sep 11 18:48:58.391476 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:48:58.391040    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bjcnp" podUID="8498e9d1-1eeb-4ce9-b225-d18452e49150"
Sep 11 18:48:58.391476 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:48:58.391130    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7c8ns" podUID="c7382740-2bba-4368-8e7f-508c64d170a7"
Sep 11 18:49:00.390912 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:00.390875    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7c8ns"
Sep 11 18:49:00.391338 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:00.390875    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnkmt"
Sep 11 18:49:00.391338 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:00.390988    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7c8ns" podUID="c7382740-2bba-4368-8e7f-508c64d170a7"
Sep 11 18:49:00.391338 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:00.391090    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vnkmt" podUID="90acf5d2-d7df-46bb-a380-73f3ae079b75"
Sep 11 18:49:00.391338 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:00.390875    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bjcnp"
Sep 11 18:49:00.391338 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:00.391209    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bjcnp" podUID="8498e9d1-1eeb-4ce9-b225-d18452e49150"
Sep 11 18:49:02.005026 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:02.004991    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8498e9d1-1eeb-4ce9-b225-d18452e49150-metrics-certs\") pod \"network-metrics-daemon-bjcnp\" (UID: \"8498e9d1-1eeb-4ce9-b225-d18452e49150\") " pod="openshift-multus/network-metrics-daemon-bjcnp"
Sep 11 18:49:02.005502 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:02.005101    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 18:49:02.005502 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:02.005152    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8498e9d1-1eeb-4ce9-b225-d18452e49150-metrics-certs podName:8498e9d1-1eeb-4ce9-b225-d18452e49150 nodeName:}" failed. No retries permitted until 2026-09-11 18:49:18.005138844 +0000 UTC m=+34.082729376 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8498e9d1-1eeb-4ce9-b225-d18452e49150-metrics-certs") pod "network-metrics-daemon-bjcnp" (UID: "8498e9d1-1eeb-4ce9-b225-d18452e49150") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 18:49:02.105833 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:02.105802    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8h4f\" (UniqueName: \"kubernetes.io/projected/c7382740-2bba-4368-8e7f-508c64d170a7-kube-api-access-f8h4f\") pod \"network-check-target-7c8ns\" (UID: \"c7382740-2bba-4368-8e7f-508c64d170a7\") " pod="openshift-network-diagnostics/network-check-target-7c8ns"
Sep 11 18:49:02.105833 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:02.105840    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/90acf5d2-d7df-46bb-a380-73f3ae079b75-original-pull-secret\") pod \"global-pull-secret-syncer-vnkmt\" (UID: \"90acf5d2-d7df-46bb-a380-73f3ae079b75\") " pod="kube-system/global-pull-secret-syncer-vnkmt"
Sep 11 18:49:02.106074 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:02.105964    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 18:49:02.106074 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:02.105981    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 18:49:02.106074 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:02.106006    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 18:49:02.106074 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:02.106018    2578 projected.go:196] Error preparing data for projected volume kube-api-access-f8h4f for pod openshift-network-diagnostics/network-check-target-7c8ns: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 18:49:02.106074 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:02.106029    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90acf5d2-d7df-46bb-a380-73f3ae079b75-original-pull-secret podName:90acf5d2-d7df-46bb-a380-73f3ae079b75 nodeName:}" failed. No retries permitted until 2026-09-11 18:49:18.106010354 +0000 UTC m=+34.183600892 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/90acf5d2-d7df-46bb-a380-73f3ae079b75-original-pull-secret") pod "global-pull-secret-syncer-vnkmt" (UID: "90acf5d2-d7df-46bb-a380-73f3ae079b75") : object "kube-system"/"original-pull-secret" not registered
Sep 11 18:49:02.106074 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:02.106062    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c7382740-2bba-4368-8e7f-508c64d170a7-kube-api-access-f8h4f podName:c7382740-2bba-4368-8e7f-508c64d170a7 nodeName:}" failed. No retries permitted until 2026-09-11 18:49:18.106051295 +0000 UTC m=+34.183641828 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-f8h4f" (UniqueName: "kubernetes.io/projected/c7382740-2bba-4368-8e7f-508c64d170a7-kube-api-access-f8h4f") pod "network-check-target-7c8ns" (UID: "c7382740-2bba-4368-8e7f-508c64d170a7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 18:49:02.391037 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:02.390950    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7c8ns"
Sep 11 18:49:02.391196 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:02.390949    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnkmt"
Sep 11 18:49:02.391196 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:02.391086    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7c8ns" podUID="c7382740-2bba-4368-8e7f-508c64d170a7"
Sep 11 18:49:02.391196 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:02.390949    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bjcnp"
Sep 11 18:49:02.391196 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:02.391172    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vnkmt" podUID="90acf5d2-d7df-46bb-a380-73f3ae079b75"
Sep 11 18:49:02.391394 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:02.391249    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bjcnp" podUID="8498e9d1-1eeb-4ce9-b225-d18452e49150"
Sep 11 18:49:04.391265 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:04.391222    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnkmt"
Sep 11 18:49:04.391747 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:04.391316    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vnkmt" podUID="90acf5d2-d7df-46bb-a380-73f3ae079b75"
Sep 11 18:49:04.391747 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:04.391398    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7c8ns"
Sep 11 18:49:04.391747 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:04.391511    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7c8ns" podUID="c7382740-2bba-4368-8e7f-508c64d170a7"
Sep 11 18:49:04.391747 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:04.391543    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bjcnp"
Sep 11 18:49:04.391747 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:04.391656    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bjcnp" podUID="8498e9d1-1eeb-4ce9-b225-d18452e49150"
Sep 11 18:49:06.391345 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:06.391201    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bjcnp"
Sep 11 18:49:06.391828 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:06.391429    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bjcnp" podUID="8498e9d1-1eeb-4ce9-b225-d18452e49150"
Sep 11 18:49:06.391828 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:06.391233    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnkmt"
Sep 11 18:49:06.391828 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:06.391488    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vnkmt" podUID="90acf5d2-d7df-46bb-a380-73f3ae079b75"
Sep 11 18:49:06.391828 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:06.391201    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7c8ns"
Sep 11 18:49:06.391828 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:06.391536    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7c8ns" podUID="c7382740-2bba-4368-8e7f-508c64d170a7"
Sep 11 18:49:06.492557 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:06.492523    2578 generic.go:356] "Generic (PLEG): container finished" podID="c088609c-51c3-4070-88eb-a329ef4204a7" containerID="1c782e85d39824b5f069a2bb3c67e0509e8d08c4756ff2ce52c8dbb9f0f74865" exitCode=0
Sep 11 18:49:06.492688 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:06.492561    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dc7p2" event={"ID":"c088609c-51c3-4070-88eb-a329ef4204a7","Type":"ContainerDied","Data":"1c782e85d39824b5f069a2bb3c67e0509e8d08c4756ff2ce52c8dbb9f0f74865"}
Sep 11 18:49:06.497026 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:06.496999    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plbjn" event={"ID":"4c6ece11-f761-4493-ba21-fa07150f8ec3","Type":"ContainerStarted","Data":"3917751ca7174cf16e73bf4c24259e54e9f0afa98a9eb4fa948749d7229cfb98"}
Sep 11 18:49:06.499490 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:06.499468    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8lq2v" event={"ID":"679f1bfe-400c-46c5-837e-39d1dccccf5f","Type":"ContainerStarted","Data":"5f69f62f596caedbbd624f7b7a5a05e79bf189afb2647a030e44759c58e70ba5"}
Sep 11 18:49:06.501354 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:06.501323    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-wmk8r" event={"ID":"107db790-53d5-4969-8bf9-2ee3b9f9df09","Type":"ContainerStarted","Data":"a68a59e4340fc5dc9422252b598e507175945b707bd79574ab762795df4bca83"}
Sep 11 18:49:06.503053 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:06.503026    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dnzn9" event={"ID":"43683762-3da6-4f69-a5ad-d685d63bbe4f","Type":"ContainerStarted","Data":"bc372b54d5260ab2162211360ee2e928c95cbcc606083dcb1ff079fb7e80d4ef"}
Sep 11 18:49:06.505320 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:06.505299    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7xkpg" event={"ID":"23c2932c-66b4-4b05-825e-afbbc27f110e","Type":"ContainerStarted","Data":"3d45227942469e0abdbe669dfd7eec163a7f62021575e3f415157a03649d0ee1"}
Sep 11 18:49:06.505414 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:06.505353    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7xkpg" event={"ID":"23c2932c-66b4-4b05-825e-afbbc27f110e","Type":"ContainerStarted","Data":"4c3f5c58ba0b671a8d6f28ceda5498f71bb7b306ee98b4673aa3066771027ba8"}
Sep 11 18:49:06.505414 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:06.505388    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7xkpg" event={"ID":"23c2932c-66b4-4b05-825e-afbbc27f110e","Type":"ContainerStarted","Data":"f2e428b7da96112366e61d5b5923a1abf63f9f812db48cb462f97c601d6afdf9"}
Sep 11 18:49:06.507147 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:06.507124    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dcbq8" event={"ID":"16f7a243-b894-446f-80f7-fe1ae5a8d66c","Type":"ContainerStarted","Data":"8f64f104edd3e7a66344d3e0e4f107538a53e604550f9513c7086a31621d2360"}
Sep 11 18:49:06.508870 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:06.508848    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fqd6d" event={"ID":"240ee9ec-9e58-435f-938f-62cf71acfb5b","Type":"ContainerStarted","Data":"ca5194e7c38c9ce023e14739df3dfc85cdd3d19d19d906d3b70065acd7baef17"}
Sep 11 18:49:06.513252 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:06.513218    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-189.ec2.internal" podStartSLOduration=21.513207214 podStartE2EDuration="21.513207214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:48:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:48:49.472208421 +0000 UTC m=+5.549798999" watchObservedRunningTime="2026-09-11 18:49:06.513207214 +0000 UTC m=+22.590797769"
Sep 11 18:49:06.527716 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:06.527680    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-fqd6d" podStartSLOduration=3.765098755 podStartE2EDuration="22.527669368s" totalImagesPullingTime="18.762570613s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:48:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:49:06.527484829 +0000 UTC m=+22.605075401" watchObservedRunningTime="2026-09-11 18:49:06.527669368 +0000 UTC m=+22.605259923"
Sep 11 18:49:06.542519 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:06.542480    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-8lq2v" podStartSLOduration=3.781899495 podStartE2EDuration="22.542466694s" totalImagesPullingTime="18.760567199s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:48:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:49:06.541825134 +0000 UTC m=+22.619415711" watchObservedRunningTime="2026-09-11 18:49:06.542466694 +0000 UTC m=+22.620057248"
Sep 11 18:49:06.555848 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:06.555808    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-wmk8r" podStartSLOduration=3.804696812 podStartE2EDuration="22.555795852s" totalImagesPullingTime="18.75109904s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:48:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:49:06.555662788 +0000 UTC m=+22.633253345" watchObservedRunningTime="2026-09-11 18:49:06.555795852 +0000 UTC m=+22.633386403"
Sep 11 18:49:06.569448 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:06.569416    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-dnzn9" podStartSLOduration=3.812939427 podStartE2EDuration="22.56940418s" totalImagesPullingTime="18.756464753s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:48:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:49:06.568990411 +0000 UTC m=+22.646580966" watchObservedRunningTime="2026-09-11 18:49:06.56940418 +0000 UTC m=+22.646994735"
Sep 11 18:49:06.592066 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:06.592037    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-dcbq8" podStartSLOduration=2.785090829 podStartE2EDuration="21.592028251s" totalImagesPullingTime="18.806937422s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:48:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:49:06.592023181 +0000 UTC m=+22.669613737" watchObservedRunningTime="2026-09-11 18:49:06.592028251 +0000 UTC m=+22.669618797"
Sep 11 18:49:07.516041 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:07.515639    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7xkpg" event={"ID":"23c2932c-66b4-4b05-825e-afbbc27f110e","Type":"ContainerStarted","Data":"174ae28f6689ac8db57de65b0dd6d100c8d719e1db1a73563b8ea63885546f22"}
Sep 11 18:49:07.516041 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:07.515927    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7xkpg" event={"ID":"23c2932c-66b4-4b05-825e-afbbc27f110e","Type":"ContainerStarted","Data":"789ff2be491cab33331c6f9824292a7fae6efb957b06008339e90d6468bad94d"}
Sep 11 18:49:07.516041 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:07.515984    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7xkpg" event={"ID":"23c2932c-66b4-4b05-825e-afbbc27f110e","Type":"ContainerStarted","Data":"1deec3f785916f0076f9cb6ba5d5045bdd11e009dadea25df54c01f245062d12"}
Sep 11 18:49:07.517290 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:07.517263    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hh9vt" event={"ID":"0905525e-5ad4-4f0a-882d-d9d55c922e0b","Type":"ContainerStarted","Data":"7fca6f1e94fea9e3d8be5263da9696d55638f3133f502487a4d539c43d595f52"}
Sep 11 18:49:07.533459 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:07.533411    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-hh9vt" podStartSLOduration=4.756171011 podStartE2EDuration="23.533392079s" totalImagesPullingTime="18.777221068s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:48:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:49:07.532850811 +0000 UTC m=+23.610441378" watchObservedRunningTime="2026-09-11 18:49:07.533392079 +0000 UTC m=+23.610982634"
Sep 11 18:49:07.693622 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:07.693578    2578 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 11 18:49:08.319110 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:08.318984    2578 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-11T18:49:07.693617997Z","UUID":"6fb2c9c0-fdd0-44d6-827d-00d7c0df6127","Handler":null,"Name":"","Endpoint":""}
Sep 11 18:49:08.320612 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:08.320579    2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 11 18:49:08.320738 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:08.320631    2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 11 18:49:08.391287 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:08.391255    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bjcnp"
Sep 11 18:49:08.391456 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:08.391399    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bjcnp" podUID="8498e9d1-1eeb-4ce9-b225-d18452e49150"
Sep 11 18:49:08.391837 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:08.391807    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnkmt"
Sep 11 18:49:08.391954 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:08.391900    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vnkmt" podUID="90acf5d2-d7df-46bb-a380-73f3ae079b75"
Sep 11 18:49:08.392016 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:08.391961    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7c8ns"
Sep 11 18:49:08.392016 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:08.392006    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7c8ns" podUID="c7382740-2bba-4368-8e7f-508c64d170a7"
Sep 11 18:49:08.522311 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:08.522271    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plbjn" event={"ID":"4c6ece11-f761-4493-ba21-fa07150f8ec3","Type":"ContainerStarted","Data":"a57339b3186f4257e35ae56e35d2f9693842de208d182d257788c885294bc276"}
Sep 11 18:49:09.528343 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:09.528302    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7xkpg" event={"ID":"23c2932c-66b4-4b05-825e-afbbc27f110e","Type":"ContainerStarted","Data":"7cc6949938737c7b8e0a737d7282ad33893bde2b92c10546f1c191223f249314"}
Sep 11 18:49:09.530710 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:09.530661    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plbjn" event={"ID":"4c6ece11-f761-4493-ba21-fa07150f8ec3","Type":"ContainerStarted","Data":"ab4fc3b495064ba583ff1ca41c8f4a330909032f2bac52c3c39f639a3eabaee5"}
Sep 11 18:49:09.550328 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:09.550284    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plbjn" podStartSLOduration=4.632939158 podStartE2EDuration="25.550269306s" totalImagesPullingTime="20.917330148s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:48:44 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:49:09.549417343 +0000 UTC m=+25.627007913" watchObservedRunningTime="2026-09-11 18:49:09.550269306 +0000 UTC m=+25.627859861"
Sep 11 18:49:10.390300 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:10.390271    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7c8ns"
Sep 11 18:49:10.390300 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:10.390287    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bjcnp"
Sep 11 18:49:10.390514 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:10.390387    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7c8ns" podUID="c7382740-2bba-4368-8e7f-508c64d170a7"
Sep 11 18:49:10.390878 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:10.390841    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bjcnp" podUID="8498e9d1-1eeb-4ce9-b225-d18452e49150"
Sep 11 18:49:10.390991 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:10.390899    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnkmt"
Sep 11 18:49:10.390991 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:10.390973    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vnkmt" podUID="90acf5d2-d7df-46bb-a380-73f3ae079b75"
Sep 11 18:49:10.672935 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:10.672866    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-8lq2v"
Sep 11 18:49:10.673373 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:10.673357    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-8lq2v"
Sep 11 18:49:11.539796 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:11.539578    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7xkpg" event={"ID":"23c2932c-66b4-4b05-825e-afbbc27f110e","Type":"ContainerStarted","Data":"8aa73f8eaa5131d4df37c7fb210cca778e10d4ecbda977cca2f7ac78797fe3c9"}
Sep 11 18:49:11.540015 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:11.539888    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7xkpg"
Sep 11 18:49:11.540015 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:11.539946    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7xkpg"
Sep 11 18:49:11.540015 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:11.539980    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7xkpg"
Sep 11 18:49:11.541937 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:11.541909    2578 generic.go:356] "Generic (PLEG): container finished" podID="c088609c-51c3-4070-88eb-a329ef4204a7" containerID="d89ad5fc7c2c039fc2f8f12529ebb0ab44f3a5ee964f63854862067479977e5f" exitCode=0
Sep 11 18:49:11.542085 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:11.542002    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dc7p2" event={"ID":"c088609c-51c3-4070-88eb-a329ef4204a7","Type":"ContainerDied","Data":"d89ad5fc7c2c039fc2f8f12529ebb0ab44f3a5ee964f63854862067479977e5f"}
Sep 11 18:49:11.542265 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:11.542247    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-8lq2v"
Sep 11 18:49:11.542703 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:11.542588    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-8lq2v"
Sep 11 18:49:11.555693 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:11.555675    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7xkpg"
Sep 11 18:49:11.555764 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:11.555758    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7xkpg"
Sep 11 18:49:11.581012 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:11.580969    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-7xkpg" podStartSLOduration=8.76916703 podStartE2EDuration="27.580957774s" totalImagesPullingTime="18.811790744s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:48:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:49:11.579910259 +0000 UTC m=+27.657500833" watchObservedRunningTime="2026-09-11 18:49:11.580957774 +0000 UTC m=+27.658548329"
Sep 11 18:49:12.391160 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:12.391130    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bjcnp"
Sep 11 18:49:12.391674 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:12.391130    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7c8ns"
Sep 11 18:49:12.391674 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:12.391274    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bjcnp" podUID="8498e9d1-1eeb-4ce9-b225-d18452e49150"
Sep 11 18:49:12.391674 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:12.391139    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnkmt"
Sep 11 18:49:12.391674 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:12.391323    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7c8ns" podUID="c7382740-2bba-4368-8e7f-508c64d170a7"
Sep 11 18:49:12.391674 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:12.391393    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vnkmt" podUID="90acf5d2-d7df-46bb-a380-73f3ae079b75"
Sep 11 18:49:12.906647 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:12.906611    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7c8ns"]
Sep 11 18:49:12.906837 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:12.906763    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7c8ns"
Sep 11 18:49:12.906912 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:12.906887    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7c8ns" podUID="c7382740-2bba-4368-8e7f-508c64d170a7"
Sep 11 18:49:12.909999 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:12.909975    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vnkmt"]
Sep 11 18:49:12.910129 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:12.910065    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnkmt"
Sep 11 18:49:12.910185 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:12.910160    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vnkmt" podUID="90acf5d2-d7df-46bb-a380-73f3ae079b75"
Sep 11 18:49:12.910729 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:12.910707    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bjcnp"]
Sep 11 18:49:12.910827 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:12.910814    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bjcnp"
Sep 11 18:49:12.910953 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:12.910931    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bjcnp" podUID="8498e9d1-1eeb-4ce9-b225-d18452e49150"
Sep 11 18:49:13.548805 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:13.548630    2578 generic.go:356] "Generic (PLEG): container finished" podID="c088609c-51c3-4070-88eb-a329ef4204a7" containerID="c30944a8b151505e328fcd6e420ab4f7ac893fb7a37f23c0595857af0f72901a" exitCode=0
Sep 11 18:49:13.549486 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:13.548711    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dc7p2" event={"ID":"c088609c-51c3-4070-88eb-a329ef4204a7","Type":"ContainerDied","Data":"c30944a8b151505e328fcd6e420ab4f7ac893fb7a37f23c0595857af0f72901a"}
Sep 11 18:49:14.396544 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:14.394769    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnkmt"
Sep 11 18:49:14.396544 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:14.394920    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vnkmt" podUID="90acf5d2-d7df-46bb-a380-73f3ae079b75"
Sep 11 18:49:14.396544 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:14.394997    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7c8ns"
Sep 11 18:49:14.396544 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:14.395083    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7c8ns" podUID="c7382740-2bba-4368-8e7f-508c64d170a7"
Sep 11 18:49:14.396544 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:14.395150    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bjcnp"
Sep 11 18:49:14.396544 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:14.395255    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bjcnp" podUID="8498e9d1-1eeb-4ce9-b225-d18452e49150"
Sep 11 18:49:14.554064 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:14.554028    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dc7p2" event={"ID":"c088609c-51c3-4070-88eb-a329ef4204a7","Type":"ContainerStarted","Data":"01b6f6cafe4c40e735995d6e2c59fd069b4bbda519a9c15226120c648e76a8bd"}
Sep 11 18:49:15.558304 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:15.558270    2578 generic.go:356] "Generic (PLEG): container finished" podID="c088609c-51c3-4070-88eb-a329ef4204a7" containerID="01b6f6cafe4c40e735995d6e2c59fd069b4bbda519a9c15226120c648e76a8bd" exitCode=0
Sep 11 18:49:15.558696 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:15.558327    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dc7p2" event={"ID":"c088609c-51c3-4070-88eb-a329ef4204a7","Type":"ContainerDied","Data":"01b6f6cafe4c40e735995d6e2c59fd069b4bbda519a9c15226120c648e76a8bd"}
Sep 11 18:49:16.390720 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:16.390689    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnkmt"
Sep 11 18:49:16.390720 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:16.390716    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7c8ns"
Sep 11 18:49:16.390971 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:16.390746    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bjcnp"
Sep 11 18:49:16.390971 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:16.390844    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7c8ns" podUID="c7382740-2bba-4368-8e7f-508c64d170a7"
Sep 11 18:49:16.390971 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:16.390893    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bjcnp" podUID="8498e9d1-1eeb-4ce9-b225-d18452e49150"
Sep 11 18:49:16.391123 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:16.390980    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vnkmt" podUID="90acf5d2-d7df-46bb-a380-73f3ae079b75"
Sep 11 18:49:18.024281 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:18.024247    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8498e9d1-1eeb-4ce9-b225-d18452e49150-metrics-certs\") pod \"network-metrics-daemon-bjcnp\" (UID: \"8498e9d1-1eeb-4ce9-b225-d18452e49150\") " pod="openshift-multus/network-metrics-daemon-bjcnp"
Sep 11 18:49:18.024754 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:18.024417    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 18:49:18.024754 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:18.024480    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8498e9d1-1eeb-4ce9-b225-d18452e49150-metrics-certs podName:8498e9d1-1eeb-4ce9-b225-d18452e49150 nodeName:}" failed. No retries permitted until 2026-09-11 18:49:50.024465241 +0000 UTC m=+66.102055773 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8498e9d1-1eeb-4ce9-b225-d18452e49150-metrics-certs") pod "network-metrics-daemon-bjcnp" (UID: "8498e9d1-1eeb-4ce9-b225-d18452e49150") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 18:49:18.125262 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:18.125216    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8h4f\" (UniqueName: \"kubernetes.io/projected/c7382740-2bba-4368-8e7f-508c64d170a7-kube-api-access-f8h4f\") pod \"network-check-target-7c8ns\" (UID: \"c7382740-2bba-4368-8e7f-508c64d170a7\") " pod="openshift-network-diagnostics/network-check-target-7c8ns"
Sep 11 18:49:18.125262 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:18.125262    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/90acf5d2-d7df-46bb-a380-73f3ae079b75-original-pull-secret\") pod \"global-pull-secret-syncer-vnkmt\" (UID: \"90acf5d2-d7df-46bb-a380-73f3ae079b75\") " pod="kube-system/global-pull-secret-syncer-vnkmt"
Sep 11 18:49:18.125497 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:18.125396    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 18:49:18.125497 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:18.125401    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 18:49:18.125497 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:18.125426    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 18:49:18.125497 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:18.125440    2578 projected.go:196] Error preparing data for projected volume kube-api-access-f8h4f for pod openshift-network-diagnostics/network-check-target-7c8ns: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 18:49:18.125497 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:18.125461    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90acf5d2-d7df-46bb-a380-73f3ae079b75-original-pull-secret podName:90acf5d2-d7df-46bb-a380-73f3ae079b75 nodeName:}" failed. No retries permitted until 2026-09-11 18:49:50.125443961 +0000 UTC m=+66.203034498 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/90acf5d2-d7df-46bb-a380-73f3ae079b75-original-pull-secret") pod "global-pull-secret-syncer-vnkmt" (UID: "90acf5d2-d7df-46bb-a380-73f3ae079b75") : object "kube-system"/"original-pull-secret" not registered
Sep 11 18:49:18.125497 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:18.125488    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c7382740-2bba-4368-8e7f-508c64d170a7-kube-api-access-f8h4f podName:c7382740-2bba-4368-8e7f-508c64d170a7 nodeName:}" failed. No retries permitted until 2026-09-11 18:49:50.125475077 +0000 UTC m=+66.203065614 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-f8h4f" (UniqueName: "kubernetes.io/projected/c7382740-2bba-4368-8e7f-508c64d170a7-kube-api-access-f8h4f") pod "network-check-target-7c8ns" (UID: "c7382740-2bba-4368-8e7f-508c64d170a7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 18:49:18.394543 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:18.394472    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnkmt"
Sep 11 18:49:18.394724 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:18.394473    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bjcnp"
Sep 11 18:49:18.394724 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:18.394618    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vnkmt" podUID="90acf5d2-d7df-46bb-a380-73f3ae079b75"
Sep 11 18:49:18.394724 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:18.394471    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7c8ns"
Sep 11 18:49:18.394724 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:18.394703    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bjcnp" podUID="8498e9d1-1eeb-4ce9-b225-d18452e49150"
Sep 11 18:49:18.394894 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:18.394779    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7c8ns" podUID="c7382740-2bba-4368-8e7f-508c64d170a7"
Sep 11 18:49:19.571941 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:19.571910    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dc7p2" event={"ID":"c088609c-51c3-4070-88eb-a329ef4204a7","Type":"ContainerStarted","Data":"0538095995f1b554354b0f590bde6e503ad0a1b28c61abf4cf58b46956143739"}
Sep 11 18:49:19.734640 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:19.734612    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-189.ec2.internal" event="NodeReady"
Sep 11 18:49:19.734785 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:19.734773    2578 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 11 18:49:19.801191 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:19.801162    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-9tzdn"]
Sep 11 18:49:19.803145 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:19.803124    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9tzdn"
Sep 11 18:49:19.804138 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:19.804121    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-bw8xb"]
Sep 11 18:49:19.805672 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:19.805650    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 11 18:49:19.805850 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:19.805835    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bw8xb"
Sep 11 18:49:19.806322 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:19.806305    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vlxp2\""
Sep 11 18:49:19.806322 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:19.806316    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 11 18:49:19.807780 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:19.807762    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 11 18:49:19.807838 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:19.807762    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 11 18:49:19.808129 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:19.808113    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 11 18:49:19.808330 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:19.808315    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-shf8m\""
Sep 11 18:49:19.822083 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:19.822064    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bw8xb"]
Sep 11 18:49:19.828627 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:19.828497    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9tzdn"]
Sep 11 18:49:19.938202 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:19.938109    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9a3b9624-a629-437f-a700-3f24e40fc0ae-metrics-tls\") pod \"dns-default-9tzdn\" (UID: \"9a3b9624-a629-437f-a700-3f24e40fc0ae\") " pod="openshift-dns/dns-default-9tzdn"
Sep 11 18:49:19.938202 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:19.938151    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jz49t\" (UniqueName: \"kubernetes.io/projected/0e108d53-e1ef-4459-9eee-2e4be2ccf6e3-kube-api-access-jz49t\") pod \"ingress-canary-bw8xb\" (UID: \"0e108d53-e1ef-4459-9eee-2e4be2ccf6e3\") " pod="openshift-ingress-canary/ingress-canary-bw8xb"
Sep 11 18:49:19.938202 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:19.938179    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9a3b9624-a629-437f-a700-3f24e40fc0ae-config-volume\") pod \"dns-default-9tzdn\" (UID: \"9a3b9624-a629-437f-a700-3f24e40fc0ae\") " pod="openshift-dns/dns-default-9tzdn"
Sep 11 18:49:19.938202 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:19.938201    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsqb4\" (UniqueName: \"kubernetes.io/projected/9a3b9624-a629-437f-a700-3f24e40fc0ae-kube-api-access-jsqb4\") pod \"dns-default-9tzdn\" (UID: \"9a3b9624-a629-437f-a700-3f24e40fc0ae\") " pod="openshift-dns/dns-default-9tzdn"
Sep 11 18:49:19.938482 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:19.938272    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0e108d53-e1ef-4459-9eee-2e4be2ccf6e3-cert\") pod \"ingress-canary-bw8xb\" (UID: \"0e108d53-e1ef-4459-9eee-2e4be2ccf6e3\") " pod="openshift-ingress-canary/ingress-canary-bw8xb"
Sep 11 18:49:19.938482 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:19.938323    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a3b9624-a629-437f-a700-3f24e40fc0ae-tmp-dir\") pod \"dns-default-9tzdn\" (UID: \"9a3b9624-a629-437f-a700-3f24e40fc0ae\") " pod="openshift-dns/dns-default-9tzdn"
Sep 11 18:49:20.039454 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:20.039417    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9a3b9624-a629-437f-a700-3f24e40fc0ae-metrics-tls\") pod \"dns-default-9tzdn\" (UID: \"9a3b9624-a629-437f-a700-3f24e40fc0ae\") " pod="openshift-dns/dns-default-9tzdn"
Sep 11 18:49:20.039590 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:20.039465    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jz49t\" (UniqueName: \"kubernetes.io/projected/0e108d53-e1ef-4459-9eee-2e4be2ccf6e3-kube-api-access-jz49t\") pod \"ingress-canary-bw8xb\" (UID: \"0e108d53-e1ef-4459-9eee-2e4be2ccf6e3\") " pod="openshift-ingress-canary/ingress-canary-bw8xb"
Sep 11 18:49:20.039590 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:20.039538    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9a3b9624-a629-437f-a700-3f24e40fc0ae-config-volume\") pod \"dns-default-9tzdn\" (UID: \"9a3b9624-a629-437f-a700-3f24e40fc0ae\") " pod="openshift-dns/dns-default-9tzdn"
Sep 11 18:49:20.039590 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:20.039560    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsqb4\" (UniqueName: \"kubernetes.io/projected/9a3b9624-a629-437f-a700-3f24e40fc0ae-kube-api-access-jsqb4\") pod \"dns-default-9tzdn\" (UID: \"9a3b9624-a629-437f-a700-3f24e40fc0ae\") " pod="openshift-dns/dns-default-9tzdn"
Sep 11 18:49:20.039590 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:20.039563    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 18:49:20.039817 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:20.039628    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0e108d53-e1ef-4459-9eee-2e4be2ccf6e3-cert\") pod \"ingress-canary-bw8xb\" (UID: \"0e108d53-e1ef-4459-9eee-2e4be2ccf6e3\") " pod="openshift-ingress-canary/ingress-canary-bw8xb"
Sep 11 18:49:20.039817 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:20.039670    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a3b9624-a629-437f-a700-3f24e40fc0ae-metrics-tls podName:9a3b9624-a629-437f-a700-3f24e40fc0ae nodeName:}" failed. No retries permitted until 2026-09-11 18:49:20.539648449 +0000 UTC m=+36.617238984 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9a3b9624-a629-437f-a700-3f24e40fc0ae-metrics-tls") pod "dns-default-9tzdn" (UID: "9a3b9624-a629-437f-a700-3f24e40fc0ae") : secret "dns-default-metrics-tls" not found
Sep 11 18:49:20.039817 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:20.039697    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a3b9624-a629-437f-a700-3f24e40fc0ae-tmp-dir\") pod \"dns-default-9tzdn\" (UID: \"9a3b9624-a629-437f-a700-3f24e40fc0ae\") " pod="openshift-dns/dns-default-9tzdn"
Sep 11 18:49:20.040387 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:20.040198    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a3b9624-a629-437f-a700-3f24e40fc0ae-tmp-dir\") pod \"dns-default-9tzdn\" (UID: \"9a3b9624-a629-437f-a700-3f24e40fc0ae\") " pod="openshift-dns/dns-default-9tzdn"
Sep 11 18:49:20.040425 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:20.040318    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9a3b9624-a629-437f-a700-3f24e40fc0ae-config-volume\") pod \"dns-default-9tzdn\" (UID: \"9a3b9624-a629-437f-a700-3f24e40fc0ae\") " pod="openshift-dns/dns-default-9tzdn"
Sep 11 18:49:20.043938 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:20.043920    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0e108d53-e1ef-4459-9eee-2e4be2ccf6e3-cert\") pod \"ingress-canary-bw8xb\" (UID: \"0e108d53-e1ef-4459-9eee-2e4be2ccf6e3\") " pod="openshift-ingress-canary/ingress-canary-bw8xb"
Sep 11 18:49:20.049492 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:20.049465    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jz49t\" (UniqueName: \"kubernetes.io/projected/0e108d53-e1ef-4459-9eee-2e4be2ccf6e3-kube-api-access-jz49t\") pod \"ingress-canary-bw8xb\" (UID: \"0e108d53-e1ef-4459-9eee-2e4be2ccf6e3\") " pod="openshift-ingress-canary/ingress-canary-bw8xb"
Sep 11 18:49:20.050692 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:20.050669    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsqb4\" (UniqueName: \"kubernetes.io/projected/9a3b9624-a629-437f-a700-3f24e40fc0ae-kube-api-access-jsqb4\") pod \"dns-default-9tzdn\" (UID: \"9a3b9624-a629-437f-a700-3f24e40fc0ae\") " pod="openshift-dns/dns-default-9tzdn"
Sep 11 18:49:20.118722 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:20.118696    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bw8xb"
Sep 11 18:49:20.391176 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:20.391142    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7c8ns"
Sep 11 18:49:20.391349 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:20.391148    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bjcnp"
Sep 11 18:49:20.391349 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:20.391265    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnkmt"
Sep 11 18:49:20.398250 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:20.398227    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 18:49:20.398395 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:20.398296    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 18:49:20.398577 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:20.398403    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 18:49:20.398577 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:20.398448    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 18:49:20.398955 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:20.398937    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-kzbqc\""
Sep 11 18:49:20.399161 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:20.399148    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-tr6q7\""
Sep 11 18:49:20.410935 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:20.410915    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bw8xb"]
Sep 11 18:49:20.412255 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:49:20.412236    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e108d53_e1ef_4459_9eee_2e4be2ccf6e3.slice/crio-eec2b97826f1afa191f32ae6d85c70f08c74ff89c57bfadb198e7a38171673e3 WatchSource:0}: Error finding container eec2b97826f1afa191f32ae6d85c70f08c74ff89c57bfadb198e7a38171673e3: Status 404 returned error can't find the container with id eec2b97826f1afa191f32ae6d85c70f08c74ff89c57bfadb198e7a38171673e3
Sep 11 18:49:20.542876 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:20.542839    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9a3b9624-a629-437f-a700-3f24e40fc0ae-metrics-tls\") pod \"dns-default-9tzdn\" (UID: \"9a3b9624-a629-437f-a700-3f24e40fc0ae\") " pod="openshift-dns/dns-default-9tzdn"
Sep 11 18:49:20.543057 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:20.542997    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 18:49:20.543126 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:20.543062    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a3b9624-a629-437f-a700-3f24e40fc0ae-metrics-tls podName:9a3b9624-a629-437f-a700-3f24e40fc0ae nodeName:}" failed. No retries permitted until 2026-09-11 18:49:21.543042014 +0000 UTC m=+37.620632547 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9a3b9624-a629-437f-a700-3f24e40fc0ae-metrics-tls") pod "dns-default-9tzdn" (UID: "9a3b9624-a629-437f-a700-3f24e40fc0ae") : secret "dns-default-metrics-tls" not found
Sep 11 18:49:20.576487 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:20.576459    2578 generic.go:356] "Generic (PLEG): container finished" podID="c088609c-51c3-4070-88eb-a329ef4204a7" containerID="0538095995f1b554354b0f590bde6e503ad0a1b28c61abf4cf58b46956143739" exitCode=0
Sep 11 18:49:20.577192 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:20.576499    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dc7p2" event={"ID":"c088609c-51c3-4070-88eb-a329ef4204a7","Type":"ContainerDied","Data":"0538095995f1b554354b0f590bde6e503ad0a1b28c61abf4cf58b46956143739"}
Sep 11 18:49:20.577794 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:20.577771    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bw8xb" event={"ID":"0e108d53-e1ef-4459-9eee-2e4be2ccf6e3","Type":"ContainerStarted","Data":"eec2b97826f1afa191f32ae6d85c70f08c74ff89c57bfadb198e7a38171673e3"}
Sep 11 18:49:21.552435 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:21.552399    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9a3b9624-a629-437f-a700-3f24e40fc0ae-metrics-tls\") pod \"dns-default-9tzdn\" (UID: \"9a3b9624-a629-437f-a700-3f24e40fc0ae\") " pod="openshift-dns/dns-default-9tzdn"
Sep 11 18:49:21.552585 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:21.552559    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 18:49:21.552674 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:21.552632    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a3b9624-a629-437f-a700-3f24e40fc0ae-metrics-tls podName:9a3b9624-a629-437f-a700-3f24e40fc0ae nodeName:}" failed. No retries permitted until 2026-09-11 18:49:23.552617521 +0000 UTC m=+39.630208067 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9a3b9624-a629-437f-a700-3f24e40fc0ae-metrics-tls") pod "dns-default-9tzdn" (UID: "9a3b9624-a629-437f-a700-3f24e40fc0ae") : secret "dns-default-metrics-tls" not found
Sep 11 18:49:21.583504 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:21.583471    2578 generic.go:356] "Generic (PLEG): container finished" podID="c088609c-51c3-4070-88eb-a329ef4204a7" containerID="ba15a6cb03619ed5e4a4b1a1670a5674b0170d40db3635c8c5e318f04e3c979b" exitCode=0
Sep 11 18:49:21.583976 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:21.583513    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dc7p2" event={"ID":"c088609c-51c3-4070-88eb-a329ef4204a7","Type":"ContainerDied","Data":"ba15a6cb03619ed5e4a4b1a1670a5674b0170d40db3635c8c5e318f04e3c979b"}
Sep 11 18:49:21.975985 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:21.975794    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-zdvjh"]
Sep 11 18:49:21.980549 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:21.980523    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-zdvjh"
Sep 11 18:49:21.982704 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:21.982681    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 11 18:49:21.982832 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:21.982683    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 11 18:49:21.983014 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:21.982999    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-wqdg6\""
Sep 11 18:49:21.988374 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:21.988345    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-zdvjh"]
Sep 11 18:49:22.057225 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:22.057187    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tr2w\" (UniqueName: \"kubernetes.io/projected/c00f816e-59b2-4fa3-a666-ac15e94fd7d3-kube-api-access-8tr2w\") pod \"volume-data-source-validator-86695b7cc7-zdvjh\" (UID: \"c00f816e-59b2-4fa3-a666-ac15e94fd7d3\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-zdvjh"
Sep 11 18:49:22.076818 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:22.076783    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-pvd8h"]
Sep 11 18:49:22.095471 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:22.095415    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-hsbsf"]
Sep 11 18:49:22.095671 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:22.095554    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-pvd8h"
Sep 11 18:49:22.100415 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:22.100393    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-rrgk5\""
Sep 11 18:49:22.100542 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:22.100458    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 11 18:49:22.100724 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:22.100687    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 11 18:49:22.100822 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:22.100776    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 11 18:49:22.101012 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:22.100993    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 11 18:49:22.121511 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:22.121491    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-pvd8h"]
Sep 11 18:49:22.121625 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:22.121527    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-hsbsf"]
Sep 11 18:49:22.121687 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:22.121622    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-hsbsf"
Sep 11 18:49:22.123672 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:22.123656    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 11 18:49:22.124050 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:22.124027    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 11 18:49:22.124344 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:22.124331    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 11 18:49:22.124631 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:22.124615    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-l28zq\""
Sep 11 18:49:22.125182 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:22.125167    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 11 18:49:22.130732 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:22.130714    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 11 18:49:22.157991 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:22.157966    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tr2w\" (UniqueName: \"kubernetes.io/projected/c00f816e-59b2-4fa3-a666-ac15e94fd7d3-kube-api-access-8tr2w\") pod \"volume-data-source-validator-86695b7cc7-zdvjh\" (UID: \"c00f816e-59b2-4fa3-a666-ac15e94fd7d3\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-zdvjh"
Sep 11 18:49:22.167449 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:22.167426    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tr2w\" (UniqueName: \"kubernetes.io/projected/c00f816e-59b2-4fa3-a666-ac15e94fd7d3-kube-api-access-8tr2w\") pod \"volume-data-source-validator-86695b7cc7-zdvjh\" (UID: \"c00f816e-59b2-4fa3-a666-ac15e94fd7d3\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-zdvjh"
Sep 11 18:49:22.185577 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:22.185547    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-nqtr7"]
Sep 11 18:49:22.203053 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:22.203017    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-jx2qj"]
Sep 11 18:49:22.204167 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:22.204145    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-nqtr7"
Sep 11 18:49:22.206466 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:22.206315    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-znvrt\""
Sep 11 18:49:22.206466 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:22.206313    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 11 18:49:22.206636 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:22.206481    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 11 18:49:22.215576 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:22.215555    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-nqtr7"]
Sep 11 18:49:22.215674 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:22.215608    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-jx2qj"]
Sep 11 18:49:22.215674 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:22.215637    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-jx2qj"
Sep 11 18:49:22.217542 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:22.217525    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-vb8bd\""
Sep 11 18:49:22.217666 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:22.217555    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 11 18:49:22.217666 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:22.217632    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 11 18:49:22.217934 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:22.217917    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 11 18:49:22.258510 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:22.258438    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2edba74f-8029-4255-980e-1132dd147c0a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-pvd8h\" (UID: \"2edba74f-8029-4255-980e-1132dd147c0a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-pvd8h"
Sep 11 18:49:22.258510 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:22.258485    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d0880abe-ec36-48e4-a511-ec663873fffa-trusted-ca\") pod \"console-operator-696884fb9c-hsbsf\" (UID: \"d0880abe-ec36-48e4-a511-ec663873fffa\") " pod="openshift-console-operator/console-operator-696884fb9c-hsbsf"
Sep 11 18:49:22.258702 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:22.258521    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d0880abe-ec36-48e4-a511-ec663873fffa-config\") pod \"console-operator-696884fb9c-hsbsf\" (UID: \"d0880abe-ec36-48e4-a511-ec663873fffa\") " pod="openshift-console-operator/console-operator-696884fb9c-hsbsf"
Sep 11 18:49:22.258702 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:22.258569    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cxm7\" (UniqueName: \"kubernetes.io/projected/d0880abe-ec36-48e4-a511-ec663873fffa-kube-api-access-8cxm7\") pod \"console-operator-696884fb9c-hsbsf\" (UID: \"d0880abe-ec36-48e4-a511-ec663873fffa\") " pod="openshift-console-operator/console-operator-696884fb9c-hsbsf"
Sep 11 18:49:22.258702 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:22.258630    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/2edba74f-8029-4255-980e-1132dd147c0a-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-pvd8h\" (UID: \"2edba74f-8029-4255-980e-1132dd147c0a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-pvd8h"
Sep 11 18:49:22.258702 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:22.258664    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm6zh\" (UniqueName: \"kubernetes.io/projected/2edba74f-8029-4255-980e-1132dd147c0a-kube-api-access-dm6zh\") pod \"cluster-monitoring-operator-6cd954b897-pvd8h\" (UID: \"2edba74f-8029-4255-980e-1132dd147c0a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-pvd8h"
Sep 11 18:49:22.258702 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:22.258686    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d0880abe-ec36-48e4-a511-ec663873fffa-serving-cert\") pod \"console-operator-696884fb9c-hsbsf\" (UID: \"d0880abe-ec36-48e4-a511-ec663873fffa\") " pod="openshift-console-operator/console-operator-696884fb9c-hsbsf"
Sep 11 18:49:22.294641 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:22.294616    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-zdvjh"
Sep 11 18:49:22.360026 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:22.359994    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d0880abe-ec36-48e4-a511-ec663873fffa-config\") pod \"console-operator-696884fb9c-hsbsf\" (UID: \"d0880abe-ec36-48e4-a511-ec663873fffa\") " pod="openshift-console-operator/console-operator-696884fb9c-hsbsf"
Sep 11 18:49:22.360200 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:22.360080    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/30878c31-acea-4b11-8991-992077a0ffca-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-jx2qj\" (UID: \"30878c31-acea-4b11-8991-992077a0ffca\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-jx2qj"
Sep 11 18:49:22.360200 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:22.360110    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dplwd\" (UniqueName: \"kubernetes.io/projected/30878c31-acea-4b11-8991-992077a0ffca-kube-api-access-dplwd\") pod \"cluster-samples-operator-86499c89c4-jx2qj\" (UID: \"30878c31-acea-4b11-8991-992077a0ffca\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-jx2qj"
Sep 11 18:49:22.360200 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:22.360133    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cxm7\" (UniqueName: \"kubernetes.io/projected/d0880abe-ec36-48e4-a511-ec663873fffa-kube-api-access-8cxm7\") pod \"console-operator-696884fb9c-hsbsf\" (UID: \"d0880abe-ec36-48e4-a511-ec663873fffa\") " pod="openshift-console-operator/console-operator-696884fb9c-hsbsf"
Sep 11 18:49:22.360200 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:22.360166    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/23b61d5d-8bb3-43c1-9ae8-38634ec8b885-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-nqtr7\" (UID: \"23b61d5d-8bb3-43c1-9ae8-38634ec8b885\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-nqtr7"
Sep 11 18:49:22.360378 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:22.360203    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/2edba74f-8029-4255-980e-1132dd147c0a-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-pvd8h\" (UID: \"2edba74f-8029-4255-980e-1132dd147c0a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-pvd8h"
Sep 11 18:49:22.360378 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:22.360247    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm6zh\" (UniqueName: \"kubernetes.io/projected/2edba74f-8029-4255-980e-1132dd147c0a-kube-api-access-dm6zh\") pod \"cluster-monitoring-operator-6cd954b897-pvd8h\" (UID: \"2edba74f-8029-4255-980e-1132dd147c0a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-pvd8h"
Sep 11 18:49:22.360378 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:22.360271    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d0880abe-ec36-48e4-a511-ec663873fffa-serving-cert\") pod \"console-operator-696884fb9c-hsbsf\" (UID: \"d0880abe-ec36-48e4-a511-ec663873fffa\") " pod="openshift-console-operator/console-operator-696884fb9c-hsbsf"
Sep 11 18:49:22.360378 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:22.360313    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2edba74f-8029-4255-980e-1132dd147c0a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-pvd8h\" (UID: \"2edba74f-8029-4255-980e-1132dd147c0a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-pvd8h"
Sep 11 18:49:22.360378 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:22.360361    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/23b61d5d-8bb3-43c1-9ae8-38634ec8b885-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-nqtr7\" (UID: \"23b61d5d-8bb3-43c1-9ae8-38634ec8b885\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-nqtr7"
Sep 11 18:49:22.360651 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:22.360393    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d0880abe-ec36-48e4-a511-ec663873fffa-trusted-ca\") pod \"console-operator-696884fb9c-hsbsf\" (UID: \"d0880abe-ec36-48e4-a511-ec663873fffa\") " pod="openshift-console-operator/console-operator-696884fb9c-hsbsf"
Sep 11 18:49:22.360814 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:22.360789    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d0880abe-ec36-48e4-a511-ec663873fffa-config\") pod \"console-operator-696884fb9c-hsbsf\" (UID: \"d0880abe-ec36-48e4-a511-ec663873fffa\") " pod="openshift-console-operator/console-operator-696884fb9c-hsbsf"
Sep 11 18:49:22.361253 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:22.361233    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d0880abe-ec36-48e4-a511-ec663873fffa-trusted-ca\") pod \"console-operator-696884fb9c-hsbsf\" (UID: \"d0880abe-ec36-48e4-a511-ec663873fffa\") " pod="openshift-console-operator/console-operator-696884fb9c-hsbsf"
Sep 11 18:49:22.361442 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:22.361419    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/2edba74f-8029-4255-980e-1132dd147c0a-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-pvd8h\" (UID: \"2edba74f-8029-4255-980e-1132dd147c0a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-pvd8h"
Sep 11 18:49:22.361504 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:22.361425    2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 11 18:49:22.361577 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:22.361550    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2edba74f-8029-4255-980e-1132dd147c0a-cluster-monitoring-operator-tls podName:2edba74f-8029-4255-980e-1132dd147c0a nodeName:}" failed. No retries permitted until 2026-09-11 18:49:22.861528708 +0000 UTC m=+38.939119243 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2edba74f-8029-4255-980e-1132dd147c0a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-pvd8h" (UID: "2edba74f-8029-4255-980e-1132dd147c0a") : secret "cluster-monitoring-operator-tls" not found
Sep 11 18:49:22.364013 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:22.363995    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d0880abe-ec36-48e4-a511-ec663873fffa-serving-cert\") pod \"console-operator-696884fb9c-hsbsf\" (UID: \"d0880abe-ec36-48e4-a511-ec663873fffa\") " pod="openshift-console-operator/console-operator-696884fb9c-hsbsf"
Sep 11 18:49:22.372395 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:22.372372    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cxm7\" (UniqueName: \"kubernetes.io/projected/d0880abe-ec36-48e4-a511-ec663873fffa-kube-api-access-8cxm7\") pod \"console-operator-696884fb9c-hsbsf\" (UID: \"d0880abe-ec36-48e4-a511-ec663873fffa\") " pod="openshift-console-operator/console-operator-696884fb9c-hsbsf"
Sep 11 18:49:22.373262 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:22.373240    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm6zh\" (UniqueName: \"kubernetes.io/projected/2edba74f-8029-4255-980e-1132dd147c0a-kube-api-access-dm6zh\") pod \"cluster-monitoring-operator-6cd954b897-pvd8h\" (UID: \"2edba74f-8029-4255-980e-1132dd147c0a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-pvd8h"
Sep 11 18:49:22.432613 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:22.432565    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-hsbsf"
Sep 11 18:49:22.461148 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:22.461120    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/23b61d5d-8bb3-43c1-9ae8-38634ec8b885-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-nqtr7\" (UID: \"23b61d5d-8bb3-43c1-9ae8-38634ec8b885\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-nqtr7"
Sep 11 18:49:22.461276 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:22.461179    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/23b61d5d-8bb3-43c1-9ae8-38634ec8b885-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-nqtr7\" (UID: \"23b61d5d-8bb3-43c1-9ae8-38634ec8b885\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-nqtr7"
Sep 11 18:49:22.461276 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:22.461238    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/30878c31-acea-4b11-8991-992077a0ffca-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-jx2qj\" (UID: \"30878c31-acea-4b11-8991-992077a0ffca\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-jx2qj"
Sep 11 18:49:22.461276 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:22.461255    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dplwd\" (UniqueName: \"kubernetes.io/projected/30878c31-acea-4b11-8991-992077a0ffca-kube-api-access-dplwd\") pod \"cluster-samples-operator-86499c89c4-jx2qj\" (UID: \"30878c31-acea-4b11-8991-992077a0ffca\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-jx2qj"
Sep 11 18:49:22.461276 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:22.461257    2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 11 18:49:22.461484 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:22.461320    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23b61d5d-8bb3-43c1-9ae8-38634ec8b885-networking-console-plugin-cert podName:23b61d5d-8bb3-43c1-9ae8-38634ec8b885 nodeName:}" failed. No retries permitted until 2026-09-11 18:49:22.961302133 +0000 UTC m=+39.038892669 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/23b61d5d-8bb3-43c1-9ae8-38634ec8b885-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-nqtr7" (UID: "23b61d5d-8bb3-43c1-9ae8-38634ec8b885") : secret "networking-console-plugin-cert" not found
Sep 11 18:49:22.461484 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:22.461363    2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 11 18:49:22.461484 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:22.461415    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30878c31-acea-4b11-8991-992077a0ffca-samples-operator-tls podName:30878c31-acea-4b11-8991-992077a0ffca nodeName:}" failed. No retries permitted until 2026-09-11 18:49:22.961403054 +0000 UTC m=+39.038993588 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/30878c31-acea-4b11-8991-992077a0ffca-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-jx2qj" (UID: "30878c31-acea-4b11-8991-992077a0ffca") : secret "samples-operator-tls" not found
Sep 11 18:49:22.461931 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:22.461909    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/23b61d5d-8bb3-43c1-9ae8-38634ec8b885-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-nqtr7\" (UID: \"23b61d5d-8bb3-43c1-9ae8-38634ec8b885\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-nqtr7"
Sep 11 18:49:22.473964 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:22.473941    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dplwd\" (UniqueName: \"kubernetes.io/projected/30878c31-acea-4b11-8991-992077a0ffca-kube-api-access-dplwd\") pod \"cluster-samples-operator-86499c89c4-jx2qj\" (UID: \"30878c31-acea-4b11-8991-992077a0ffca\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-jx2qj"
Sep 11 18:49:22.595165 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:22.595131    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dc7p2" event={"ID":"c088609c-51c3-4070-88eb-a329ef4204a7","Type":"ContainerStarted","Data":"80101622bcea01d47be631271a63b88b229c16e05188ac0277b3442843c1c706"}
Sep 11 18:49:22.618265 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:22.618220    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-dc7p2" podStartSLOduration=9.806826363 podStartE2EDuration="38.618204458s" totalImagesPullingTime="28.811378095s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:48:44 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:49:22.617376765 +0000 UTC m=+38.694967330" watchObservedRunningTime="2026-09-11 18:49:22.618204458 +0000 UTC m=+38.695795012"
Sep 11 18:49:22.774378 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:22.774319    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-zdvjh"]
Sep 11 18:49:22.776635 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:22.776617    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-hsbsf"]
Sep 11 18:49:22.778134 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:49:22.778030    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc00f816e_59b2_4fa3_a666_ac15e94fd7d3.slice/crio-b41a97e9afd234b0236cd1fa2c6b1b0148a3a0d27fcb0fc313ac28d3539b04f2 WatchSource:0}: Error finding container b41a97e9afd234b0236cd1fa2c6b1b0148a3a0d27fcb0fc313ac28d3539b04f2: Status 404 returned error can't find the container with id b41a97e9afd234b0236cd1fa2c6b1b0148a3a0d27fcb0fc313ac28d3539b04f2
Sep 11 18:49:22.780698 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:49:22.780680    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0880abe_ec36_48e4_a511_ec663873fffa.slice/crio-9a2580116c0b11b1d5c95b36f316376ad98b9d85dbccf2c7c489f070c8fad34d WatchSource:0}: Error finding container 9a2580116c0b11b1d5c95b36f316376ad98b9d85dbccf2c7c489f070c8fad34d: Status 404 returned error can't find the container with id 9a2580116c0b11b1d5c95b36f316376ad98b9d85dbccf2c7c489f070c8fad34d
Sep 11 18:49:22.864329 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:22.864291    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2edba74f-8029-4255-980e-1132dd147c0a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-pvd8h\" (UID: \"2edba74f-8029-4255-980e-1132dd147c0a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-pvd8h"
Sep 11 18:49:22.864495 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:22.864473    2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 11 18:49:22.864572 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:22.864560    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2edba74f-8029-4255-980e-1132dd147c0a-cluster-monitoring-operator-tls podName:2edba74f-8029-4255-980e-1132dd147c0a nodeName:}" failed. No retries permitted until 2026-09-11 18:49:23.864539252 +0000 UTC m=+39.942129799 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2edba74f-8029-4255-980e-1132dd147c0a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-pvd8h" (UID: "2edba74f-8029-4255-980e-1132dd147c0a") : secret "cluster-monitoring-operator-tls" not found
Sep 11 18:49:22.964657 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:22.964619    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/23b61d5d-8bb3-43c1-9ae8-38634ec8b885-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-nqtr7\" (UID: \"23b61d5d-8bb3-43c1-9ae8-38634ec8b885\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-nqtr7"
Sep 11 18:49:22.964818 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:22.964737    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/30878c31-acea-4b11-8991-992077a0ffca-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-jx2qj\" (UID: \"30878c31-acea-4b11-8991-992077a0ffca\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-jx2qj"
Sep 11 18:49:22.964818 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:22.964785    2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 11 18:49:22.964886 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:22.964840    2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 11 18:49:22.964915 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:22.964897    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30878c31-acea-4b11-8991-992077a0ffca-samples-operator-tls podName:30878c31-acea-4b11-8991-992077a0ffca nodeName:}" failed. No retries permitted until 2026-09-11 18:49:23.964880425 +0000 UTC m=+40.042471253 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/30878c31-acea-4b11-8991-992077a0ffca-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-jx2qj" (UID: "30878c31-acea-4b11-8991-992077a0ffca") : secret "samples-operator-tls" not found
Sep 11 18:49:22.964915 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:22.964911    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23b61d5d-8bb3-43c1-9ae8-38634ec8b885-networking-console-plugin-cert podName:23b61d5d-8bb3-43c1-9ae8-38634ec8b885 nodeName:}" failed. No retries permitted until 2026-09-11 18:49:23.96490461 +0000 UTC m=+40.042495143 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/23b61d5d-8bb3-43c1-9ae8-38634ec8b885-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-nqtr7" (UID: "23b61d5d-8bb3-43c1-9ae8-38634ec8b885") : secret "networking-console-plugin-cert" not found
Sep 11 18:49:23.569467 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:23.569431    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9a3b9624-a629-437f-a700-3f24e40fc0ae-metrics-tls\") pod \"dns-default-9tzdn\" (UID: \"9a3b9624-a629-437f-a700-3f24e40fc0ae\") " pod="openshift-dns/dns-default-9tzdn"
Sep 11 18:49:23.569656 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:23.569611    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 18:49:23.569699 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:23.569689    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a3b9624-a629-437f-a700-3f24e40fc0ae-metrics-tls podName:9a3b9624-a629-437f-a700-3f24e40fc0ae nodeName:}" failed. No retries permitted until 2026-09-11 18:49:27.569667747 +0000 UTC m=+43.647258280 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9a3b9624-a629-437f-a700-3f24e40fc0ae-metrics-tls") pod "dns-default-9tzdn" (UID: "9a3b9624-a629-437f-a700-3f24e40fc0ae") : secret "dns-default-metrics-tls" not found
Sep 11 18:49:23.599480 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:23.599448    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bw8xb" event={"ID":"0e108d53-e1ef-4459-9eee-2e4be2ccf6e3","Type":"ContainerStarted","Data":"be78e6c32de79a30cfed511dba6ea0cbe5b59f74ad82164599ff3f6c71b80ebf"}
Sep 11 18:49:23.601230 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:23.601206    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-zdvjh" event={"ID":"c00f816e-59b2-4fa3-a666-ac15e94fd7d3","Type":"ContainerStarted","Data":"b41a97e9afd234b0236cd1fa2c6b1b0148a3a0d27fcb0fc313ac28d3539b04f2"}
Sep 11 18:49:23.602501 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:23.602468    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-hsbsf" event={"ID":"d0880abe-ec36-48e4-a511-ec663873fffa","Type":"ContainerStarted","Data":"9a2580116c0b11b1d5c95b36f316376ad98b9d85dbccf2c7c489f070c8fad34d"}
Sep 11 18:49:23.619214 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:23.619156    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-bw8xb" podStartSLOduration=2.358431606 podStartE2EDuration="4.619141278s" totalImagesPullingTime="2.260709672s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:49:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:49:23.618539235 +0000 UTC m=+39.696129789" watchObservedRunningTime="2026-09-11 18:49:23.619141278 +0000 UTC m=+39.696731833"
Sep 11 18:49:23.873791 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:23.873561    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2edba74f-8029-4255-980e-1132dd147c0a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-pvd8h\" (UID: \"2edba74f-8029-4255-980e-1132dd147c0a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-pvd8h"
Sep 11 18:49:23.873917 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:23.873889    2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 11 18:49:23.873959 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:23.873951    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2edba74f-8029-4255-980e-1132dd147c0a-cluster-monitoring-operator-tls podName:2edba74f-8029-4255-980e-1132dd147c0a nodeName:}" failed. No retries permitted until 2026-09-11 18:49:25.873934889 +0000 UTC m=+41.951525422 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2edba74f-8029-4255-980e-1132dd147c0a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-pvd8h" (UID: "2edba74f-8029-4255-980e-1132dd147c0a") : secret "cluster-monitoring-operator-tls" not found
Sep 11 18:49:23.974673 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:23.974631    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/30878c31-acea-4b11-8991-992077a0ffca-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-jx2qj\" (UID: \"30878c31-acea-4b11-8991-992077a0ffca\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-jx2qj"
Sep 11 18:49:23.974833 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:23.974682    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/23b61d5d-8bb3-43c1-9ae8-38634ec8b885-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-nqtr7\" (UID: \"23b61d5d-8bb3-43c1-9ae8-38634ec8b885\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-nqtr7"
Sep 11 18:49:23.974833 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:23.974792    2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 11 18:49:23.974923 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:23.974832    2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 11 18:49:23.974923 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:23.974861    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30878c31-acea-4b11-8991-992077a0ffca-samples-operator-tls podName:30878c31-acea-4b11-8991-992077a0ffca nodeName:}" failed. No retries permitted until 2026-09-11 18:49:25.97484536 +0000 UTC m=+42.052435893 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/30878c31-acea-4b11-8991-992077a0ffca-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-jx2qj" (UID: "30878c31-acea-4b11-8991-992077a0ffca") : secret "samples-operator-tls" not found
Sep 11 18:49:23.974923 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:23.974879    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23b61d5d-8bb3-43c1-9ae8-38634ec8b885-networking-console-plugin-cert podName:23b61d5d-8bb3-43c1-9ae8-38634ec8b885 nodeName:}" failed. No retries permitted until 2026-09-11 18:49:25.974866522 +0000 UTC m=+42.052457055 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/23b61d5d-8bb3-43c1-9ae8-38634ec8b885-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-nqtr7" (UID: "23b61d5d-8bb3-43c1-9ae8-38634ec8b885") : secret "networking-console-plugin-cert" not found
Sep 11 18:49:25.607943 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:25.607904    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-zdvjh" event={"ID":"c00f816e-59b2-4fa3-a666-ac15e94fd7d3","Type":"ContainerStarted","Data":"52c7b381d6870b9d67f8dd86940920d7da52694b0f0ecf624250865e34d8f802"}
Sep 11 18:49:25.609528 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:25.609507    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-hsbsf_d0880abe-ec36-48e4-a511-ec663873fffa/console-operator/0.log"
Sep 11 18:49:25.609642 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:25.609541    2578 generic.go:356] "Generic (PLEG): container finished" podID="d0880abe-ec36-48e4-a511-ec663873fffa" containerID="c8c6d7907e2abba80e0c083370c3865cfbd651e1af8d95383a511b560bc008ed" exitCode=255
Sep 11 18:49:25.609642 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:25.609570    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-hsbsf" event={"ID":"d0880abe-ec36-48e4-a511-ec663873fffa","Type":"ContainerDied","Data":"c8c6d7907e2abba80e0c083370c3865cfbd651e1af8d95383a511b560bc008ed"}
Sep 11 18:49:25.609784 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:25.609770    2578 scope.go:117] "RemoveContainer" containerID="c8c6d7907e2abba80e0c083370c3865cfbd651e1af8d95383a511b560bc008ed"
Sep 11 18:49:25.642288 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:25.642248    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-zdvjh" podStartSLOduration=2.407673077 podStartE2EDuration="4.642232501s" totalImagesPullingTime="2.234559424s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:49:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:49:25.625175563 +0000 UTC m=+41.702766119" watchObservedRunningTime="2026-09-11 18:49:25.642232501 +0000 UTC m=+41.719823055"
Sep 11 18:49:25.888336 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:25.888259    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2edba74f-8029-4255-980e-1132dd147c0a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-pvd8h\" (UID: \"2edba74f-8029-4255-980e-1132dd147c0a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-pvd8h"
Sep 11 18:49:25.888472 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:25.888397    2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 11 18:49:25.888472 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:25.888464    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2edba74f-8029-4255-980e-1132dd147c0a-cluster-monitoring-operator-tls podName:2edba74f-8029-4255-980e-1132dd147c0a nodeName:}" failed. No retries permitted until 2026-09-11 18:49:29.888447164 +0000 UTC m=+45.966037696 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2edba74f-8029-4255-980e-1132dd147c0a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-pvd8h" (UID: "2edba74f-8029-4255-980e-1132dd147c0a") : secret "cluster-monitoring-operator-tls" not found
Sep 11 18:49:25.988720 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:25.988683    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/30878c31-acea-4b11-8991-992077a0ffca-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-jx2qj\" (UID: \"30878c31-acea-4b11-8991-992077a0ffca\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-jx2qj"
Sep 11 18:49:25.988866 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:25.988729    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/23b61d5d-8bb3-43c1-9ae8-38634ec8b885-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-nqtr7\" (UID: \"23b61d5d-8bb3-43c1-9ae8-38634ec8b885\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-nqtr7"
Sep 11 18:49:25.988866 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:25.988817    2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 11 18:49:25.988866 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:25.988846    2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 11 18:49:25.988999 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:25.988879    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30878c31-acea-4b11-8991-992077a0ffca-samples-operator-tls podName:30878c31-acea-4b11-8991-992077a0ffca nodeName:}" failed. No retries permitted until 2026-09-11 18:49:29.988863268 +0000 UTC m=+46.066453801 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/30878c31-acea-4b11-8991-992077a0ffca-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-jx2qj" (UID: "30878c31-acea-4b11-8991-992077a0ffca") : secret "samples-operator-tls" not found
Sep 11 18:49:25.988999 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:25.988902    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23b61d5d-8bb3-43c1-9ae8-38634ec8b885-networking-console-plugin-cert podName:23b61d5d-8bb3-43c1-9ae8-38634ec8b885 nodeName:}" failed. No retries permitted until 2026-09-11 18:49:29.988886524 +0000 UTC m=+46.066477074 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/23b61d5d-8bb3-43c1-9ae8-38634ec8b885-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-nqtr7" (UID: "23b61d5d-8bb3-43c1-9ae8-38634ec8b885") : secret "networking-console-plugin-cert" not found
Sep 11 18:49:26.613280 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:26.613254    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-hsbsf_d0880abe-ec36-48e4-a511-ec663873fffa/console-operator/1.log"
Sep 11 18:49:26.613699 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:26.613659    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-hsbsf_d0880abe-ec36-48e4-a511-ec663873fffa/console-operator/0.log"
Sep 11 18:49:26.613699 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:26.613691    2578 generic.go:356] "Generic (PLEG): container finished" podID="d0880abe-ec36-48e4-a511-ec663873fffa" containerID="2a494d0d3a48f6fdf07596d7c6d5643a31af8756cf4f6d816f1442dfc30a0f6d" exitCode=255
Sep 11 18:49:26.613812 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:26.613788    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-hsbsf" event={"ID":"d0880abe-ec36-48e4-a511-ec663873fffa","Type":"ContainerDied","Data":"2a494d0d3a48f6fdf07596d7c6d5643a31af8756cf4f6d816f1442dfc30a0f6d"}
Sep 11 18:49:26.613853 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:26.613845    2578 scope.go:117] "RemoveContainer" containerID="c8c6d7907e2abba80e0c083370c3865cfbd651e1af8d95383a511b560bc008ed"
Sep 11 18:49:26.613982 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:26.613972    2578 scope.go:117] "RemoveContainer" containerID="2a494d0d3a48f6fdf07596d7c6d5643a31af8756cf4f6d816f1442dfc30a0f6d"
Sep 11 18:49:26.614156 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:26.614138    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-hsbsf_openshift-console-operator(d0880abe-ec36-48e4-a511-ec663873fffa)\"" pod="openshift-console-operator/console-operator-696884fb9c-hsbsf" podUID="d0880abe-ec36-48e4-a511-ec663873fffa"
Sep 11 18:49:27.600332 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:27.600297    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9a3b9624-a629-437f-a700-3f24e40fc0ae-metrics-tls\") pod \"dns-default-9tzdn\" (UID: \"9a3b9624-a629-437f-a700-3f24e40fc0ae\") " pod="openshift-dns/dns-default-9tzdn"
Sep 11 18:49:27.600488 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:27.600406    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 18:49:27.600488 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:27.600459    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a3b9624-a629-437f-a700-3f24e40fc0ae-metrics-tls podName:9a3b9624-a629-437f-a700-3f24e40fc0ae nodeName:}" failed. No retries permitted until 2026-09-11 18:49:35.600446005 +0000 UTC m=+51.678036539 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9a3b9624-a629-437f-a700-3f24e40fc0ae-metrics-tls") pod "dns-default-9tzdn" (UID: "9a3b9624-a629-437f-a700-3f24e40fc0ae") : secret "dns-default-metrics-tls" not found
Sep 11 18:49:27.617386 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:27.617362    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-hsbsf_d0880abe-ec36-48e4-a511-ec663873fffa/console-operator/1.log"
Sep 11 18:49:27.617783 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:27.617656    2578 scope.go:117] "RemoveContainer" containerID="2a494d0d3a48f6fdf07596d7c6d5643a31af8756cf4f6d816f1442dfc30a0f6d"
Sep 11 18:49:27.617832 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:27.617783    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-hsbsf_openshift-console-operator(d0880abe-ec36-48e4-a511-ec663873fffa)\"" pod="openshift-console-operator/console-operator-696884fb9c-hsbsf" podUID="d0880abe-ec36-48e4-a511-ec663873fffa"
Sep 11 18:49:29.918246 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:29.918208    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2edba74f-8029-4255-980e-1132dd147c0a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-pvd8h\" (UID: \"2edba74f-8029-4255-980e-1132dd147c0a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-pvd8h"
Sep 11 18:49:29.918747 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:29.918390    2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 11 18:49:29.918747 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:29.918456    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2edba74f-8029-4255-980e-1132dd147c0a-cluster-monitoring-operator-tls podName:2edba74f-8029-4255-980e-1132dd147c0a nodeName:}" failed. No retries permitted until 2026-09-11 18:49:37.918441633 +0000 UTC m=+53.996032166 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2edba74f-8029-4255-980e-1132dd147c0a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-pvd8h" (UID: "2edba74f-8029-4255-980e-1132dd147c0a") : secret "cluster-monitoring-operator-tls" not found
Sep 11 18:49:30.019051 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:30.019023    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/30878c31-acea-4b11-8991-992077a0ffca-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-jx2qj\" (UID: \"30878c31-acea-4b11-8991-992077a0ffca\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-jx2qj"
Sep 11 18:49:30.019160 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:30.019061    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/23b61d5d-8bb3-43c1-9ae8-38634ec8b885-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-nqtr7\" (UID: \"23b61d5d-8bb3-43c1-9ae8-38634ec8b885\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-nqtr7"
Sep 11 18:49:30.019199 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:30.019178    2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 11 18:49:30.019199 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:30.019178    2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 11 18:49:30.019262 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:30.019240    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23b61d5d-8bb3-43c1-9ae8-38634ec8b885-networking-console-plugin-cert podName:23b61d5d-8bb3-43c1-9ae8-38634ec8b885 nodeName:}" failed. No retries permitted until 2026-09-11 18:49:38.019223325 +0000 UTC m=+54.096813871 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/23b61d5d-8bb3-43c1-9ae8-38634ec8b885-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-nqtr7" (UID: "23b61d5d-8bb3-43c1-9ae8-38634ec8b885") : secret "networking-console-plugin-cert" not found
Sep 11 18:49:30.019262 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:30.019257    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30878c31-acea-4b11-8991-992077a0ffca-samples-operator-tls podName:30878c31-acea-4b11-8991-992077a0ffca nodeName:}" failed. No retries permitted until 2026-09-11 18:49:38.019248174 +0000 UTC m=+54.096838707 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/30878c31-acea-4b11-8991-992077a0ffca-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-jx2qj" (UID: "30878c31-acea-4b11-8991-992077a0ffca") : secret "samples-operator-tls" not found
Sep 11 18:49:30.048925 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:30.048895    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-mtksz"]
Sep 11 18:49:30.076265 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:30.076238    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mtksz"]
Sep 11 18:49:30.076392 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:30.076356    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mtksz"
Sep 11 18:49:30.080584 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:30.080515    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 11 18:49:30.080584 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:30.080580    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 11 18:49:30.080803 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:30.080644    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 11 18:49:30.084470 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:30.084446    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 11 18:49:30.084927 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:30.084903    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-6nt5d\""
Sep 11 18:49:30.220752 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:30.220727    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcbzt\" (UniqueName: \"kubernetes.io/projected/1e6ec1bf-e524-430b-950b-6ef0039b353b-kube-api-access-qcbzt\") pod \"insights-runtime-extractor-mtksz\" (UID: \"1e6ec1bf-e524-430b-950b-6ef0039b353b\") " pod="openshift-insights/insights-runtime-extractor-mtksz"
Sep 11 18:49:30.220893 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:30.220781    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1e6ec1bf-e524-430b-950b-6ef0039b353b-data-volume\") pod \"insights-runtime-extractor-mtksz\" (UID: \"1e6ec1bf-e524-430b-950b-6ef0039b353b\") " pod="openshift-insights/insights-runtime-extractor-mtksz"
Sep 11 18:49:30.220893 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:30.220838    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1e6ec1bf-e524-430b-950b-6ef0039b353b-crio-socket\") pod \"insights-runtime-extractor-mtksz\" (UID: \"1e6ec1bf-e524-430b-950b-6ef0039b353b\") " pod="openshift-insights/insights-runtime-extractor-mtksz"
Sep 11 18:49:30.220893 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:30.220883    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1e6ec1bf-e524-430b-950b-6ef0039b353b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mtksz\" (UID: \"1e6ec1bf-e524-430b-950b-6ef0039b353b\") " pod="openshift-insights/insights-runtime-extractor-mtksz"
Sep 11 18:49:30.221001 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:30.220947    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1e6ec1bf-e524-430b-950b-6ef0039b353b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mtksz\" (UID: \"1e6ec1bf-e524-430b-950b-6ef0039b353b\") " pod="openshift-insights/insights-runtime-extractor-mtksz"
Sep 11 18:49:30.321741 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:30.321713    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1e6ec1bf-e524-430b-950b-6ef0039b353b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mtksz\" (UID: \"1e6ec1bf-e524-430b-950b-6ef0039b353b\") " pod="openshift-insights/insights-runtime-extractor-mtksz"
Sep 11 18:49:30.321866 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:30.321777    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1e6ec1bf-e524-430b-950b-6ef0039b353b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mtksz\" (UID: \"1e6ec1bf-e524-430b-950b-6ef0039b353b\") " pod="openshift-insights/insights-runtime-extractor-mtksz"
Sep 11 18:49:30.321866 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:30.321809    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcbzt\" (UniqueName: \"kubernetes.io/projected/1e6ec1bf-e524-430b-950b-6ef0039b353b-kube-api-access-qcbzt\") pod \"insights-runtime-extractor-mtksz\" (UID: \"1e6ec1bf-e524-430b-950b-6ef0039b353b\") " pod="openshift-insights/insights-runtime-extractor-mtksz"
Sep 11 18:49:30.321866 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:30.321828    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 18:49:30.321866 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:30.321838    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1e6ec1bf-e524-430b-950b-6ef0039b353b-data-volume\") pod \"insights-runtime-extractor-mtksz\" (UID: \"1e6ec1bf-e524-430b-950b-6ef0039b353b\") " pod="openshift-insights/insights-runtime-extractor-mtksz"
Sep 11 18:49:30.321866 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:30.321859    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1e6ec1bf-e524-430b-950b-6ef0039b353b-crio-socket\") pod \"insights-runtime-extractor-mtksz\" (UID: \"1e6ec1bf-e524-430b-950b-6ef0039b353b\") " pod="openshift-insights/insights-runtime-extractor-mtksz"
Sep 11 18:49:30.322056 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:30.321917    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1e6ec1bf-e524-430b-950b-6ef0039b353b-kube-rbac-proxy-cm podName:1e6ec1bf-e524-430b-950b-6ef0039b353b nodeName:}" failed. No retries permitted until 2026-09-11 18:49:30.821897994 +0000 UTC m=+46.899488532 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1e6ec1bf-e524-430b-950b-6ef0039b353b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mtksz" (UID: "1e6ec1bf-e524-430b-950b-6ef0039b353b") : configmap "kube-rbac-proxy" not found
Sep 11 18:49:30.322056 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:30.322027    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1e6ec1bf-e524-430b-950b-6ef0039b353b-crio-socket\") pod \"insights-runtime-extractor-mtksz\" (UID: \"1e6ec1bf-e524-430b-950b-6ef0039b353b\") " pod="openshift-insights/insights-runtime-extractor-mtksz"
Sep 11 18:49:30.322056 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:30.322039    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 18:49:30.322152 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:30.322111    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e6ec1bf-e524-430b-950b-6ef0039b353b-insights-runtime-extractor-tls podName:1e6ec1bf-e524-430b-950b-6ef0039b353b nodeName:}" failed. No retries permitted until 2026-09-11 18:49:30.822093569 +0000 UTC m=+46.899684104 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1e6ec1bf-e524-430b-950b-6ef0039b353b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mtksz" (UID: "1e6ec1bf-e524-430b-950b-6ef0039b353b") : secret "insights-runtime-extractor-tls" not found
Sep 11 18:49:30.322192 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:30.322156    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1e6ec1bf-e524-430b-950b-6ef0039b353b-data-volume\") pod \"insights-runtime-extractor-mtksz\" (UID: \"1e6ec1bf-e524-430b-950b-6ef0039b353b\") " pod="openshift-insights/insights-runtime-extractor-mtksz"
Sep 11 18:49:30.336548 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:30.336527    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcbzt\" (UniqueName: \"kubernetes.io/projected/1e6ec1bf-e524-430b-950b-6ef0039b353b-kube-api-access-qcbzt\") pod \"insights-runtime-extractor-mtksz\" (UID: \"1e6ec1bf-e524-430b-950b-6ef0039b353b\") " pod="openshift-insights/insights-runtime-extractor-mtksz"
Sep 11 18:49:30.825753 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:30.825722    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1e6ec1bf-e524-430b-950b-6ef0039b353b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mtksz\" (UID: \"1e6ec1bf-e524-430b-950b-6ef0039b353b\") " pod="openshift-insights/insights-runtime-extractor-mtksz"
Sep 11 18:49:30.825920 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:30.825801    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1e6ec1bf-e524-430b-950b-6ef0039b353b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mtksz\" (UID: \"1e6ec1bf-e524-430b-950b-6ef0039b353b\") " pod="openshift-insights/insights-runtime-extractor-mtksz"
Sep 11 18:49:30.825920 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:30.825873    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 18:49:30.825920 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:30.825886    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 18:49:30.826022 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:30.825936    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1e6ec1bf-e524-430b-950b-6ef0039b353b-kube-rbac-proxy-cm podName:1e6ec1bf-e524-430b-950b-6ef0039b353b nodeName:}" failed. No retries permitted until 2026-09-11 18:49:31.825922247 +0000 UTC m=+47.903512780 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1e6ec1bf-e524-430b-950b-6ef0039b353b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mtksz" (UID: "1e6ec1bf-e524-430b-950b-6ef0039b353b") : configmap "kube-rbac-proxy" not found
Sep 11 18:49:30.826022 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:30.825947    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e6ec1bf-e524-430b-950b-6ef0039b353b-insights-runtime-extractor-tls podName:1e6ec1bf-e524-430b-950b-6ef0039b353b nodeName:}" failed. No retries permitted until 2026-09-11 18:49:31.825941959 +0000 UTC m=+47.903532492 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1e6ec1bf-e524-430b-950b-6ef0039b353b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mtksz" (UID: "1e6ec1bf-e524-430b-950b-6ef0039b353b") : secret "insights-runtime-extractor-tls" not found
Sep 11 18:49:31.833270 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:31.833236    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1e6ec1bf-e524-430b-950b-6ef0039b353b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mtksz\" (UID: \"1e6ec1bf-e524-430b-950b-6ef0039b353b\") " pod="openshift-insights/insights-runtime-extractor-mtksz"
Sep 11 18:49:31.833698 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:31.833310    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1e6ec1bf-e524-430b-950b-6ef0039b353b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mtksz\" (UID: \"1e6ec1bf-e524-430b-950b-6ef0039b353b\") " pod="openshift-insights/insights-runtime-extractor-mtksz"
Sep 11 18:49:31.833698 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:31.833366    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 18:49:31.833698 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:31.833420    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 18:49:31.833698 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:31.833436    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1e6ec1bf-e524-430b-950b-6ef0039b353b-kube-rbac-proxy-cm podName:1e6ec1bf-e524-430b-950b-6ef0039b353b nodeName:}" failed. No retries permitted until 2026-09-11 18:49:33.833420408 +0000 UTC m=+49.911010941 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1e6ec1bf-e524-430b-950b-6ef0039b353b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mtksz" (UID: "1e6ec1bf-e524-430b-950b-6ef0039b353b") : configmap "kube-rbac-proxy" not found
Sep 11 18:49:31.833698 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:31.833458    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e6ec1bf-e524-430b-950b-6ef0039b353b-insights-runtime-extractor-tls podName:1e6ec1bf-e524-430b-950b-6ef0039b353b nodeName:}" failed. No retries permitted until 2026-09-11 18:49:33.833446796 +0000 UTC m=+49.911037330 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1e6ec1bf-e524-430b-950b-6ef0039b353b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mtksz" (UID: "1e6ec1bf-e524-430b-950b-6ef0039b353b") : secret "insights-runtime-extractor-tls" not found
Sep 11 18:49:32.433203 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:32.433171    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-hsbsf"
Sep 11 18:49:32.433364 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:32.433217    2578 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-hsbsf"
Sep 11 18:49:32.433528 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:32.433516    2578 scope.go:117] "RemoveContainer" containerID="2a494d0d3a48f6fdf07596d7c6d5643a31af8756cf4f6d816f1442dfc30a0f6d"
Sep 11 18:49:32.433688 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:32.433672    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-hsbsf_openshift-console-operator(d0880abe-ec36-48e4-a511-ec663873fffa)\"" pod="openshift-console-operator/console-operator-696884fb9c-hsbsf" podUID="d0880abe-ec36-48e4-a511-ec663873fffa"
Sep 11 18:49:32.610589 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:32.610564    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dnzn9_43683762-3da6-4f69-a5ad-d685d63bbe4f/dns-node-resolver/0.log"
Sep 11 18:49:33.848208 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:33.848167    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1e6ec1bf-e524-430b-950b-6ef0039b353b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mtksz\" (UID: \"1e6ec1bf-e524-430b-950b-6ef0039b353b\") " pod="openshift-insights/insights-runtime-extractor-mtksz"
Sep 11 18:49:33.848690 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:33.848282    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1e6ec1bf-e524-430b-950b-6ef0039b353b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mtksz\" (UID: \"1e6ec1bf-e524-430b-950b-6ef0039b353b\") " pod="openshift-insights/insights-runtime-extractor-mtksz"
Sep 11 18:49:33.848690 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:33.848325    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 18:49:33.848690 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:33.848382    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 18:49:33.848690 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:33.848412    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e6ec1bf-e524-430b-950b-6ef0039b353b-insights-runtime-extractor-tls podName:1e6ec1bf-e524-430b-950b-6ef0039b353b nodeName:}" failed. No retries permitted until 2026-09-11 18:49:37.848395975 +0000 UTC m=+53.925986511 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1e6ec1bf-e524-430b-950b-6ef0039b353b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mtksz" (UID: "1e6ec1bf-e524-430b-950b-6ef0039b353b") : secret "insights-runtime-extractor-tls" not found
Sep 11 18:49:33.848690 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:33.848436    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1e6ec1bf-e524-430b-950b-6ef0039b353b-kube-rbac-proxy-cm podName:1e6ec1bf-e524-430b-950b-6ef0039b353b nodeName:}" failed. No retries permitted until 2026-09-11 18:49:37.848419528 +0000 UTC m=+53.926010082 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1e6ec1bf-e524-430b-950b-6ef0039b353b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mtksz" (UID: "1e6ec1bf-e524-430b-950b-6ef0039b353b") : configmap "kube-rbac-proxy" not found
Sep 11 18:49:34.002190 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:34.002165    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-wmk8r_107db790-53d5-4969-8bf9-2ee3b9f9df09/node-ca/0.log"
Sep 11 18:49:34.417257 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:34.417219    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bw8xb_0e108d53-e1ef-4459-9eee-2e4be2ccf6e3/serve-healthcheck-canary/0.log"
Sep 11 18:49:35.665101 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:35.665069    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9a3b9624-a629-437f-a700-3f24e40fc0ae-metrics-tls\") pod \"dns-default-9tzdn\" (UID: \"9a3b9624-a629-437f-a700-3f24e40fc0ae\") " pod="openshift-dns/dns-default-9tzdn"
Sep 11 18:49:35.665502 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:35.665213    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 18:49:35.665502 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:35.665270    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a3b9624-a629-437f-a700-3f24e40fc0ae-metrics-tls podName:9a3b9624-a629-437f-a700-3f24e40fc0ae nodeName:}" failed. No retries permitted until 2026-09-11 18:49:51.665256025 +0000 UTC m=+67.742846558 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9a3b9624-a629-437f-a700-3f24e40fc0ae-metrics-tls") pod "dns-default-9tzdn" (UID: "9a3b9624-a629-437f-a700-3f24e40fc0ae") : secret "dns-default-metrics-tls" not found
Sep 11 18:49:37.882015 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:37.881974    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1e6ec1bf-e524-430b-950b-6ef0039b353b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mtksz\" (UID: \"1e6ec1bf-e524-430b-950b-6ef0039b353b\") " pod="openshift-insights/insights-runtime-extractor-mtksz"
Sep 11 18:49:37.882449 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:37.882072    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1e6ec1bf-e524-430b-950b-6ef0039b353b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mtksz\" (UID: \"1e6ec1bf-e524-430b-950b-6ef0039b353b\") " pod="openshift-insights/insights-runtime-extractor-mtksz"
Sep 11 18:49:37.882449 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:37.882101    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 18:49:37.882449 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:37.882172    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1e6ec1bf-e524-430b-950b-6ef0039b353b-kube-rbac-proxy-cm podName:1e6ec1bf-e524-430b-950b-6ef0039b353b nodeName:}" failed. No retries permitted until 2026-09-11 18:49:45.882154011 +0000 UTC m=+61.959744564 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1e6ec1bf-e524-430b-950b-6ef0039b353b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mtksz" (UID: "1e6ec1bf-e524-430b-950b-6ef0039b353b") : configmap "kube-rbac-proxy" not found
Sep 11 18:49:37.882449 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:37.882193    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 18:49:37.882449 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:37.882229    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e6ec1bf-e524-430b-950b-6ef0039b353b-insights-runtime-extractor-tls podName:1e6ec1bf-e524-430b-950b-6ef0039b353b nodeName:}" failed. No retries permitted until 2026-09-11 18:49:45.882218529 +0000 UTC m=+61.959809076 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1e6ec1bf-e524-430b-950b-6ef0039b353b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mtksz" (UID: "1e6ec1bf-e524-430b-950b-6ef0039b353b") : secret "insights-runtime-extractor-tls" not found
Sep 11 18:49:37.983059 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:37.983028    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2edba74f-8029-4255-980e-1132dd147c0a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-pvd8h\" (UID: \"2edba74f-8029-4255-980e-1132dd147c0a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-pvd8h"
Sep 11 18:49:37.983220 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:37.983130    2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 11 18:49:37.983220 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:37.983192    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2edba74f-8029-4255-980e-1132dd147c0a-cluster-monitoring-operator-tls podName:2edba74f-8029-4255-980e-1132dd147c0a nodeName:}" failed. No retries permitted until 2026-09-11 18:49:53.98317774 +0000 UTC m=+70.060768273 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2edba74f-8029-4255-980e-1132dd147c0a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-pvd8h" (UID: "2edba74f-8029-4255-980e-1132dd147c0a") : secret "cluster-monitoring-operator-tls" not found
Sep 11 18:49:38.083405 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:38.083373    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/30878c31-acea-4b11-8991-992077a0ffca-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-jx2qj\" (UID: \"30878c31-acea-4b11-8991-992077a0ffca\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-jx2qj"
Sep 11 18:49:38.083551 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:38.083421    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/23b61d5d-8bb3-43c1-9ae8-38634ec8b885-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-nqtr7\" (UID: \"23b61d5d-8bb3-43c1-9ae8-38634ec8b885\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-nqtr7"
Sep 11 18:49:38.083648 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:38.083552    2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 11 18:49:38.083702 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:49:38.083654    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23b61d5d-8bb3-43c1-9ae8-38634ec8b885-networking-console-plugin-cert podName:23b61d5d-8bb3-43c1-9ae8-38634ec8b885 nodeName:}" failed. No retries permitted until 2026-09-11 18:49:54.083633579 +0000 UTC m=+70.161224125 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/23b61d5d-8bb3-43c1-9ae8-38634ec8b885-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-nqtr7" (UID: "23b61d5d-8bb3-43c1-9ae8-38634ec8b885") : secret "networking-console-plugin-cert" not found
Sep 11 18:49:38.085916 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:38.085893    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/30878c31-acea-4b11-8991-992077a0ffca-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-jx2qj\" (UID: \"30878c31-acea-4b11-8991-992077a0ffca\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-jx2qj"
Sep 11 18:49:38.124687 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:38.124654    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-jx2qj"
Sep 11 18:49:38.394687 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:38.394645    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-jx2qj"]
Sep 11 18:49:38.644360 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:38.644272    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-jx2qj" event={"ID":"30878c31-acea-4b11-8991-992077a0ffca","Type":"ContainerStarted","Data":"63ac697f917fcd3bebe259f3ef42364daca382aaf92d2bb88ce9f4e566d43697"}
Sep 11 18:49:40.652699 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:40.652614    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-jx2qj" event={"ID":"30878c31-acea-4b11-8991-992077a0ffca","Type":"ContainerStarted","Data":"a073a413cb805c38d51973f69306edfd0d76056a694c8d2ff662042e9321572e"}
Sep 11 18:49:40.652699 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:40.652666    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-jx2qj" event={"ID":"30878c31-acea-4b11-8991-992077a0ffca","Type":"ContainerStarted","Data":"03e7216a28c63fd038c57a336e91fa4daf3466beed3295112dff0a4739e133f5"}
Sep 11 18:49:40.671507 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:40.671464    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-jx2qj" podStartSLOduration=16.872763741 podStartE2EDuration="18.671449545s" totalImagesPullingTime="1.798685804s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:49:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:49:40.671368246 +0000 UTC m=+56.748958827" watchObservedRunningTime="2026-09-11 18:49:40.671449545 +0000 UTC m=+56.749040102"
Sep 11 18:49:43.562091 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:43.562065    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7xkpg"
Sep 11 18:49:45.390586 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:45.390557    2578 scope.go:117] "RemoveContainer" containerID="2a494d0d3a48f6fdf07596d7c6d5643a31af8756cf4f6d816f1442dfc30a0f6d"
Sep 11 18:49:45.667065 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:45.666984    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-hsbsf_d0880abe-ec36-48e4-a511-ec663873fffa/console-operator/1.log"
Sep 11 18:49:45.667065 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:45.667043    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-hsbsf" event={"ID":"d0880abe-ec36-48e4-a511-ec663873fffa","Type":"ContainerStarted","Data":"e9f17e9e49bc0c7bea4ea664ce3121742bffcf18c2e6b973ab4a43cd54029305"}
Sep 11 18:49:45.667335 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:45.667315    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-hsbsf"
Sep 11 18:49:45.686061 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:45.686018    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-696884fb9c-hsbsf" podStartSLOduration=21.449422834 podStartE2EDuration="23.686004661s" totalImagesPullingTime="2.236581827s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:49:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:49:45.685226664 +0000 UTC m=+61.762817239" watchObservedRunningTime="2026-09-11 18:49:45.686004661 +0000 UTC m=+61.763595226"
Sep 11 18:49:45.938422 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:45.938337    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1e6ec1bf-e524-430b-950b-6ef0039b353b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mtksz\" (UID: \"1e6ec1bf-e524-430b-950b-6ef0039b353b\") " pod="openshift-insights/insights-runtime-extractor-mtksz"
Sep 11 18:49:45.938422 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:45.938401    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1e6ec1bf-e524-430b-950b-6ef0039b353b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mtksz\" (UID: \"1e6ec1bf-e524-430b-950b-6ef0039b353b\") " pod="openshift-insights/insights-runtime-extractor-mtksz"
Sep 11 18:49:45.939010 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:45.938982    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1e6ec1bf-e524-430b-950b-6ef0039b353b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mtksz\" (UID: \"1e6ec1bf-e524-430b-950b-6ef0039b353b\") " pod="openshift-insights/insights-runtime-extractor-mtksz"
Sep 11 18:49:45.940734 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:45.940717    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1e6ec1bf-e524-430b-950b-6ef0039b353b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mtksz\" (UID: \"1e6ec1bf-e524-430b-950b-6ef0039b353b\") " pod="openshift-insights/insights-runtime-extractor-mtksz"
Sep 11 18:49:45.988819 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:45.988797    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-6nt5d\""
Sep 11 18:49:45.997454 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:45.997435    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mtksz"
Sep 11 18:49:46.197754 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:46.197670    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-696884fb9c-hsbsf"
Sep 11 18:49:46.278443 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:46.278414    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mtksz"]
Sep 11 18:49:46.281673 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:49:46.281648    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e6ec1bf_e524_430b_950b_6ef0039b353b.slice/crio-cbe09595b4d677e3c936501ce3d53e6d1be23b9338f20b5ad7419d9b7e27938c WatchSource:0}: Error finding container cbe09595b4d677e3c936501ce3d53e6d1be23b9338f20b5ad7419d9b7e27938c: Status 404 returned error can't find the container with id cbe09595b4d677e3c936501ce3d53e6d1be23b9338f20b5ad7419d9b7e27938c
Sep 11 18:49:46.672940 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:46.672829    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mtksz" event={"ID":"1e6ec1bf-e524-430b-950b-6ef0039b353b","Type":"ContainerStarted","Data":"79dae07beca5be4591faad2bf6c80d9c4e0a87e34d048225842463abc4198ccf"}
Sep 11 18:49:46.672940 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:46.672888    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mtksz" event={"ID":"1e6ec1bf-e524-430b-950b-6ef0039b353b","Type":"ContainerStarted","Data":"cbe09595b4d677e3c936501ce3d53e6d1be23b9338f20b5ad7419d9b7e27938c"}
Sep 11 18:49:47.676887 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:47.676847    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mtksz" event={"ID":"1e6ec1bf-e524-430b-950b-6ef0039b353b","Type":"ContainerStarted","Data":"b6f350ea2e276ebb960171a8c4044699e2da2ebfa41ee7d8314ab7761fdd126c"}
Sep 11 18:49:48.681989 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:48.681947    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mtksz" event={"ID":"1e6ec1bf-e524-430b-950b-6ef0039b353b","Type":"ContainerStarted","Data":"0f889048b52c1a40ba96f4c2a83621516f54fefeaaf5c17f55a55ea03061338e"}
Sep 11 18:49:48.705752 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:48.705709    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-mtksz" podStartSLOduration=17.040283088 podStartE2EDuration="18.705695831s" totalImagesPullingTime="1.665412743s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:49:30 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:49:48.705112454 +0000 UTC m=+64.782703028" watchObservedRunningTime="2026-09-11 18:49:48.705695831 +0000 UTC m=+64.783286427"
Sep 11 18:49:50.070421 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:50.070390    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8498e9d1-1eeb-4ce9-b225-d18452e49150-metrics-certs\") pod \"network-metrics-daemon-bjcnp\" (UID: \"8498e9d1-1eeb-4ce9-b225-d18452e49150\") " pod="openshift-multus/network-metrics-daemon-bjcnp"
Sep 11 18:49:50.073012 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:50.072993    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 18:49:50.083032 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:50.083010    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8498e9d1-1eeb-4ce9-b225-d18452e49150-metrics-certs\") pod \"network-metrics-daemon-bjcnp\" (UID: \"8498e9d1-1eeb-4ce9-b225-d18452e49150\") " pod="openshift-multus/network-metrics-daemon-bjcnp"
Sep 11 18:49:50.108510 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:50.108485    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-kzbqc\""
Sep 11 18:49:50.116624 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:50.116606    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bjcnp"
Sep 11 18:49:50.171022 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:50.170995    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8h4f\" (UniqueName: \"kubernetes.io/projected/c7382740-2bba-4368-8e7f-508c64d170a7-kube-api-access-f8h4f\") pod \"network-check-target-7c8ns\" (UID: \"c7382740-2bba-4368-8e7f-508c64d170a7\") " pod="openshift-network-diagnostics/network-check-target-7c8ns"
Sep 11 18:49:50.171114 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:50.171031    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/90acf5d2-d7df-46bb-a380-73f3ae079b75-original-pull-secret\") pod \"global-pull-secret-syncer-vnkmt\" (UID: \"90acf5d2-d7df-46bb-a380-73f3ae079b75\") " pod="kube-system/global-pull-secret-syncer-vnkmt"
Sep 11 18:49:50.173686 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:50.173661    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 18:49:50.173782 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:50.173691    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 18:49:50.183827 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:50.183804    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 18:49:50.184153 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:50.184135    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/90acf5d2-d7df-46bb-a380-73f3ae079b75-original-pull-secret\") pod \"global-pull-secret-syncer-vnkmt\" (UID: \"90acf5d2-d7df-46bb-a380-73f3ae079b75\") " pod="kube-system/global-pull-secret-syncer-vnkmt"
Sep 11 18:49:50.194778 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:50.194755    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8h4f\" (UniqueName: \"kubernetes.io/projected/c7382740-2bba-4368-8e7f-508c64d170a7-kube-api-access-f8h4f\") pod \"network-check-target-7c8ns\" (UID: \"c7382740-2bba-4368-8e7f-508c64d170a7\") " pod="openshift-network-diagnostics/network-check-target-7c8ns"
Sep 11 18:49:50.384987 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:50.384918    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bjcnp"]
Sep 11 18:49:50.388205 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:49:50.388176    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8498e9d1_1eeb_4ce9_b225_d18452e49150.slice/crio-3f7489fc6d22ccefb9ebc904e9c743464a726857e76128813707e51adf1446a6 WatchSource:0}: Error finding container 3f7489fc6d22ccefb9ebc904e9c743464a726857e76128813707e51adf1446a6: Status 404 returned error can't find the container with id 3f7489fc6d22ccefb9ebc904e9c743464a726857e76128813707e51adf1446a6
Sep 11 18:49:50.403191 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:50.403171    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-tr6q7\""
Sep 11 18:49:50.412095 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:50.412079    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7c8ns"
Sep 11 18:49:50.419610 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:50.419575    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnkmt"
Sep 11 18:49:50.692251 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:50.692175    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7c8ns"]
Sep 11 18:49:50.692832 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:50.692809    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bjcnp" event={"ID":"8498e9d1-1eeb-4ce9-b225-d18452e49150","Type":"ContainerStarted","Data":"3f7489fc6d22ccefb9ebc904e9c743464a726857e76128813707e51adf1446a6"}
Sep 11 18:49:50.694941 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:50.694915    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vnkmt"]
Sep 11 18:49:50.695282 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:49:50.695262    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7382740_2bba_4368_8e7f_508c64d170a7.slice/crio-df1c81f46c3394c77da1adf31c4aef32449681b3c84e51ff77f54a2468f58750 WatchSource:0}: Error finding container df1c81f46c3394c77da1adf31c4aef32449681b3c84e51ff77f54a2468f58750: Status 404 returned error can't find the container with id df1c81f46c3394c77da1adf31c4aef32449681b3c84e51ff77f54a2468f58750
Sep 11 18:49:50.697816 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:49:50.697798    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90acf5d2_d7df_46bb_a380_73f3ae079b75.slice/crio-328e905218589a092fbe0c93e31b12fe5708164d332d0724156295419e44e0bb WatchSource:0}: Error finding container 328e905218589a092fbe0c93e31b12fe5708164d332d0724156295419e44e0bb: Status 404 returned error can't find the container with id 328e905218589a092fbe0c93e31b12fe5708164d332d0724156295419e44e0bb
Sep 11 18:49:51.683196 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:51.682756    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9a3b9624-a629-437f-a700-3f24e40fc0ae-metrics-tls\") pod \"dns-default-9tzdn\" (UID: \"9a3b9624-a629-437f-a700-3f24e40fc0ae\") " pod="openshift-dns/dns-default-9tzdn"
Sep 11 18:49:51.685502 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:51.685477    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9a3b9624-a629-437f-a700-3f24e40fc0ae-metrics-tls\") pod \"dns-default-9tzdn\" (UID: \"9a3b9624-a629-437f-a700-3f24e40fc0ae\") " pod="openshift-dns/dns-default-9tzdn"
Sep 11 18:49:51.697695 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:51.697626    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vnkmt" event={"ID":"90acf5d2-d7df-46bb-a380-73f3ae079b75","Type":"ContainerStarted","Data":"328e905218589a092fbe0c93e31b12fe5708164d332d0724156295419e44e0bb"}
Sep 11 18:49:51.698905 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:51.698878    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7c8ns" event={"ID":"c7382740-2bba-4368-8e7f-508c64d170a7","Type":"ContainerStarted","Data":"df1c81f46c3394c77da1adf31c4aef32449681b3c84e51ff77f54a2468f58750"}
Sep 11 18:49:51.915772 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:51.915736    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vlxp2\""
Sep 11 18:49:51.923778 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:51.923750    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9tzdn"
Sep 11 18:49:52.280536 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:52.280494    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9tzdn"]
Sep 11 18:49:52.293962 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:49:52.293914    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a3b9624_a629_437f_a700_3f24e40fc0ae.slice/crio-e37cd1f5d76682e8d506dffeffc8b4a4a432e75c6cb15372c875b068adc6ee56 WatchSource:0}: Error finding container e37cd1f5d76682e8d506dffeffc8b4a4a432e75c6cb15372c875b068adc6ee56: Status 404 returned error can't find the container with id e37cd1f5d76682e8d506dffeffc8b4a4a432e75c6cb15372c875b068adc6ee56
Sep 11 18:49:52.704455 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:52.704420    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9tzdn" event={"ID":"9a3b9624-a629-437f-a700-3f24e40fc0ae","Type":"ContainerStarted","Data":"e37cd1f5d76682e8d506dffeffc8b4a4a432e75c6cb15372c875b068adc6ee56"}
Sep 11 18:49:52.707414 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:52.707381    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bjcnp" event={"ID":"8498e9d1-1eeb-4ce9-b225-d18452e49150","Type":"ContainerStarted","Data":"6715409d8989bc583b0691421db358c43b12133c8ab8459eb1fac79e7ebd29e9"}
Sep 11 18:49:52.707542 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:52.707442    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bjcnp" event={"ID":"8498e9d1-1eeb-4ce9-b225-d18452e49150","Type":"ContainerStarted","Data":"69e98c2a7813e42b6b159847a02452189e53b48281efbba0d91009222dd9da77"}
Sep 11 18:49:52.724408 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:52.724354    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-bjcnp" podStartSLOduration=67.109102191 podStartE2EDuration="1m8.724333503s" totalImagesPullingTime="1.615231312s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:48:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:49:52.723276068 +0000 UTC m=+68.800866669" watchObservedRunningTime="2026-09-11 18:49:52.724333503 +0000 UTC m=+68.801924060"
Sep 11 18:49:54.003230 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:54.003194    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2edba74f-8029-4255-980e-1132dd147c0a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-pvd8h\" (UID: \"2edba74f-8029-4255-980e-1132dd147c0a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-pvd8h"
Sep 11 18:49:54.006215 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:54.006183    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2edba74f-8029-4255-980e-1132dd147c0a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-pvd8h\" (UID: \"2edba74f-8029-4255-980e-1132dd147c0a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-pvd8h"
Sep 11 18:49:54.104039 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:54.103992    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/23b61d5d-8bb3-43c1-9ae8-38634ec8b885-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-nqtr7\" (UID: \"23b61d5d-8bb3-43c1-9ae8-38634ec8b885\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-nqtr7"
Sep 11 18:49:54.106944 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:54.106917    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/23b61d5d-8bb3-43c1-9ae8-38634ec8b885-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-nqtr7\" (UID: \"23b61d5d-8bb3-43c1-9ae8-38634ec8b885\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-nqtr7"
Sep 11 18:49:54.209390 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:54.209354    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-rrgk5\""
Sep 11 18:49:54.217366 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:54.217344    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-pvd8h"
Sep 11 18:49:54.318611 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:54.318520    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-znvrt\""
Sep 11 18:49:54.326208 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:54.326176    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-nqtr7"
Sep 11 18:49:55.319693 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:55.319653    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-795d6d9b4b-8b8vm"]
Sep 11 18:49:55.322214 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:55.322186    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8745d449b-xs82b"]
Sep 11 18:49:55.322357 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:55.322323    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-795d6d9b4b-8b8vm"
Sep 11 18:49:55.325132 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:55.324818    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8745d449b-xs82b"
Sep 11 18:49:55.339647 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:55.339621    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 11 18:49:55.339969 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:55.339955    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 11 18:49:55.340154 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:55.340139    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 11 18:49:55.340462 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:55.340444    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 11 18:49:55.340524 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:55.340487    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 11 18:49:55.340584 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:55.340534    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 11 18:49:55.340584 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:55.340547    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 11 18:49:55.340584 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:55.340488    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 11 18:49:55.340781 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:55.340765    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-jnw2x\""
Sep 11 18:49:55.357968 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:55.357943    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-795d6d9b4b-8b8vm"]
Sep 11 18:49:55.359058 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:55.359009    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8745d449b-xs82b"]
Sep 11 18:49:55.413138 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:55.413101    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/93a9fbdc-3b62-478e-908e-cd80fafac50f-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-795d6d9b4b-8b8vm\" (UID: \"93a9fbdc-3b62-478e-908e-cd80fafac50f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-795d6d9b4b-8b8vm"
Sep 11 18:49:55.413271 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:55.413147    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/775d805e-0f7c-4909-a007-941d6eb75af0-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-8745d449b-xs82b\" (UID: \"775d805e-0f7c-4909-a007-941d6eb75af0\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8745d449b-xs82b"
Sep 11 18:49:55.413271 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:55.413179    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/93a9fbdc-3b62-478e-908e-cd80fafac50f-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-795d6d9b4b-8b8vm\" (UID: \"93a9fbdc-3b62-478e-908e-cd80fafac50f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-795d6d9b4b-8b8vm"
Sep 11 18:49:55.413271 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:55.413209    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/93a9fbdc-3b62-478e-908e-cd80fafac50f-ca\") pod \"cluster-proxy-proxy-agent-795d6d9b4b-8b8vm\" (UID: \"93a9fbdc-3b62-478e-908e-cd80fafac50f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-795d6d9b4b-8b8vm"
Sep 11 18:49:55.413271 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:55.413235    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/93a9fbdc-3b62-478e-908e-cd80fafac50f-hub\") pod \"cluster-proxy-proxy-agent-795d6d9b4b-8b8vm\" (UID: \"93a9fbdc-3b62-478e-908e-cd80fafac50f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-795d6d9b4b-8b8vm"
Sep 11 18:49:55.413271 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:55.413269    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/93a9fbdc-3b62-478e-908e-cd80fafac50f-additional-service-ca\") pod \"cluster-proxy-proxy-agent-795d6d9b4b-8b8vm\" (UID: \"93a9fbdc-3b62-478e-908e-cd80fafac50f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-795d6d9b4b-8b8vm"
Sep 11 18:49:55.413480 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:55.413294    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sz8f5\" (UniqueName: \"kubernetes.io/projected/93a9fbdc-3b62-478e-908e-cd80fafac50f-kube-api-access-sz8f5\") pod \"cluster-proxy-proxy-agent-795d6d9b4b-8b8vm\" (UID: \"93a9fbdc-3b62-478e-908e-cd80fafac50f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-795d6d9b4b-8b8vm"
Sep 11 18:49:55.413480 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:55.413324    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdx72\" (UniqueName: \"kubernetes.io/projected/775d805e-0f7c-4909-a007-941d6eb75af0-kube-api-access-zdx72\") pod \"managed-serviceaccount-addon-agent-8745d449b-xs82b\" (UID: \"775d805e-0f7c-4909-a007-941d6eb75af0\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8745d449b-xs82b"
Sep 11 18:49:55.513747 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:55.513709    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/93a9fbdc-3b62-478e-908e-cd80fafac50f-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-795d6d9b4b-8b8vm\" (UID: \"93a9fbdc-3b62-478e-908e-cd80fafac50f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-795d6d9b4b-8b8vm"
Sep 11 18:49:55.513913 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:55.513755    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/775d805e-0f7c-4909-a007-941d6eb75af0-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-8745d449b-xs82b\" (UID: \"775d805e-0f7c-4909-a007-941d6eb75af0\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8745d449b-xs82b"
Sep 11 18:49:55.513913 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:55.513774    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/93a9fbdc-3b62-478e-908e-cd80fafac50f-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-795d6d9b4b-8b8vm\" (UID: \"93a9fbdc-3b62-478e-908e-cd80fafac50f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-795d6d9b4b-8b8vm"
Sep 11 18:49:55.513913 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:55.513900    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/93a9fbdc-3b62-478e-908e-cd80fafac50f-ca\") pod \"cluster-proxy-proxy-agent-795d6d9b4b-8b8vm\" (UID: \"93a9fbdc-3b62-478e-908e-cd80fafac50f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-795d6d9b4b-8b8vm"
Sep 11 18:49:55.514074 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:55.513942    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/93a9fbdc-3b62-478e-908e-cd80fafac50f-hub\") pod \"cluster-proxy-proxy-agent-795d6d9b4b-8b8vm\" (UID: \"93a9fbdc-3b62-478e-908e-cd80fafac50f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-795d6d9b4b-8b8vm"
Sep 11 18:49:55.514074 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:55.513993    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/93a9fbdc-3b62-478e-908e-cd80fafac50f-additional-service-ca\") pod \"cluster-proxy-proxy-agent-795d6d9b4b-8b8vm\" (UID: \"93a9fbdc-3b62-478e-908e-cd80fafac50f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-795d6d9b4b-8b8vm"
Sep 11 18:49:55.514176 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:55.514129    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sz8f5\" (UniqueName: \"kubernetes.io/projected/93a9fbdc-3b62-478e-908e-cd80fafac50f-kube-api-access-sz8f5\") pod \"cluster-proxy-proxy-agent-795d6d9b4b-8b8vm\" (UID: \"93a9fbdc-3b62-478e-908e-cd80fafac50f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-795d6d9b4b-8b8vm"
Sep 11 18:49:55.514230 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:55.514177    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdx72\" (UniqueName: \"kubernetes.io/projected/775d805e-0f7c-4909-a007-941d6eb75af0-kube-api-access-zdx72\") pod \"managed-serviceaccount-addon-agent-8745d449b-xs82b\" (UID: \"775d805e-0f7c-4909-a007-941d6eb75af0\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8745d449b-xs82b"
Sep 11 18:49:55.514965 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:55.514909    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/93a9fbdc-3b62-478e-908e-cd80fafac50f-additional-service-ca\") pod \"cluster-proxy-proxy-agent-795d6d9b4b-8b8vm\" (UID: \"93a9fbdc-3b62-478e-908e-cd80fafac50f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-795d6d9b4b-8b8vm"
Sep 11 18:49:55.516914 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:55.516893    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/93a9fbdc-3b62-478e-908e-cd80fafac50f-ca\") pod \"cluster-proxy-proxy-agent-795d6d9b4b-8b8vm\" (UID: \"93a9fbdc-3b62-478e-908e-cd80fafac50f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-795d6d9b4b-8b8vm"
Sep 11 18:49:55.517037 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:55.517019    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/93a9fbdc-3b62-478e-908e-cd80fafac50f-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-795d6d9b4b-8b8vm\" (UID: \"93a9fbdc-3b62-478e-908e-cd80fafac50f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-795d6d9b4b-8b8vm"
Sep 11 18:49:55.517134 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:55.517114    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/93a9fbdc-3b62-478e-908e-cd80fafac50f-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-795d6d9b4b-8b8vm\" (UID: \"93a9fbdc-3b62-478e-908e-cd80fafac50f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-795d6d9b4b-8b8vm"
Sep 11 18:49:55.517218 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:55.517197    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/775d805e-0f7c-4909-a007-941d6eb75af0-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-8745d449b-xs82b\" (UID: \"775d805e-0f7c-4909-a007-941d6eb75af0\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8745d449b-xs82b"
Sep 11 18:49:55.517830 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:55.517800    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/93a9fbdc-3b62-478e-908e-cd80fafac50f-hub\") pod \"cluster-proxy-proxy-agent-795d6d9b4b-8b8vm\" (UID: \"93a9fbdc-3b62-478e-908e-cd80fafac50f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-795d6d9b4b-8b8vm"
Sep 11 18:49:55.551053 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:55.551027    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdx72\" (UniqueName: \"kubernetes.io/projected/775d805e-0f7c-4909-a007-941d6eb75af0-kube-api-access-zdx72\") pod \"managed-serviceaccount-addon-agent-8745d449b-xs82b\" (UID: \"775d805e-0f7c-4909-a007-941d6eb75af0\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8745d449b-xs82b"
Sep 11 18:49:55.551724 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:55.551679    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sz8f5\" (UniqueName: \"kubernetes.io/projected/93a9fbdc-3b62-478e-908e-cd80fafac50f-kube-api-access-sz8f5\") pod \"cluster-proxy-proxy-agent-795d6d9b4b-8b8vm\" (UID: \"93a9fbdc-3b62-478e-908e-cd80fafac50f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-795d6d9b4b-8b8vm"
Sep 11 18:49:55.644336 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:55.644253    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-795d6d9b4b-8b8vm"
Sep 11 18:49:55.651284 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:55.651263    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8745d449b-xs82b"
Sep 11 18:49:56.260846 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:56.260815    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-nqtr7"]
Sep 11 18:49:56.264080 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:49:56.264046    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23b61d5d_8bb3_43c1_9ae8_38634ec8b885.slice/crio-9d44c571a857f00b5b94d689d77376de7aed99ba72b4194ad440dfa498027660 WatchSource:0}: Error finding container 9d44c571a857f00b5b94d689d77376de7aed99ba72b4194ad440dfa498027660: Status 404 returned error can't find the container with id 9d44c571a857f00b5b94d689d77376de7aed99ba72b4194ad440dfa498027660
Sep 11 18:49:56.273222 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:56.273118    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8745d449b-xs82b"]
Sep 11 18:49:56.275287 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:56.275267    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-pvd8h"]
Sep 11 18:49:56.276858 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:49:56.276833    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod775d805e_0f7c_4909_a007_941d6eb75af0.slice/crio-aa6c436f73bb58856083e475f108edfa912e1535c3dbf4d7b51eb91f175b533c WatchSource:0}: Error finding container aa6c436f73bb58856083e475f108edfa912e1535c3dbf4d7b51eb91f175b533c: Status 404 returned error can't find the container with id aa6c436f73bb58856083e475f108edfa912e1535c3dbf4d7b51eb91f175b533c
Sep 11 18:49:56.277836 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:49:56.277813    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2edba74f_8029_4255_980e_1132dd147c0a.slice/crio-208c1db5eb0c80956e594ecbdcab572fb09311822539c960d04d6867fcd84c79 WatchSource:0}: Error finding container 208c1db5eb0c80956e594ecbdcab572fb09311822539c960d04d6867fcd84c79: Status 404 returned error can't find the container with id 208c1db5eb0c80956e594ecbdcab572fb09311822539c960d04d6867fcd84c79
Sep 11 18:49:56.288301 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:56.288275    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-795d6d9b4b-8b8vm"]
Sep 11 18:49:56.294123 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:49:56.294097    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod93a9fbdc_3b62_478e_908e_cd80fafac50f.slice/crio-32349184bdd608ca7fe89c3857b0697fe4d4e0c7ad082fde38bb0d9d06a43c4b WatchSource:0}: Error finding container 32349184bdd608ca7fe89c3857b0697fe4d4e0c7ad082fde38bb0d9d06a43c4b: Status 404 returned error can't find the container with id 32349184bdd608ca7fe89c3857b0697fe4d4e0c7ad082fde38bb0d9d06a43c4b
Sep 11 18:49:56.723423 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:56.723384    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vnkmt" event={"ID":"90acf5d2-d7df-46bb-a380-73f3ae079b75","Type":"ContainerStarted","Data":"9a24b275209efd34459fe834976816dd366eed30343b38063ed51c7ab77b0a82"}
Sep 11 18:49:56.724761 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:56.724738    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-pvd8h" event={"ID":"2edba74f-8029-4255-980e-1132dd147c0a","Type":"ContainerStarted","Data":"208c1db5eb0c80956e594ecbdcab572fb09311822539c960d04d6867fcd84c79"}
Sep 11 18:49:56.725908 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:56.725882    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-nqtr7" event={"ID":"23b61d5d-8bb3-43c1-9ae8-38634ec8b885","Type":"ContainerStarted","Data":"9d44c571a857f00b5b94d689d77376de7aed99ba72b4194ad440dfa498027660"}
Sep 11 18:49:56.727634 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:56.727612    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9tzdn" event={"ID":"9a3b9624-a629-437f-a700-3f24e40fc0ae","Type":"ContainerStarted","Data":"d15b9f7ec59a4fd167a7e8e2d2251fe497ab32ba30d021be0af88bda755af07b"}
Sep 11 18:49:56.727719 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:56.727654    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9tzdn" event={"ID":"9a3b9624-a629-437f-a700-3f24e40fc0ae","Type":"ContainerStarted","Data":"4bc1b110b60c8d951389eabe8722108d0a1a1b007dfd00f1b9893c0746861e74"}
Sep 11 18:49:56.727719 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:56.727697    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-9tzdn"
Sep 11 18:49:56.728850 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:56.728829    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-795d6d9b4b-8b8vm" event={"ID":"93a9fbdc-3b62-478e-908e-cd80fafac50f","Type":"ContainerStarted","Data":"32349184bdd608ca7fe89c3857b0697fe4d4e0c7ad082fde38bb0d9d06a43c4b"}
Sep 11 18:49:56.729951 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:56.729928    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8745d449b-xs82b" event={"ID":"775d805e-0f7c-4909-a007-941d6eb75af0","Type":"ContainerStarted","Data":"aa6c436f73bb58856083e475f108edfa912e1535c3dbf4d7b51eb91f175b533c"}
Sep 11 18:49:56.731302 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:56.731284    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7c8ns" event={"ID":"c7382740-2bba-4368-8e7f-508c64d170a7","Type":"ContainerStarted","Data":"a9ec426f40610461e70da348ffbd0969068b3cac9ffd5821ad23fa3253617f7a"}
Sep 11 18:49:56.731419 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:56.731404    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-7c8ns"
Sep 11 18:49:56.741867 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:56.741825    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-vnkmt" podStartSLOduration=66.468872059 podStartE2EDuration="1m11.741812471s" totalImagesPullingTime="5.272940412s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:48:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:49:56.740730998 +0000 UTC m=+72.818321556" watchObservedRunningTime="2026-09-11 18:49:56.741812471 +0000 UTC m=+72.819403041"
Sep 11 18:49:56.766992 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:56.766952    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-7c8ns" podStartSLOduration=67.502046509 podStartE2EDuration="1m12.76693848s" totalImagesPullingTime="5.264891971s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:48:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:49:56.765792753 +0000 UTC m=+72.843383308" watchObservedRunningTime="2026-09-11 18:49:56.76693848 +0000 UTC m=+72.844529035"
Sep 11 18:49:56.790810 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:56.790766    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-9tzdn" podStartSLOduration=34.124109184 podStartE2EDuration="37.790751851s" totalImagesPullingTime="3.666642667s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:49:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:49:56.790087287 +0000 UTC m=+72.867677887" watchObservedRunningTime="2026-09-11 18:49:56.790751851 +0000 UTC m=+72.868342406"
Sep 11 18:49:58.746309 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:58.746268    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-nqtr7" event={"ID":"23b61d5d-8bb3-43c1-9ae8-38634ec8b885","Type":"ContainerStarted","Data":"e4b0596190b7557d70c591c84837df2b78ed4a3fff8af10f0ddfd8618a67fc24"}
Sep 11 18:49:58.763251 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:49:58.763192    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-nqtr7" podStartSLOduration=35.382154168 podStartE2EDuration="36.763171186s" totalImagesPullingTime="1.381017018s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:49:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:49:58.761780761 +0000 UTC m=+74.839371335" watchObservedRunningTime="2026-09-11 18:49:58.763171186 +0000 UTC m=+74.840761757"
Sep 11 18:50:01.763915 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:50:01.763878    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-795d6d9b4b-8b8vm" event={"ID":"93a9fbdc-3b62-478e-908e-cd80fafac50f","Type":"ContainerStarted","Data":"7eddd1db8b61f208a7f6762e64136c43ffe33614b62c1c1a4588c91ea62f5575"}
Sep 11 18:50:01.764338 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:50:01.763928    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-795d6d9b4b-8b8vm" event={"ID":"93a9fbdc-3b62-478e-908e-cd80fafac50f","Type":"ContainerStarted","Data":"c3ed77a615ba57e4640e9b0ebe0bee8fb542b75e6479fef718068b865582c47c"}
Sep 11 18:50:01.764338 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:50:01.763949    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-795d6d9b4b-8b8vm" event={"ID":"93a9fbdc-3b62-478e-908e-cd80fafac50f","Type":"ContainerStarted","Data":"c0a3c3937ee6dfa7fe62f148ececabee4394de758d9de4812feccf7848ad9a48"}
Sep 11 18:50:01.765890 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:50:01.765860    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8745d449b-xs82b" event={"ID":"775d805e-0f7c-4909-a007-941d6eb75af0","Type":"ContainerStarted","Data":"6526b27b3ac8b3cf218d9b6fc00f0a513028f18f4b24637232e75f5500d1dc52"}
Sep 11 18:50:01.767390 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:50:01.767365    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-pvd8h" event={"ID":"2edba74f-8029-4255-980e-1132dd147c0a","Type":"ContainerStarted","Data":"76557dd3160d78d4857609336bb9010c20894bf4c20af840a81b446ada908d16"}
Sep 11 18:50:01.787417 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:50:01.787374    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-795d6d9b4b-8b8vm" podStartSLOduration=2.212859307 podStartE2EDuration="6.787361045s" totalImagesPullingTime="4.574501738s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:49:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:50:01.78497731 +0000 UTC m=+77.862567865" watchObservedRunningTime="2026-09-11 18:50:01.787361045 +0000 UTC m=+77.864951600"
Sep 11 18:50:01.806240 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:50:01.806197    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8745d449b-xs82b" podStartSLOduration=2.220195985 podStartE2EDuration="6.806181477s" totalImagesPullingTime="4.585985492s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:49:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:50:01.805634735 +0000 UTC m=+77.883225290" watchObservedRunningTime="2026-09-11 18:50:01.806181477 +0000 UTC m=+77.883772031"
Sep 11 18:50:01.826099 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:50:01.826055    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-pvd8h" podStartSLOduration=35.255701838 podStartE2EDuration="39.826036196s" totalImagesPullingTime="4.570334358s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:49:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:50:01.825302173 +0000 UTC m=+77.902892795" watchObservedRunningTime="2026-09-11 18:50:01.826036196 +0000 UTC m=+77.903626752"
Sep 11 18:50:06.743569 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:50:06.743461    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-9tzdn"
Sep 11 18:50:17.223043 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:50:17.223010    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-wm4kw"]
Sep 11 18:50:17.227633 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:50:17.227584    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-wm4kw"
Sep 11 18:50:17.231315 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:50:17.231291    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 11 18:50:17.231438 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:50:17.231292    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-2wfkj\""
Sep 11 18:50:17.231643 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:50:17.231624    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 11 18:50:17.231735 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:50:17.231664    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 11 18:50:17.231735 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:50:17.231670    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 11 18:50:17.259119 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:50:17.259091    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqjx5\" (UniqueName: \"kubernetes.io/projected/eddf1446-f626-4056-815d-32d95e637ae8-kube-api-access-bqjx5\") pod \"node-exporter-wm4kw\" (UID: \"eddf1446-f626-4056-815d-32d95e637ae8\") " pod="openshift-monitoring/node-exporter-wm4kw"
Sep 11 18:50:17.259223 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:50:17.259129    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eddf1446-f626-4056-815d-32d95e637ae8-metrics-client-ca\") pod \"node-exporter-wm4kw\" (UID: \"eddf1446-f626-4056-815d-32d95e637ae8\") " pod="openshift-monitoring/node-exporter-wm4kw"
Sep 11 18:50:17.259223 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:50:17.259158    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/eddf1446-f626-4056-815d-32d95e637ae8-root\") pod \"node-exporter-wm4kw\" (UID: \"eddf1446-f626-4056-815d-32d95e637ae8\") " pod="openshift-monitoring/node-exporter-wm4kw"
Sep 11 18:50:17.259223 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:50:17.259174    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/eddf1446-f626-4056-815d-32d95e637ae8-node-exporter-tls\") pod \"node-exporter-wm4kw\" (UID: \"eddf1446-f626-4056-815d-32d95e637ae8\") " pod="openshift-monitoring/node-exporter-wm4kw"
Sep 11 18:50:17.259223 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:50:17.259189    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/eddf1446-f626-4056-815d-32d95e637ae8-node-exporter-wtmp\") pod \"node-exporter-wm4kw\" (UID: \"eddf1446-f626-4056-815d-32d95e637ae8\") " pod="openshift-monitoring/node-exporter-wm4kw"
Sep 11 18:50:17.259372 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:50:17.259226    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eddf1446-f626-4056-815d-32d95e637ae8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wm4kw\" (UID: \"eddf1446-f626-4056-815d-32d95e637ae8\") " pod="openshift-monitoring/node-exporter-wm4kw"
Sep 11 18:50:17.259372 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:50:17.259242    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/eddf1446-f626-4056-815d-32d95e637ae8-node-exporter-accelerators-collector-config\") pod \"node-exporter-wm4kw\" (UID: \"eddf1446-f626-4056-815d-32d95e637ae8\") " pod="openshift-monitoring/node-exporter-wm4kw"
Sep 11 18:50:17.259372 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:50:17.259260    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eddf1446-f626-4056-815d-32d95e637ae8-sys\") pod \"node-exporter-wm4kw\" (UID: \"eddf1446-f626-4056-815d-32d95e637ae8\") " pod="openshift-monitoring/node-exporter-wm4kw"
Sep 11 18:50:17.259372 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:50:17.259273    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/eddf1446-f626-4056-815d-32d95e637ae8-node-exporter-textfile\") pod \"node-exporter-wm4kw\" (UID: \"eddf1446-f626-4056-815d-32d95e637ae8\") " pod="openshift-monitoring/node-exporter-wm4kw"
Sep 11 18:50:17.359796 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:50:17.359761    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/eddf1446-f626-4056-815d-32d95e637ae8-root\") pod \"node-exporter-wm4kw\" (UID: \"eddf1446-f626-4056-815d-32d95e637ae8\") " pod="openshift-monitoring/node-exporter-wm4kw"
Sep 11 18:50:17.359796 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:50:17.359794    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/eddf1446-f626-4056-815d-32d95e637ae8-node-exporter-tls\") pod \"node-exporter-wm4kw\" (UID: \"eddf1446-f626-4056-815d-32d95e637ae8\") " pod="openshift-monitoring/node-exporter-wm4kw"
Sep 11 18:50:17.360073 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:50:17.359810    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/eddf1446-f626-4056-815d-32d95e637ae8-node-exporter-wtmp\") pod \"node-exporter-wm4kw\" (UID: \"eddf1446-f626-4056-815d-32d95e637ae8\") " pod="openshift-monitoring/node-exporter-wm4kw"
Sep 11 18:50:17.360073 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:50:17.359831    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eddf1446-f626-4056-815d-32d95e637ae8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wm4kw\" (UID: \"eddf1446-f626-4056-815d-32d95e637ae8\") " pod="openshift-monitoring/node-exporter-wm4kw"
Sep 11 18:50:17.360073 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:50:17.359849    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/eddf1446-f626-4056-815d-32d95e637ae8-node-exporter-accelerators-collector-config\") pod \"node-exporter-wm4kw\" (UID: \"eddf1446-f626-4056-815d-32d95e637ae8\") " pod="openshift-monitoring/node-exporter-wm4kw"
Sep 11 18:50:17.360073 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:50:17.359871    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eddf1446-f626-4056-815d-32d95e637ae8-sys\") pod \"node-exporter-wm4kw\" (UID: \"eddf1446-f626-4056-815d-32d95e637ae8\") " pod="openshift-monitoring/node-exporter-wm4kw"
Sep 11 18:50:17.360073 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:50:17.359875    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/eddf1446-f626-4056-815d-32d95e637ae8-root\") pod \"node-exporter-wm4kw\" (UID: \"eddf1446-f626-4056-815d-32d95e637ae8\") " pod="openshift-monitoring/node-exporter-wm4kw"
Sep 11 18:50:17.360073 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:50:17.359907    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/eddf1446-f626-4056-815d-32d95e637ae8-node-exporter-textfile\") pod \"node-exporter-wm4kw\" (UID: \"eddf1446-f626-4056-815d-32d95e637ae8\") " pod="openshift-monitoring/node-exporter-wm4kw"
Sep 11 18:50:17.360073 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:50:17.359933    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqjx5\" (UniqueName: \"kubernetes.io/projected/eddf1446-f626-4056-815d-32d95e637ae8-kube-api-access-bqjx5\") pod \"node-exporter-wm4kw\" (UID: \"eddf1446-f626-4056-815d-32d95e637ae8\") " pod="openshift-monitoring/node-exporter-wm4kw"
Sep 11 18:50:17.360073 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:50:17.359942    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eddf1446-f626-4056-815d-32d95e637ae8-sys\") pod \"node-exporter-wm4kw\" (UID: \"eddf1446-f626-4056-815d-32d95e637ae8\") " pod="openshift-monitoring/node-exporter-wm4kw"
Sep 11 18:50:17.360073 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:50:17.359976    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/eddf1446-f626-4056-815d-32d95e637ae8-node-exporter-wtmp\") pod \"node-exporter-wm4kw\" (UID: \"eddf1446-f626-4056-815d-32d95e637ae8\") " pod="openshift-monitoring/node-exporter-wm4kw"
Sep 11 18:50:17.360073 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:50:17.359992    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eddf1446-f626-4056-815d-32d95e637ae8-metrics-client-ca\") pod \"node-exporter-wm4kw\" (UID: \"eddf1446-f626-4056-815d-32d95e637ae8\") " pod="openshift-monitoring/node-exporter-wm4kw"
Sep 11 18:50:17.360507 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:50:17.360361    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/eddf1446-f626-4056-815d-32d95e637ae8-node-exporter-textfile\") pod \"node-exporter-wm4kw\" (UID: \"eddf1446-f626-4056-815d-32d95e637ae8\") " pod="openshift-monitoring/node-exporter-wm4kw"
Sep 11 18:50:17.360558 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:50:17.360533    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eddf1446-f626-4056-815d-32d95e637ae8-metrics-client-ca\") pod \"node-exporter-wm4kw\" (UID: \"eddf1446-f626-4056-815d-32d95e637ae8\") " pod="openshift-monitoring/node-exporter-wm4kw"
Sep 11 18:50:17.360622 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:50:17.360577    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/eddf1446-f626-4056-815d-32d95e637ae8-node-exporter-accelerators-collector-config\") pod \"node-exporter-wm4kw\" (UID: \"eddf1446-f626-4056-815d-32d95e637ae8\") " pod="openshift-monitoring/node-exporter-wm4kw"
Sep 11 18:50:17.362257 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:50:17.362240    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eddf1446-f626-4056-815d-32d95e637ae8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wm4kw\" (UID: \"eddf1446-f626-4056-815d-32d95e637ae8\") " pod="openshift-monitoring/node-exporter-wm4kw"
Sep 11 18:50:17.362369 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:50:17.362351    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/eddf1446-f626-4056-815d-32d95e637ae8-node-exporter-tls\") pod \"node-exporter-wm4kw\" (UID: \"eddf1446-f626-4056-815d-32d95e637ae8\") " pod="openshift-monitoring/node-exporter-wm4kw"
Sep 11 18:50:17.371856 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:50:17.371824    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqjx5\" (UniqueName: \"kubernetes.io/projected/eddf1446-f626-4056-815d-32d95e637ae8-kube-api-access-bqjx5\") pod \"node-exporter-wm4kw\" (UID: \"eddf1446-f626-4056-815d-32d95e637ae8\") " pod="openshift-monitoring/node-exporter-wm4kw"
Sep 11 18:50:17.536808 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:50:17.536731    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-wm4kw"
Sep 11 18:50:17.547767 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:50:17.547736    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeddf1446_f626_4056_815d_32d95e637ae8.slice/crio-ed75466a33a856aa52e2d086fe33ad9b58c54718fe5e6fdc753a72970d8335fe WatchSource:0}: Error finding container ed75466a33a856aa52e2d086fe33ad9b58c54718fe5e6fdc753a72970d8335fe: Status 404 returned error can't find the container with id ed75466a33a856aa52e2d086fe33ad9b58c54718fe5e6fdc753a72970d8335fe
Sep 11 18:50:17.822032 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:50:17.821938    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wm4kw" event={"ID":"eddf1446-f626-4056-815d-32d95e637ae8","Type":"ContainerStarted","Data":"ed75466a33a856aa52e2d086fe33ad9b58c54718fe5e6fdc753a72970d8335fe"}
Sep 11 18:50:18.827172 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:50:18.827089    2578 generic.go:356] "Generic (PLEG): container finished" podID="eddf1446-f626-4056-815d-32d95e637ae8" containerID="3a51379cbc1dc284ac98e570d696b5ea8a7f087f7065534c9b681f63b88d3166" exitCode=0
Sep 11 18:50:18.827172 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:50:18.827140    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wm4kw" event={"ID":"eddf1446-f626-4056-815d-32d95e637ae8","Type":"ContainerDied","Data":"3a51379cbc1dc284ac98e570d696b5ea8a7f087f7065534c9b681f63b88d3166"}
Sep 11 18:50:19.833465 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:50:19.833426    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wm4kw" event={"ID":"eddf1446-f626-4056-815d-32d95e637ae8","Type":"ContainerStarted","Data":"fbf190724bbd88800eb53f936f637bec18f95b77efe0d12e4085af2a3aae60a1"}
Sep 11 18:50:19.833892 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:50:19.833517    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wm4kw" event={"ID":"eddf1446-f626-4056-815d-32d95e637ae8","Type":"ContainerStarted","Data":"45d5f005fc1f94f7963634f82394aeb72000b3bb01aeec5a656b6e0c3933ddd2"}
Sep 11 18:50:19.854896 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:50:19.854848    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-wm4kw" podStartSLOduration=1.756679806 podStartE2EDuration="2.854834404s" totalImagesPullingTime="1.029221981s" totalInitContainerRuntime="68.932617ms" isStatefulPod=true podCreationTimestamp="2026-09-11 18:50:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:50:19.852826263 +0000 UTC m=+95.930416831" watchObservedRunningTime="2026-09-11 18:50:19.854834404 +0000 UTC m=+95.932424958"
Sep 11 18:50:27.742722 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:50:27.742687    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-7c8ns"
Sep 11 18:50:46.077279 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:50:46.077249    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bw8xb_0e108d53-e1ef-4459-9eee-2e4be2ccf6e3/serve-healthcheck-canary/0.log"
Sep 11 18:53:00.879605 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:00.879571    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jqk42"]
Sep 11 18:53:00.881585 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:00.881570    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jqk42"
Sep 11 18:53:00.883821 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:00.883798    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 11 18:53:00.883934 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:00.883807    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-mhgfb\""
Sep 11 18:53:00.884241 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:00.884226    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 11 18:53:00.892084 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:00.892064    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jqk42"]
Sep 11 18:53:00.897083 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:00.897058    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/28145fdf-93bd-4892-b678-ad0670e859dd-tmp\") pod \"openshift-lws-operator-fd8ccff4c-jqk42\" (UID: \"28145fdf-93bd-4892-b678-ad0670e859dd\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jqk42"
Sep 11 18:53:00.897165 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:00.897127    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcztp\" (UniqueName: \"kubernetes.io/projected/28145fdf-93bd-4892-b678-ad0670e859dd-kube-api-access-xcztp\") pod \"openshift-lws-operator-fd8ccff4c-jqk42\" (UID: \"28145fdf-93bd-4892-b678-ad0670e859dd\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jqk42"
Sep 11 18:53:00.998123 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:00.998095    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/28145fdf-93bd-4892-b678-ad0670e859dd-tmp\") pod \"openshift-lws-operator-fd8ccff4c-jqk42\" (UID: \"28145fdf-93bd-4892-b678-ad0670e859dd\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jqk42"
Sep 11 18:53:00.998268 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:00.998145    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcztp\" (UniqueName: \"kubernetes.io/projected/28145fdf-93bd-4892-b678-ad0670e859dd-kube-api-access-xcztp\") pod \"openshift-lws-operator-fd8ccff4c-jqk42\" (UID: \"28145fdf-93bd-4892-b678-ad0670e859dd\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jqk42"
Sep 11 18:53:00.998479 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:00.998446    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/28145fdf-93bd-4892-b678-ad0670e859dd-tmp\") pod \"openshift-lws-operator-fd8ccff4c-jqk42\" (UID: \"28145fdf-93bd-4892-b678-ad0670e859dd\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jqk42"
Sep 11 18:53:01.006899 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:01.006876    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcztp\" (UniqueName: \"kubernetes.io/projected/28145fdf-93bd-4892-b678-ad0670e859dd-kube-api-access-xcztp\") pod \"openshift-lws-operator-fd8ccff4c-jqk42\" (UID: \"28145fdf-93bd-4892-b678-ad0670e859dd\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jqk42"
Sep 11 18:53:01.190081 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:01.190004    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jqk42"
Sep 11 18:53:01.470892 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:01.470862    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jqk42"]
Sep 11 18:53:01.474254 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:53:01.474223    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod28145fdf_93bd_4892_b678_ad0670e859dd.slice/crio-bfe6e1bc7068890bff1204f6c4f1982d6ab33fa7f8f28212e4561cef99072fd0 WatchSource:0}: Error finding container bfe6e1bc7068890bff1204f6c4f1982d6ab33fa7f8f28212e4561cef99072fd0: Status 404 returned error can't find the container with id bfe6e1bc7068890bff1204f6c4f1982d6ab33fa7f8f28212e4561cef99072fd0
Sep 11 18:53:02.387559 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:02.387520    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jqk42" event={"ID":"28145fdf-93bd-4892-b678-ad0670e859dd","Type":"ContainerStarted","Data":"bfe6e1bc7068890bff1204f6c4f1982d6ab33fa7f8f28212e4561cef99072fd0"}
Sep 11 18:53:04.396758 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:04.396718    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jqk42" event={"ID":"28145fdf-93bd-4892-b678-ad0670e859dd","Type":"ContainerStarted","Data":"c5a9d29d10683c49d58010dd47992001da24275d004258a1281a9aecbae3cc23"}
Sep 11 18:53:04.413931 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:04.413880    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jqk42" podStartSLOduration=1.644319603 podStartE2EDuration="4.413865083s" totalImagesPullingTime="2.76954548s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:53:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:53:04.412218664 +0000 UTC m=+260.489809220" watchObservedRunningTime="2026-09-11 18:53:04.413865083 +0000 UTC m=+260.491455638"
Sep 11 18:53:23.481930 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:23.481895    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-c9988978f-lm6lb"]
Sep 11 18:53:23.489810 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:23.489794    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-c9988978f-lm6lb"
Sep 11 18:53:23.495515 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:23.495492    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-snlds\""
Sep 11 18:53:23.495661 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:23.495523    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 11 18:53:23.496431 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:23.496412    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 11 18:53:23.496551 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:23.496536    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 11 18:53:23.499355 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:23.499340    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 11 18:53:23.533160 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:23.533135    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-c9988978f-lm6lb"]
Sep 11 18:53:23.558207 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:23.558183    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a8434d8c-e2eb-4f72-94cd-d34c6a10b5b6-manifests\") pod \"opendatahub-operator-controller-manager-c9988978f-lm6lb\" (UID: \"a8434d8c-e2eb-4f72-94cd-d34c6a10b5b6\") " pod="opendatahub/opendatahub-operator-controller-manager-c9988978f-lm6lb"
Sep 11 18:53:23.558349 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:23.558220    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a8434d8c-e2eb-4f72-94cd-d34c6a10b5b6-tmp\") pod \"opendatahub-operator-controller-manager-c9988978f-lm6lb\" (UID: \"a8434d8c-e2eb-4f72-94cd-d34c6a10b5b6\") " pod="opendatahub/opendatahub-operator-controller-manager-c9988978f-lm6lb"
Sep 11 18:53:23.558349 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:23.558251    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a8434d8c-e2eb-4f72-94cd-d34c6a10b5b6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c9988978f-lm6lb\" (UID: \"a8434d8c-e2eb-4f72-94cd-d34c6a10b5b6\") " pod="opendatahub/opendatahub-operator-controller-manager-c9988978f-lm6lb"
Sep 11 18:53:23.558349 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:23.558310    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a8434d8c-e2eb-4f72-94cd-d34c6a10b5b6-webhook-cert\") pod \"opendatahub-operator-controller-manager-c9988978f-lm6lb\" (UID: \"a8434d8c-e2eb-4f72-94cd-d34c6a10b5b6\") " pod="opendatahub/opendatahub-operator-controller-manager-c9988978f-lm6lb"
Sep 11 18:53:23.558450 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:23.558358    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6vv9\" (UniqueName: \"kubernetes.io/projected/a8434d8c-e2eb-4f72-94cd-d34c6a10b5b6-kube-api-access-s6vv9\") pod \"opendatahub-operator-controller-manager-c9988978f-lm6lb\" (UID: \"a8434d8c-e2eb-4f72-94cd-d34c6a10b5b6\") " pod="opendatahub/opendatahub-operator-controller-manager-c9988978f-lm6lb"
Sep 11 18:53:23.658941 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:23.658904    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a8434d8c-e2eb-4f72-94cd-d34c6a10b5b6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c9988978f-lm6lb\" (UID: \"a8434d8c-e2eb-4f72-94cd-d34c6a10b5b6\") " pod="opendatahub/opendatahub-operator-controller-manager-c9988978f-lm6lb"
Sep 11 18:53:23.658941 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:23.658948    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a8434d8c-e2eb-4f72-94cd-d34c6a10b5b6-webhook-cert\") pod \"opendatahub-operator-controller-manager-c9988978f-lm6lb\" (UID: \"a8434d8c-e2eb-4f72-94cd-d34c6a10b5b6\") " pod="opendatahub/opendatahub-operator-controller-manager-c9988978f-lm6lb"
Sep 11 18:53:23.659204 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:23.658966    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6vv9\" (UniqueName: \"kubernetes.io/projected/a8434d8c-e2eb-4f72-94cd-d34c6a10b5b6-kube-api-access-s6vv9\") pod \"opendatahub-operator-controller-manager-c9988978f-lm6lb\" (UID: \"a8434d8c-e2eb-4f72-94cd-d34c6a10b5b6\") " pod="opendatahub/opendatahub-operator-controller-manager-c9988978f-lm6lb"
Sep 11 18:53:23.659204 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:23.658999    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a8434d8c-e2eb-4f72-94cd-d34c6a10b5b6-manifests\") pod \"opendatahub-operator-controller-manager-c9988978f-lm6lb\" (UID: \"a8434d8c-e2eb-4f72-94cd-d34c6a10b5b6\") " pod="opendatahub/opendatahub-operator-controller-manager-c9988978f-lm6lb"
Sep 11 18:53:23.659204 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:23.659024    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a8434d8c-e2eb-4f72-94cd-d34c6a10b5b6-tmp\") pod \"opendatahub-operator-controller-manager-c9988978f-lm6lb\" (UID: \"a8434d8c-e2eb-4f72-94cd-d34c6a10b5b6\") " pod="opendatahub/opendatahub-operator-controller-manager-c9988978f-lm6lb"
Sep 11 18:53:23.659503 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:23.659479    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a8434d8c-e2eb-4f72-94cd-d34c6a10b5b6-manifests\") pod \"opendatahub-operator-controller-manager-c9988978f-lm6lb\" (UID: \"a8434d8c-e2eb-4f72-94cd-d34c6a10b5b6\") " pod="opendatahub/opendatahub-operator-controller-manager-c9988978f-lm6lb"
Sep 11 18:53:23.661424 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:23.661395    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a8434d8c-e2eb-4f72-94cd-d34c6a10b5b6-tmp\") pod \"opendatahub-operator-controller-manager-c9988978f-lm6lb\" (UID: \"a8434d8c-e2eb-4f72-94cd-d34c6a10b5b6\") " pod="opendatahub/opendatahub-operator-controller-manager-c9988978f-lm6lb"
Sep 11 18:53:23.661628 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:23.661587    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a8434d8c-e2eb-4f72-94cd-d34c6a10b5b6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c9988978f-lm6lb\" (UID: \"a8434d8c-e2eb-4f72-94cd-d34c6a10b5b6\") " pod="opendatahub/opendatahub-operator-controller-manager-c9988978f-lm6lb"
Sep 11 18:53:23.661677 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:23.661639    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a8434d8c-e2eb-4f72-94cd-d34c6a10b5b6-webhook-cert\") pod \"opendatahub-operator-controller-manager-c9988978f-lm6lb\" (UID: \"a8434d8c-e2eb-4f72-94cd-d34c6a10b5b6\") " pod="opendatahub/opendatahub-operator-controller-manager-c9988978f-lm6lb"
Sep 11 18:53:23.668442 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:23.668424    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6vv9\" (UniqueName: \"kubernetes.io/projected/a8434d8c-e2eb-4f72-94cd-d34c6a10b5b6-kube-api-access-s6vv9\") pod \"opendatahub-operator-controller-manager-c9988978f-lm6lb\" (UID: \"a8434d8c-e2eb-4f72-94cd-d34c6a10b5b6\") " pod="opendatahub/opendatahub-operator-controller-manager-c9988978f-lm6lb"
Sep 11 18:53:23.799677 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:23.799562    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-c9988978f-lm6lb"
Sep 11 18:53:24.090927 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:24.090852    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-c9988978f-lm6lb"]
Sep 11 18:53:24.093970 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:53:24.093944    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8434d8c_e2eb_4f72_94cd_d34c6a10b5b6.slice/crio-e921925a9abaf31620365fa6f6d9cfa6538f62e74790153c9cd0c019dedc02a6 WatchSource:0}: Error finding container e921925a9abaf31620365fa6f6d9cfa6538f62e74790153c9cd0c019dedc02a6: Status 404 returned error can't find the container with id e921925a9abaf31620365fa6f6d9cfa6538f62e74790153c9cd0c019dedc02a6
Sep 11 18:53:24.467077 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:24.467045    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-c9988978f-lm6lb" event={"ID":"a8434d8c-e2eb-4f72-94cd-d34c6a10b5b6","Type":"ContainerStarted","Data":"e921925a9abaf31620365fa6f6d9cfa6538f62e74790153c9cd0c019dedc02a6"}
Sep 11 18:53:27.481376 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:27.481340    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-c9988978f-lm6lb" event={"ID":"a8434d8c-e2eb-4f72-94cd-d34c6a10b5b6","Type":"ContainerStarted","Data":"44765032bae6d5d2e22038f37b16a418e1af368eafebc9364cb925bba0e6ff11"}
Sep 11 18:53:28.485777 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:28.485743    2578 generic.go:356] "Generic (PLEG): container finished" podID="a8434d8c-e2eb-4f72-94cd-d34c6a10b5b6" containerID="44765032bae6d5d2e22038f37b16a418e1af368eafebc9364cb925bba0e6ff11" exitCode=0
Sep 11 18:53:28.486187 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:28.485824    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-c9988978f-lm6lb" event={"ID":"a8434d8c-e2eb-4f72-94cd-d34c6a10b5b6","Type":"ContainerDied","Data":"44765032bae6d5d2e22038f37b16a418e1af368eafebc9364cb925bba0e6ff11"}
Sep 11 18:53:29.492008 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:29.491979    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-c9988978f-lm6lb" event={"ID":"a8434d8c-e2eb-4f72-94cd-d34c6a10b5b6","Type":"ContainerStarted","Data":"99b9b436c3ed0d4d0d4cdb0adc70554358a5a5ce9f6504164d3e60f816938708"}
Sep 11 18:53:29.492392 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:29.492109    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-c9988978f-lm6lb"
Sep 11 18:53:29.514906 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:29.514857    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-c9988978f-lm6lb" podStartSLOduration=2.188215131 podStartE2EDuration="6.514841254s" totalImagesPullingTime="3.307088706s" totalInitContainerRuntime="1.019537417s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:53:23 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:53:29.513172919 +0000 UTC m=+285.590763474" watchObservedRunningTime="2026-09-11 18:53:29.514841254 +0000 UTC m=+285.592431840"
Sep 11 18:53:40.497526 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:40.497496    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-c9988978f-lm6lb"
Sep 11 18:53:43.487613 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:43.487570    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5868776b49-nkdlk"]
Sep 11 18:53:43.490624 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:43.490576    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5868776b49-nkdlk"
Sep 11 18:53:43.497652 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:43.497629    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 11 18:53:43.497760 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:43.497685    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 11 18:53:43.499216 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:43.499200    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 11 18:53:43.503834 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:43.503819    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 11 18:53:43.504313 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:43.504299    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-xp752\""
Sep 11 18:53:43.520101 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:43.520078    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5868776b49-nkdlk"]
Sep 11 18:53:43.590772 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:43.590735    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsftq\" (UniqueName: \"kubernetes.io/projected/70de1f38-8114-4f41-8d3f-76200ab1ff91-kube-api-access-hsftq\") pod \"kube-auth-proxy-5868776b49-nkdlk\" (UID: \"70de1f38-8114-4f41-8d3f-76200ab1ff91\") " pod="openshift-ingress/kube-auth-proxy-5868776b49-nkdlk"
Sep 11 18:53:43.590905 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:43.590789    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/70de1f38-8114-4f41-8d3f-76200ab1ff91-tmp\") pod \"kube-auth-proxy-5868776b49-nkdlk\" (UID: \"70de1f38-8114-4f41-8d3f-76200ab1ff91\") " pod="openshift-ingress/kube-auth-proxy-5868776b49-nkdlk"
Sep 11 18:53:43.590905 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:43.590854    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70de1f38-8114-4f41-8d3f-76200ab1ff91-tls-certs\") pod \"kube-auth-proxy-5868776b49-nkdlk\" (UID: \"70de1f38-8114-4f41-8d3f-76200ab1ff91\") " pod="openshift-ingress/kube-auth-proxy-5868776b49-nkdlk"
Sep 11 18:53:43.691718 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:43.691675    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70de1f38-8114-4f41-8d3f-76200ab1ff91-tls-certs\") pod \"kube-auth-proxy-5868776b49-nkdlk\" (UID: \"70de1f38-8114-4f41-8d3f-76200ab1ff91\") " pod="openshift-ingress/kube-auth-proxy-5868776b49-nkdlk"
Sep 11 18:53:43.691862 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:43.691736    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsftq\" (UniqueName: \"kubernetes.io/projected/70de1f38-8114-4f41-8d3f-76200ab1ff91-kube-api-access-hsftq\") pod \"kube-auth-proxy-5868776b49-nkdlk\" (UID: \"70de1f38-8114-4f41-8d3f-76200ab1ff91\") " pod="openshift-ingress/kube-auth-proxy-5868776b49-nkdlk"
Sep 11 18:53:43.691862 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:43.691763    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/70de1f38-8114-4f41-8d3f-76200ab1ff91-tmp\") pod \"kube-auth-proxy-5868776b49-nkdlk\" (UID: \"70de1f38-8114-4f41-8d3f-76200ab1ff91\") " pod="openshift-ingress/kube-auth-proxy-5868776b49-nkdlk"
Sep 11 18:53:43.694101 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:43.694073    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/70de1f38-8114-4f41-8d3f-76200ab1ff91-tmp\") pod \"kube-auth-proxy-5868776b49-nkdlk\" (UID: \"70de1f38-8114-4f41-8d3f-76200ab1ff91\") " pod="openshift-ingress/kube-auth-proxy-5868776b49-nkdlk"
Sep 11 18:53:43.694214 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:43.694175    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70de1f38-8114-4f41-8d3f-76200ab1ff91-tls-certs\") pod \"kube-auth-proxy-5868776b49-nkdlk\" (UID: \"70de1f38-8114-4f41-8d3f-76200ab1ff91\") " pod="openshift-ingress/kube-auth-proxy-5868776b49-nkdlk"
Sep 11 18:53:43.711511 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:43.711485    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsftq\" (UniqueName: \"kubernetes.io/projected/70de1f38-8114-4f41-8d3f-76200ab1ff91-kube-api-access-hsftq\") pod \"kube-auth-proxy-5868776b49-nkdlk\" (UID: \"70de1f38-8114-4f41-8d3f-76200ab1ff91\") " pod="openshift-ingress/kube-auth-proxy-5868776b49-nkdlk"
Sep 11 18:53:43.800670 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:43.800582    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5868776b49-nkdlk"
Sep 11 18:53:44.098226 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:44.098149    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5868776b49-nkdlk"]
Sep 11 18:53:44.101906 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:53:44.101882    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70de1f38_8114_4f41_8d3f_76200ab1ff91.slice/crio-cafdc0446c84f750bdb4597e72707cac82a43d960704e2f7ab23b8bd8db9aab0 WatchSource:0}: Error finding container cafdc0446c84f750bdb4597e72707cac82a43d960704e2f7ab23b8bd8db9aab0: Status 404 returned error can't find the container with id cafdc0446c84f750bdb4597e72707cac82a43d960704e2f7ab23b8bd8db9aab0
Sep 11 18:53:44.273382 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:44.273350    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-hsbsf_d0880abe-ec36-48e4-a511-ec663873fffa/console-operator/1.log"
Sep 11 18:53:44.274376 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:44.274346    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-hsbsf_d0880abe-ec36-48e4-a511-ec663873fffa/console-operator/1.log"
Sep 11 18:53:44.284018 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:44.283992    2578 kubelet.go:1648] "Image garbage collection succeeded"
Sep 11 18:53:44.545537 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:44.545500    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5868776b49-nkdlk" event={"ID":"70de1f38-8114-4f41-8d3f-76200ab1ff91","Type":"ContainerStarted","Data":"cafdc0446c84f750bdb4597e72707cac82a43d960704e2f7ab23b8bd8db9aab0"}
Sep 11 18:53:47.014470 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:47.014445    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 11 18:53:47.560343 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:47.560305    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5868776b49-nkdlk" event={"ID":"70de1f38-8114-4f41-8d3f-76200ab1ff91","Type":"ContainerStarted","Data":"71fd5bd82acfa8e6a9013a39b6db442e830acc0e8d47645f8ca84ab29d364c2b"}
Sep 11 18:53:47.580296 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:47.580248    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5868776b49-nkdlk" podStartSLOduration=1.672197411 podStartE2EDuration="4.58023564s" totalImagesPullingTime="2.908038229s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:53:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:53:47.578961837 +0000 UTC m=+303.656552391" watchObservedRunningTime="2026-09-11 18:53:47.58023564 +0000 UTC m=+303.657826211"
Sep 11 18:53:51.429346 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:51.429315    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kjl6k"]
Sep 11 18:53:51.432460 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:51.432444    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kjl6k"
Sep 11 18:53:51.435711 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:51.435689    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-m4cg6\""
Sep 11 18:53:51.435840 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:51.435739    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 11 18:53:51.448427 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:51.448403    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kjl6k"]
Sep 11 18:53:51.555791 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:51.555762    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzbz4\" (UniqueName: \"kubernetes.io/projected/db1ed3da-67dc-4a03-a4e3-5976780ac68d-kube-api-access-qzbz4\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-kjl6k\" (UID: \"db1ed3da-67dc-4a03-a4e3-5976780ac68d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kjl6k"
Sep 11 18:53:51.555961 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:51.555799    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db1ed3da-67dc-4a03-a4e3-5976780ac68d-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-kjl6k\" (UID: \"db1ed3da-67dc-4a03-a4e3-5976780ac68d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kjl6k"
Sep 11 18:53:51.555961 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:51.555838    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db1ed3da-67dc-4a03-a4e3-5976780ac68d-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-kjl6k\" (UID: \"db1ed3da-67dc-4a03-a4e3-5976780ac68d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kjl6k"
Sep 11 18:53:51.555961 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:51.555935    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db1ed3da-67dc-4a03-a4e3-5976780ac68d-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-kjl6k\" (UID: \"db1ed3da-67dc-4a03-a4e3-5976780ac68d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kjl6k"
Sep 11 18:53:51.556078 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:51.555967    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db1ed3da-67dc-4a03-a4e3-5976780ac68d-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-kjl6k\" (UID: \"db1ed3da-67dc-4a03-a4e3-5976780ac68d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kjl6k"
Sep 11 18:53:51.556078 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:51.555985    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db1ed3da-67dc-4a03-a4e3-5976780ac68d-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-kjl6k\" (UID: \"db1ed3da-67dc-4a03-a4e3-5976780ac68d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kjl6k"
Sep 11 18:53:51.556078 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:51.556033    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db1ed3da-67dc-4a03-a4e3-5976780ac68d-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-kjl6k\" (UID: \"db1ed3da-67dc-4a03-a4e3-5976780ac68d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kjl6k"
Sep 11 18:53:51.556078 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:51.556060    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db1ed3da-67dc-4a03-a4e3-5976780ac68d-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-kjl6k\" (UID: \"db1ed3da-67dc-4a03-a4e3-5976780ac68d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kjl6k"
Sep 11 18:53:51.556078 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:51.556075    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db1ed3da-67dc-4a03-a4e3-5976780ac68d-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-kjl6k\" (UID: \"db1ed3da-67dc-4a03-a4e3-5976780ac68d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kjl6k"
Sep 11 18:53:51.657094 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:51.657060    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db1ed3da-67dc-4a03-a4e3-5976780ac68d-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-kjl6k\" (UID: \"db1ed3da-67dc-4a03-a4e3-5976780ac68d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kjl6k"
Sep 11 18:53:51.657252 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:51.657113    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db1ed3da-67dc-4a03-a4e3-5976780ac68d-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-kjl6k\" (UID: \"db1ed3da-67dc-4a03-a4e3-5976780ac68d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kjl6k"
Sep 11 18:53:51.657252 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:51.657142    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db1ed3da-67dc-4a03-a4e3-5976780ac68d-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-kjl6k\" (UID: \"db1ed3da-67dc-4a03-a4e3-5976780ac68d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kjl6k"
Sep 11 18:53:51.657252 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:51.657159    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db1ed3da-67dc-4a03-a4e3-5976780ac68d-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-kjl6k\" (UID: \"db1ed3da-67dc-4a03-a4e3-5976780ac68d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kjl6k"
Sep 11 18:53:51.657252 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:51.657180    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db1ed3da-67dc-4a03-a4e3-5976780ac68d-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-kjl6k\" (UID: \"db1ed3da-67dc-4a03-a4e3-5976780ac68d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kjl6k"
Sep 11 18:53:51.657252 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:51.657207    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db1ed3da-67dc-4a03-a4e3-5976780ac68d-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-kjl6k\" (UID: \"db1ed3da-67dc-4a03-a4e3-5976780ac68d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kjl6k"
Sep 11 18:53:51.657252 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:51.657231    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db1ed3da-67dc-4a03-a4e3-5976780ac68d-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-kjl6k\" (UID: \"db1ed3da-67dc-4a03-a4e3-5976780ac68d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kjl6k"
Sep 11 18:53:51.657508 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:51.657265    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzbz4\" (UniqueName: \"kubernetes.io/projected/db1ed3da-67dc-4a03-a4e3-5976780ac68d-kube-api-access-qzbz4\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-kjl6k\" (UID: \"db1ed3da-67dc-4a03-a4e3-5976780ac68d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kjl6k"
Sep 11 18:53:51.657508 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:51.657289    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db1ed3da-67dc-4a03-a4e3-5976780ac68d-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-kjl6k\" (UID: \"db1ed3da-67dc-4a03-a4e3-5976780ac68d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kjl6k"
Sep 11 18:53:51.657637 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:51.657566    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db1ed3da-67dc-4a03-a4e3-5976780ac68d-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-kjl6k\" (UID: \"db1ed3da-67dc-4a03-a4e3-5976780ac68d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kjl6k"
Sep 11 18:53:51.657697 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:51.657669    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db1ed3da-67dc-4a03-a4e3-5976780ac68d-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-kjl6k\" (UID: \"db1ed3da-67dc-4a03-a4e3-5976780ac68d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kjl6k"
Sep 11 18:53:51.657807 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:51.657782    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db1ed3da-67dc-4a03-a4e3-5976780ac68d-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-kjl6k\" (UID: \"db1ed3da-67dc-4a03-a4e3-5976780ac68d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kjl6k"
Sep 11 18:53:51.658091 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:51.657831    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db1ed3da-67dc-4a03-a4e3-5976780ac68d-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-kjl6k\" (UID: \"db1ed3da-67dc-4a03-a4e3-5976780ac68d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kjl6k"
Sep 11 18:53:51.658160 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:51.658094    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db1ed3da-67dc-4a03-a4e3-5976780ac68d-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-kjl6k\" (UID: \"db1ed3da-67dc-4a03-a4e3-5976780ac68d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kjl6k"
Sep 11 18:53:51.659580 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:51.659559    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db1ed3da-67dc-4a03-a4e3-5976780ac68d-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-kjl6k\" (UID: \"db1ed3da-67dc-4a03-a4e3-5976780ac68d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kjl6k"
Sep 11 18:53:51.659812 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:51.659793    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db1ed3da-67dc-4a03-a4e3-5976780ac68d-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-kjl6k\" (UID: \"db1ed3da-67dc-4a03-a4e3-5976780ac68d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kjl6k"
Sep 11 18:53:51.667341 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:51.667314    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzbz4\" (UniqueName: \"kubernetes.io/projected/db1ed3da-67dc-4a03-a4e3-5976780ac68d-kube-api-access-qzbz4\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-kjl6k\" (UID: \"db1ed3da-67dc-4a03-a4e3-5976780ac68d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kjl6k"
Sep 11 18:53:51.667551 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:51.667532    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db1ed3da-67dc-4a03-a4e3-5976780ac68d-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-kjl6k\" (UID: \"db1ed3da-67dc-4a03-a4e3-5976780ac68d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kjl6k"
Sep 11 18:53:51.742331 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:51.742307    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kjl6k"
Sep 11 18:53:52.019536 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:52.019514    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kjl6k"]
Sep 11 18:53:52.021904 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:53:52.021883    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb1ed3da_67dc_4a03_a4e3_5976780ac68d.slice/crio-75279997caad06194c3339eb17a02aeba4b54ccb43b4dc6b03eabd7ddfa1665e WatchSource:0}: Error finding container 75279997caad06194c3339eb17a02aeba4b54ccb43b4dc6b03eabd7ddfa1665e: Status 404 returned error can't find the container with id 75279997caad06194c3339eb17a02aeba4b54ccb43b4dc6b03eabd7ddfa1665e
Sep 11 18:53:52.023936 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:52.023920    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 18:53:52.582129 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:52.582091    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kjl6k" event={"ID":"db1ed3da-67dc-4a03-a4e3-5976780ac68d","Type":"ContainerStarted","Data":"75279997caad06194c3339eb17a02aeba4b54ccb43b4dc6b03eabd7ddfa1665e"}
Sep 11 18:53:54.136089 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:54.136053    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-7qmhn"]
Sep 11 18:53:54.142943 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:54.142903    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-7qmhn"
Sep 11 18:53:54.145390 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:54.145365    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Sep 11 18:53:54.145747 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:54.145732    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-pbh2f\""
Sep 11 18:53:54.151301 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:54.151245    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-7qmhn"]
Sep 11 18:53:54.282911 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:54.282881    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f15ed915-2f6f-4809-bf86-8cf905cca8f2-cert\") pod \"odh-model-controller-7998c67dc4-7qmhn\" (UID: \"f15ed915-2f6f-4809-bf86-8cf905cca8f2\") " pod="opendatahub/odh-model-controller-7998c67dc4-7qmhn"
Sep 11 18:53:54.283071 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:54.282925    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zvn7\" (UniqueName: \"kubernetes.io/projected/f15ed915-2f6f-4809-bf86-8cf905cca8f2-kube-api-access-8zvn7\") pod \"odh-model-controller-7998c67dc4-7qmhn\" (UID: \"f15ed915-2f6f-4809-bf86-8cf905cca8f2\") " pod="opendatahub/odh-model-controller-7998c67dc4-7qmhn"
Sep 11 18:53:54.383957 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:54.383927    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f15ed915-2f6f-4809-bf86-8cf905cca8f2-cert\") pod \"odh-model-controller-7998c67dc4-7qmhn\" (UID: \"f15ed915-2f6f-4809-bf86-8cf905cca8f2\") " pod="opendatahub/odh-model-controller-7998c67dc4-7qmhn"
Sep 11 18:53:54.384096 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:54.383969    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zvn7\" (UniqueName: \"kubernetes.io/projected/f15ed915-2f6f-4809-bf86-8cf905cca8f2-kube-api-access-8zvn7\") pod \"odh-model-controller-7998c67dc4-7qmhn\" (UID: \"f15ed915-2f6f-4809-bf86-8cf905cca8f2\") " pod="opendatahub/odh-model-controller-7998c67dc4-7qmhn"
Sep 11 18:53:54.386387 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:54.386337    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f15ed915-2f6f-4809-bf86-8cf905cca8f2-cert\") pod \"odh-model-controller-7998c67dc4-7qmhn\" (UID: \"f15ed915-2f6f-4809-bf86-8cf905cca8f2\") " pod="opendatahub/odh-model-controller-7998c67dc4-7qmhn"
Sep 11 18:53:54.393309 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:54.393286    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zvn7\" (UniqueName: \"kubernetes.io/projected/f15ed915-2f6f-4809-bf86-8cf905cca8f2-kube-api-access-8zvn7\") pod \"odh-model-controller-7998c67dc4-7qmhn\" (UID: \"f15ed915-2f6f-4809-bf86-8cf905cca8f2\") " pod="opendatahub/odh-model-controller-7998c67dc4-7qmhn"
Sep 11 18:53:54.445029 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:54.444988    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 18:53:54.445128 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:54.445062    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 18:53:54.445128 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:54.445095    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 18:53:54.458904 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:54.458883    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-7qmhn"
Sep 11 18:53:54.592752 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:54.592714    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kjl6k" event={"ID":"db1ed3da-67dc-4a03-a4e3-5976780ac68d","Type":"ContainerStarted","Data":"aa9321ec5cfa6e8d5c058ecd69713704d34ce6d7fa874d9d3fe0fd37b61c8863"}
Sep 11 18:53:54.630611 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:54.630544    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kjl6k" podStartSLOduration=1.209817163 podStartE2EDuration="3.630523168s" totalImagesPullingTime="2.420706005s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:53:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:53:54.628526581 +0000 UTC m=+310.706117147" watchObservedRunningTime="2026-09-11 18:53:54.630523168 +0000 UTC m=+310.708113723"
Sep 11 18:53:54.735501 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:54.735472    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-7qmhn"]
Sep 11 18:53:54.737896 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:53:54.737867    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf15ed915_2f6f_4809_bf86_8cf905cca8f2.slice/crio-9d703b15a30f95c143497addab75e01dc0c2d287c28c8ba757243ac862714d5b WatchSource:0}: Error finding container 9d703b15a30f95c143497addab75e01dc0c2d287c28c8ba757243ac862714d5b: Status 404 returned error can't find the container with id 9d703b15a30f95c143497addab75e01dc0c2d287c28c8ba757243ac862714d5b
Sep 11 18:53:54.742395 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:54.742370    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kjl6k"
Sep 11 18:53:54.752876 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:54.752855    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-8b2g4"]
Sep 11 18:53:54.756330 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:54.756314    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-8b2g4"
Sep 11 18:53:54.758455 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:54.758434    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Sep 11 18:53:54.765502 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:54.765479    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-8b2g4"]
Sep 11 18:53:54.888542 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:54.888507    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7f755f2-ae41-42eb-80ee-5024178bf0b4-tls-certs\") pod \"model-serving-api-54d89bd5fc-8b2g4\" (UID: \"c7f755f2-ae41-42eb-80ee-5024178bf0b4\") " pod="opendatahub/model-serving-api-54d89bd5fc-8b2g4"
Sep 11 18:53:54.888542 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:54.888545    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47hsb\" (UniqueName: \"kubernetes.io/projected/c7f755f2-ae41-42eb-80ee-5024178bf0b4-kube-api-access-47hsb\") pod \"model-serving-api-54d89bd5fc-8b2g4\" (UID: \"c7f755f2-ae41-42eb-80ee-5024178bf0b4\") " pod="opendatahub/model-serving-api-54d89bd5fc-8b2g4"
Sep 11 18:53:54.989393 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:54.989316    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7f755f2-ae41-42eb-80ee-5024178bf0b4-tls-certs\") pod \"model-serving-api-54d89bd5fc-8b2g4\" (UID: \"c7f755f2-ae41-42eb-80ee-5024178bf0b4\") " pod="opendatahub/model-serving-api-54d89bd5fc-8b2g4"
Sep 11 18:53:54.989393 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:54.989353    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47hsb\" (UniqueName: \"kubernetes.io/projected/c7f755f2-ae41-42eb-80ee-5024178bf0b4-kube-api-access-47hsb\") pod \"model-serving-api-54d89bd5fc-8b2g4\" (UID: \"c7f755f2-ae41-42eb-80ee-5024178bf0b4\") " pod="opendatahub/model-serving-api-54d89bd5fc-8b2g4"
Sep 11 18:53:54.991781 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:54.991761    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7f755f2-ae41-42eb-80ee-5024178bf0b4-tls-certs\") pod \"model-serving-api-54d89bd5fc-8b2g4\" (UID: \"c7f755f2-ae41-42eb-80ee-5024178bf0b4\") " pod="opendatahub/model-serving-api-54d89bd5fc-8b2g4"
Sep 11 18:53:54.997983 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:54.997959    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47hsb\" (UniqueName: \"kubernetes.io/projected/c7f755f2-ae41-42eb-80ee-5024178bf0b4-kube-api-access-47hsb\") pod \"model-serving-api-54d89bd5fc-8b2g4\" (UID: \"c7f755f2-ae41-42eb-80ee-5024178bf0b4\") " pod="opendatahub/model-serving-api-54d89bd5fc-8b2g4"
Sep 11 18:53:55.067432 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:55.067409    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-8b2g4"
Sep 11 18:53:55.346065 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:55.345981    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-8b2g4"]
Sep 11 18:53:55.349872 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:53:55.349841    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7f755f2_ae41_42eb_80ee_5024178bf0b4.slice/crio-25d2142159834a6742a00a7e1c1e42e26dfb8709b82965edb17a819e5ee2eba1 WatchSource:0}: Error finding container 25d2142159834a6742a00a7e1c1e42e26dfb8709b82965edb17a819e5ee2eba1: Status 404 returned error can't find the container with id 25d2142159834a6742a00a7e1c1e42e26dfb8709b82965edb17a819e5ee2eba1
Sep 11 18:53:55.599509 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:55.599374    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-8b2g4" event={"ID":"c7f755f2-ae41-42eb-80ee-5024178bf0b4","Type":"ContainerStarted","Data":"25d2142159834a6742a00a7e1c1e42e26dfb8709b82965edb17a819e5ee2eba1"}
Sep 11 18:53:55.601203 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:55.601174    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-7qmhn" event={"ID":"f15ed915-2f6f-4809-bf86-8cf905cca8f2","Type":"ContainerStarted","Data":"9d703b15a30f95c143497addab75e01dc0c2d287c28c8ba757243ac862714d5b"}
Sep 11 18:53:55.749166 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:55.749140    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kjl6k"
Sep 11 18:53:56.606066 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:56.606031    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kjl6k"
Sep 11 18:53:56.607198 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:56.607174    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kjl6k"
Sep 11 18:53:58.615609 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:58.615568    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-7qmhn" event={"ID":"f15ed915-2f6f-4809-bf86-8cf905cca8f2","Type":"ContainerStarted","Data":"b5cd12c8a14a825b3d504784bf9d8c6ceb6137f8089e22ce9e91f11407e7a995"}
Sep 11 18:53:58.616036 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:58.615668    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-7qmhn"
Sep 11 18:53:58.617226 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:58.617199    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-8b2g4" event={"ID":"c7f755f2-ae41-42eb-80ee-5024178bf0b4","Type":"ContainerStarted","Data":"26da82099161402ffff89c615fd06785ab7cd47fe8c1d313677377b23d5fb29c"}
Sep 11 18:53:58.617320 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:58.617287    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-8b2g4"
Sep 11 18:53:58.635329 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:58.635290    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-7qmhn" podStartSLOduration=1.607389751 podStartE2EDuration="4.635276643s" totalImagesPullingTime="3.027886892s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:53:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:53:58.634089729 +0000 UTC m=+314.711680298" watchObservedRunningTime="2026-09-11 18:53:58.635276643 +0000 UTC m=+314.712867198"
Sep 11 18:53:58.652964 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:53:58.652904    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-8b2g4" podStartSLOduration=2.241565352 podStartE2EDuration="4.652892222s" totalImagesPullingTime="2.41132687s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:53:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:53:58.651577172 +0000 UTC m=+314.729167748" watchObservedRunningTime="2026-09-11 18:53:58.652892222 +0000 UTC m=+314.730482787"
Sep 11 18:54:01.211995 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:54:01.211960    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-744bc9d7dd-wnrvb"]
Sep 11 18:54:01.222118 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:54:01.222090    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-744bc9d7dd-wnrvb"
Sep 11 18:54:01.228106 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:54:01.228085    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 11 18:54:01.228257 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:54:01.228130    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-mmf2p\""
Sep 11 18:54:01.228257 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:54:01.228147    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 11 18:54:01.228440 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:54:01.228424    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 11 18:54:01.230088 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:54:01.230067    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-744bc9d7dd-wnrvb"]
Sep 11 18:54:01.340419 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:54:01.340388    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/432e191f-ddf8-4565-a879-99accb3723bd-cert\") pod \"lws-controller-manager-744bc9d7dd-wnrvb\" (UID: \"432e191f-ddf8-4565-a879-99accb3723bd\") " pod="openshift-lws-operator/lws-controller-manager-744bc9d7dd-wnrvb"
Sep 11 18:54:01.340572 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:54:01.340448    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/432e191f-ddf8-4565-a879-99accb3723bd-metrics-cert\") pod \"lws-controller-manager-744bc9d7dd-wnrvb\" (UID: \"432e191f-ddf8-4565-a879-99accb3723bd\") " pod="openshift-lws-operator/lws-controller-manager-744bc9d7dd-wnrvb"
Sep 11 18:54:01.340572 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:54:01.340522    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/432e191f-ddf8-4565-a879-99accb3723bd-manager-config\") pod \"lws-controller-manager-744bc9d7dd-wnrvb\" (UID: \"432e191f-ddf8-4565-a879-99accb3723bd\") " pod="openshift-lws-operator/lws-controller-manager-744bc9d7dd-wnrvb"
Sep 11 18:54:01.340572 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:54:01.340556    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkknc\" (UniqueName: \"kubernetes.io/projected/432e191f-ddf8-4565-a879-99accb3723bd-kube-api-access-tkknc\") pod \"lws-controller-manager-744bc9d7dd-wnrvb\" (UID: \"432e191f-ddf8-4565-a879-99accb3723bd\") " pod="openshift-lws-operator/lws-controller-manager-744bc9d7dd-wnrvb"
Sep 11 18:54:01.441510 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:54:01.441472    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/432e191f-ddf8-4565-a879-99accb3723bd-metrics-cert\") pod \"lws-controller-manager-744bc9d7dd-wnrvb\" (UID: \"432e191f-ddf8-4565-a879-99accb3723bd\") " pod="openshift-lws-operator/lws-controller-manager-744bc9d7dd-wnrvb"
Sep 11 18:54:01.441510 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:54:01.441511    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/432e191f-ddf8-4565-a879-99accb3723bd-manager-config\") pod \"lws-controller-manager-744bc9d7dd-wnrvb\" (UID: \"432e191f-ddf8-4565-a879-99accb3723bd\") " pod="openshift-lws-operator/lws-controller-manager-744bc9d7dd-wnrvb"
Sep 11 18:54:01.441733 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:54:01.441533    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkknc\" (UniqueName: \"kubernetes.io/projected/432e191f-ddf8-4565-a879-99accb3723bd-kube-api-access-tkknc\") pod \"lws-controller-manager-744bc9d7dd-wnrvb\" (UID: \"432e191f-ddf8-4565-a879-99accb3723bd\") " pod="openshift-lws-operator/lws-controller-manager-744bc9d7dd-wnrvb"
Sep 11 18:54:01.441733 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:54:01.441579    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/432e191f-ddf8-4565-a879-99accb3723bd-cert\") pod \"lws-controller-manager-744bc9d7dd-wnrvb\" (UID: \"432e191f-ddf8-4565-a879-99accb3723bd\") " pod="openshift-lws-operator/lws-controller-manager-744bc9d7dd-wnrvb"
Sep 11 18:54:01.442184 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:54:01.442151    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/432e191f-ddf8-4565-a879-99accb3723bd-manager-config\") pod \"lws-controller-manager-744bc9d7dd-wnrvb\" (UID: \"432e191f-ddf8-4565-a879-99accb3723bd\") " pod="openshift-lws-operator/lws-controller-manager-744bc9d7dd-wnrvb"
Sep 11 18:54:01.444094 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:54:01.444072    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/432e191f-ddf8-4565-a879-99accb3723bd-cert\") pod \"lws-controller-manager-744bc9d7dd-wnrvb\" (UID: \"432e191f-ddf8-4565-a879-99accb3723bd\") " pod="openshift-lws-operator/lws-controller-manager-744bc9d7dd-wnrvb"
Sep 11 18:54:01.444196 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:54:01.444179    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/432e191f-ddf8-4565-a879-99accb3723bd-metrics-cert\") pod \"lws-controller-manager-744bc9d7dd-wnrvb\" (UID: \"432e191f-ddf8-4565-a879-99accb3723bd\") " pod="openshift-lws-operator/lws-controller-manager-744bc9d7dd-wnrvb"
Sep 11 18:54:01.456438 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:54:01.456415    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkknc\" (UniqueName: \"kubernetes.io/projected/432e191f-ddf8-4565-a879-99accb3723bd-kube-api-access-tkknc\") pod \"lws-controller-manager-744bc9d7dd-wnrvb\" (UID: \"432e191f-ddf8-4565-a879-99accb3723bd\") " pod="openshift-lws-operator/lws-controller-manager-744bc9d7dd-wnrvb"
Sep 11 18:54:01.531845 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:54:01.531767    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-744bc9d7dd-wnrvb"
Sep 11 18:54:01.812395 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:54:01.812318    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-744bc9d7dd-wnrvb"]
Sep 11 18:54:01.816784 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:54:01.816748    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod432e191f_ddf8_4565_a879_99accb3723bd.slice/crio-38047bdc9f62876cdeefa10e99d8a04efff7b9d4a853ef8cfb1494496b2e289a WatchSource:0}: Error finding container 38047bdc9f62876cdeefa10e99d8a04efff7b9d4a853ef8cfb1494496b2e289a: Status 404 returned error can't find the container with id 38047bdc9f62876cdeefa10e99d8a04efff7b9d4a853ef8cfb1494496b2e289a
Sep 11 18:54:02.633814 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:54:02.633782    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-744bc9d7dd-wnrvb" event={"ID":"432e191f-ddf8-4565-a879-99accb3723bd","Type":"ContainerStarted","Data":"38047bdc9f62876cdeefa10e99d8a04efff7b9d4a853ef8cfb1494496b2e289a"}
Sep 11 18:54:03.638951 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:54:03.638919    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-744bc9d7dd-wnrvb" event={"ID":"432e191f-ddf8-4565-a879-99accb3723bd","Type":"ContainerStarted","Data":"ae266074fb878069fa6eff65fd1a83b1752f217aafca8ad89f41a3e559e642d7"}
Sep 11 18:54:03.639305 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:54:03.639019    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-744bc9d7dd-wnrvb"
Sep 11 18:54:09.622807 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:54:09.622779    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-7qmhn"
Sep 11 18:54:09.624835 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:54:09.624806    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-8b2g4"
Sep 11 18:54:09.656855 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:54:09.656807    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-744bc9d7dd-wnrvb" podStartSLOduration=6.993873334 podStartE2EDuration="8.656788526s" totalImagesPullingTime="1.662915192s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:54:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:54:03.714173121 +0000 UTC m=+319.791763699" watchObservedRunningTime="2026-09-11 18:54:09.656788526 +0000 UTC m=+325.734379082"
Sep 11 18:54:14.645499 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:54:14.645470    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-744bc9d7dd-wnrvb"
Sep 11 18:56:29.220061 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:56:29.220029    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-mm6s7"]
Sep 11 18:56:29.223162 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:56:29.223128    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-mm6s7"
Sep 11 18:56:29.226303 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:56:29.225548    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 11 18:56:29.226303 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:56:29.225869    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 11 18:56:29.226303 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:56:29.226186    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-6x5h8\""
Sep 11 18:56:29.232787 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:56:29.232757    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-mm6s7"]
Sep 11 18:56:29.357040 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:56:29.357017    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-j8zzz"]
Sep 11 18:56:29.360023 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:56:29.360007    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-j8zzz"
Sep 11 18:56:29.366862 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:56:29.366840    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-j8zzz"]
Sep 11 18:56:29.377402 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:56:29.377382    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4trv\" (UniqueName: \"kubernetes.io/projected/084e9faa-3c2f-44a3-9f2a-e5ed1ef6b388-kube-api-access-l4trv\") pod \"authorino-7bcfd8f9cd-mm6s7\" (UID: \"084e9faa-3c2f-44a3-9f2a-e5ed1ef6b388\") " pod="kuadrant-system/authorino-7bcfd8f9cd-mm6s7"
Sep 11 18:56:29.478456 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:56:29.478390    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cslh\" (UniqueName: \"kubernetes.io/projected/2bdbe663-58e7-49a1-b4e0-cb5a8c581828-kube-api-access-6cslh\") pod \"authorino-59c974bd48-j8zzz\" (UID: \"2bdbe663-58e7-49a1-b4e0-cb5a8c581828\") " pod="kuadrant-system/authorino-59c974bd48-j8zzz"
Sep 11 18:56:29.478456 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:56:29.478432    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4trv\" (UniqueName: \"kubernetes.io/projected/084e9faa-3c2f-44a3-9f2a-e5ed1ef6b388-kube-api-access-l4trv\") pod \"authorino-7bcfd8f9cd-mm6s7\" (UID: \"084e9faa-3c2f-44a3-9f2a-e5ed1ef6b388\") " pod="kuadrant-system/authorino-7bcfd8f9cd-mm6s7"
Sep 11 18:56:29.494154 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:56:29.494134    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4trv\" (UniqueName: \"kubernetes.io/projected/084e9faa-3c2f-44a3-9f2a-e5ed1ef6b388-kube-api-access-l4trv\") pod \"authorino-7bcfd8f9cd-mm6s7\" (UID: \"084e9faa-3c2f-44a3-9f2a-e5ed1ef6b388\") " pod="kuadrant-system/authorino-7bcfd8f9cd-mm6s7"
Sep 11 18:56:29.538656 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:56:29.538625    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-mm6s7"
Sep 11 18:56:29.578845 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:56:29.578807    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cslh\" (UniqueName: \"kubernetes.io/projected/2bdbe663-58e7-49a1-b4e0-cb5a8c581828-kube-api-access-6cslh\") pod \"authorino-59c974bd48-j8zzz\" (UID: \"2bdbe663-58e7-49a1-b4e0-cb5a8c581828\") " pod="kuadrant-system/authorino-59c974bd48-j8zzz"
Sep 11 18:56:29.588096 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:56:29.588065    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cslh\" (UniqueName: \"kubernetes.io/projected/2bdbe663-58e7-49a1-b4e0-cb5a8c581828-kube-api-access-6cslh\") pod \"authorino-59c974bd48-j8zzz\" (UID: \"2bdbe663-58e7-49a1-b4e0-cb5a8c581828\") " pod="kuadrant-system/authorino-59c974bd48-j8zzz"
Sep 11 18:56:29.669142 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:56:29.669096    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-j8zzz"
Sep 11 18:56:29.805019 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:56:29.804954    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-mm6s7"]
Sep 11 18:56:29.808293 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:56:29.808263    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod084e9faa_3c2f_44a3_9f2a_e5ed1ef6b388.slice/crio-a40b222d2ee4a66a0472e6bd4055efbd4f59b1bd531262b8053215664e48207e WatchSource:0}: Error finding container a40b222d2ee4a66a0472e6bd4055efbd4f59b1bd531262b8053215664e48207e: Status 404 returned error can't find the container with id a40b222d2ee4a66a0472e6bd4055efbd4f59b1bd531262b8053215664e48207e
Sep 11 18:56:29.935992 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:56:29.935958    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-j8zzz"]
Sep 11 18:56:29.938695 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:56:29.938667    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2bdbe663_58e7_49a1_b4e0_cb5a8c581828.slice/crio-159538790ea26352e94083d56487d85a639ad230397fa33eeed3abc1d1517c50 WatchSource:0}: Error finding container 159538790ea26352e94083d56487d85a639ad230397fa33eeed3abc1d1517c50: Status 404 returned error can't find the container with id 159538790ea26352e94083d56487d85a639ad230397fa33eeed3abc1d1517c50
Sep 11 18:56:30.240791 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:56:30.240746    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-j8zzz" event={"ID":"2bdbe663-58e7-49a1-b4e0-cb5a8c581828","Type":"ContainerStarted","Data":"159538790ea26352e94083d56487d85a639ad230397fa33eeed3abc1d1517c50"}
Sep 11 18:56:30.244647 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:56:30.244621    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-mm6s7" event={"ID":"084e9faa-3c2f-44a3-9f2a-e5ed1ef6b388","Type":"ContainerStarted","Data":"a40b222d2ee4a66a0472e6bd4055efbd4f59b1bd531262b8053215664e48207e"}
Sep 11 18:56:33.262842 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:56:33.262800    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-j8zzz" event={"ID":"2bdbe663-58e7-49a1-b4e0-cb5a8c581828","Type":"ContainerStarted","Data":"2bd7a3e91f26ebbe1ff69cf19f1fcefd6d0e6a18d55943a09e547480e64ddb43"}
Sep 11 18:56:33.264322 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:56:33.264300    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-mm6s7" event={"ID":"084e9faa-3c2f-44a3-9f2a-e5ed1ef6b388","Type":"ContainerStarted","Data":"ab962124f3a5081ac7c16891315f5182b4e4bbb053ff1f72bfdea3462ac6ac07"}
Sep 11 18:56:33.278618 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:56:33.278556    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-j8zzz" podStartSLOduration=1.255314188 podStartE2EDuration="4.278542426s" totalImagesPullingTime="3.023228238s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:56:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:56:33.278049431 +0000 UTC m=+469.355639990" watchObservedRunningTime="2026-09-11 18:56:33.278542426 +0000 UTC m=+469.356132981"
Sep 11 18:56:33.292216 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:56:33.292175    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-mm6s7" podStartSLOduration=1.148705701 podStartE2EDuration="4.292159462s" totalImagesPullingTime="3.143453761s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:56:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:56:33.291118338 +0000 UTC m=+469.368708890" watchObservedRunningTime="2026-09-11 18:56:33.292159462 +0000 UTC m=+469.369750017"
Sep 11 18:56:33.304652 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:56:33.304631    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-mm6s7"]
Sep 11 18:56:35.273173 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:56:35.273134    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-mm6s7" podUID="084e9faa-3c2f-44a3-9f2a-e5ed1ef6b388" containerName="authorino" containerID="cri-o://ab962124f3a5081ac7c16891315f5182b4e4bbb053ff1f72bfdea3462ac6ac07" gracePeriod=30
Sep 11 18:56:35.510969 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:56:35.510941    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-mm6s7"
Sep 11 18:56:35.625469 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:56:35.625401    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l4trv\" (UniqueName: \"kubernetes.io/projected/084e9faa-3c2f-44a3-9f2a-e5ed1ef6b388-kube-api-access-l4trv\") pod \"084e9faa-3c2f-44a3-9f2a-e5ed1ef6b388\" (UID: \"084e9faa-3c2f-44a3-9f2a-e5ed1ef6b388\") "
Sep 11 18:56:35.627640 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:56:35.627615    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/084e9faa-3c2f-44a3-9f2a-e5ed1ef6b388-kube-api-access-l4trv" (OuterVolumeSpecName: "kube-api-access-l4trv") pod "084e9faa-3c2f-44a3-9f2a-e5ed1ef6b388" (UID: "084e9faa-3c2f-44a3-9f2a-e5ed1ef6b388"). InnerVolumeSpecName "kube-api-access-l4trv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:56:35.725979 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:56:35.725934    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4trv\" (UniqueName: \"kubernetes.io/projected/084e9faa-3c2f-44a3-9f2a-e5ed1ef6b388-kube-api-access-l4trv\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 18:56:36.278162 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:56:36.278126    2578 generic.go:356] "Generic (PLEG): container finished" podID="084e9faa-3c2f-44a3-9f2a-e5ed1ef6b388" containerID="ab962124f3a5081ac7c16891315f5182b4e4bbb053ff1f72bfdea3462ac6ac07" exitCode=0
Sep 11 18:56:36.278162 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:56:36.278164    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-mm6s7"
Sep 11 18:56:36.278619 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:56:36.278213    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-mm6s7" event={"ID":"084e9faa-3c2f-44a3-9f2a-e5ed1ef6b388","Type":"ContainerDied","Data":"ab962124f3a5081ac7c16891315f5182b4e4bbb053ff1f72bfdea3462ac6ac07"}
Sep 11 18:56:36.278619 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:56:36.278248    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-mm6s7" event={"ID":"084e9faa-3c2f-44a3-9f2a-e5ed1ef6b388","Type":"ContainerDied","Data":"a40b222d2ee4a66a0472e6bd4055efbd4f59b1bd531262b8053215664e48207e"}
Sep 11 18:56:36.278619 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:56:36.278267    2578 scope.go:117] "RemoveContainer" containerID="ab962124f3a5081ac7c16891315f5182b4e4bbb053ff1f72bfdea3462ac6ac07"
Sep 11 18:56:36.286818 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:56:36.286803    2578 scope.go:117] "RemoveContainer" containerID="ab962124f3a5081ac7c16891315f5182b4e4bbb053ff1f72bfdea3462ac6ac07"
Sep 11 18:56:36.287080 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:56:36.287060    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab962124f3a5081ac7c16891315f5182b4e4bbb053ff1f72bfdea3462ac6ac07\": container with ID starting with ab962124f3a5081ac7c16891315f5182b4e4bbb053ff1f72bfdea3462ac6ac07 not found: ID does not exist" containerID="ab962124f3a5081ac7c16891315f5182b4e4bbb053ff1f72bfdea3462ac6ac07"
Sep 11 18:56:36.287129 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:56:36.287087    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab962124f3a5081ac7c16891315f5182b4e4bbb053ff1f72bfdea3462ac6ac07"} err="failed to get container status \"ab962124f3a5081ac7c16891315f5182b4e4bbb053ff1f72bfdea3462ac6ac07\": rpc error: code = NotFound desc = could not find container \"ab962124f3a5081ac7c16891315f5182b4e4bbb053ff1f72bfdea3462ac6ac07\": container with ID starting with ab962124f3a5081ac7c16891315f5182b4e4bbb053ff1f72bfdea3462ac6ac07 not found: ID does not exist"
Sep 11 18:56:36.297852 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:56:36.297827    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-mm6s7"]
Sep 11 18:56:36.300308 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:56:36.300287    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-mm6s7"]
Sep 11 18:56:36.395556 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:56:36.395520    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="084e9faa-3c2f-44a3-9f2a-e5ed1ef6b388" path="/var/lib/kubelet/pods/084e9faa-3c2f-44a3-9f2a-e5ed1ef6b388/volumes"
Sep 11 18:56:58.060583 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:56:58.060546    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-dznv5"]
Sep 11 18:56:58.061050 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:56:58.060915    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="084e9faa-3c2f-44a3-9f2a-e5ed1ef6b388" containerName="authorino"
Sep 11 18:56:58.061050 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:56:58.060927    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="084e9faa-3c2f-44a3-9f2a-e5ed1ef6b388" containerName="authorino"
Sep 11 18:56:58.061050 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:56:58.060998    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="084e9faa-3c2f-44a3-9f2a-e5ed1ef6b388" containerName="authorino"
Sep 11 18:56:58.067196 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:56:58.067178    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-dznv5"
Sep 11 18:56:58.070394 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:56:58.070374    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-dznv5"]
Sep 11 18:56:58.079030 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:56:58.078998    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n69d8\" (UniqueName: \"kubernetes.io/projected/acaf3e61-d4ad-4daf-8fd2-a696d0dfc195-kube-api-access-n69d8\") pod \"authorino-557f876f69-dznv5\" (UID: \"acaf3e61-d4ad-4daf-8fd2-a696d0dfc195\") " pod="kuadrant-system/authorino-557f876f69-dznv5"
Sep 11 18:56:58.152350 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:56:58.152323    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-dznv5"]
Sep 11 18:56:58.152524 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:56:58.152506    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-n69d8], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-557f876f69-dznv5" podUID="acaf3e61-d4ad-4daf-8fd2-a696d0dfc195"
Sep 11 18:56:58.175263 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:56:58.175240    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-kbl8k"]
Sep 11 18:56:58.178825 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:56:58.178811    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-kbl8k"
Sep 11 18:56:58.179294 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:56:58.179276    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2h2b\" (UniqueName: \"kubernetes.io/projected/282da65e-1cbc-45c8-8ee2-566e7e650fd2-kube-api-access-x2h2b\") pod \"authorino-6d4dfc649f-kbl8k\" (UID: \"282da65e-1cbc-45c8-8ee2-566e7e650fd2\") " pod="kuadrant-system/authorino-6d4dfc649f-kbl8k"
Sep 11 18:56:58.179349 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:56:58.179314    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n69d8\" (UniqueName: \"kubernetes.io/projected/acaf3e61-d4ad-4daf-8fd2-a696d0dfc195-kube-api-access-n69d8\") pod \"authorino-557f876f69-dznv5\" (UID: \"acaf3e61-d4ad-4daf-8fd2-a696d0dfc195\") " pod="kuadrant-system/authorino-557f876f69-dznv5"
Sep 11 18:56:58.179388 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:56:58.179353    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/282da65e-1cbc-45c8-8ee2-566e7e650fd2-tls-cert\") pod \"authorino-6d4dfc649f-kbl8k\" (UID: \"282da65e-1cbc-45c8-8ee2-566e7e650fd2\") " pod="kuadrant-system/authorino-6d4dfc649f-kbl8k"
Sep 11 18:56:58.180712 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:56:58.180695    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 11 18:56:58.187260 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:56:58.187239    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-kbl8k"]
Sep 11 18:56:58.191336 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:56:58.191312    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n69d8\" (UniqueName: \"kubernetes.io/projected/acaf3e61-d4ad-4daf-8fd2-a696d0dfc195-kube-api-access-n69d8\") pod \"authorino-557f876f69-dznv5\" (UID: \"acaf3e61-d4ad-4daf-8fd2-a696d0dfc195\") " pod="kuadrant-system/authorino-557f876f69-dznv5"
Sep 11 18:56:58.279802 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:56:58.279771    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2h2b\" (UniqueName: \"kubernetes.io/projected/282da65e-1cbc-45c8-8ee2-566e7e650fd2-kube-api-access-x2h2b\") pod \"authorino-6d4dfc649f-kbl8k\" (UID: \"282da65e-1cbc-45c8-8ee2-566e7e650fd2\") " pod="kuadrant-system/authorino-6d4dfc649f-kbl8k"
Sep 11 18:56:58.279932 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:56:58.279843    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/282da65e-1cbc-45c8-8ee2-566e7e650fd2-tls-cert\") pod \"authorino-6d4dfc649f-kbl8k\" (UID: \"282da65e-1cbc-45c8-8ee2-566e7e650fd2\") " pod="kuadrant-system/authorino-6d4dfc649f-kbl8k"
Sep 11 18:56:58.282288 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:56:58.282268    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/282da65e-1cbc-45c8-8ee2-566e7e650fd2-tls-cert\") pod \"authorino-6d4dfc649f-kbl8k\" (UID: \"282da65e-1cbc-45c8-8ee2-566e7e650fd2\") " pod="kuadrant-system/authorino-6d4dfc649f-kbl8k"
Sep 11 18:56:58.288732 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:56:58.288711    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-kbl8k"]
Sep 11 18:56:58.288934 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:56:58.288915    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-x2h2b], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-kbl8k" podUID="282da65e-1cbc-45c8-8ee2-566e7e650fd2"
Sep 11 18:56:58.291539 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:56:58.291519    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2h2b\" (UniqueName: \"kubernetes.io/projected/282da65e-1cbc-45c8-8ee2-566e7e650fd2-kube-api-access-x2h2b\") pod \"authorino-6d4dfc649f-kbl8k\" (UID: \"282da65e-1cbc-45c8-8ee2-566e7e650fd2\") " pod="kuadrant-system/authorino-6d4dfc649f-kbl8k"
Sep 11 18:56:58.367673 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:56:58.367565    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-dznv5"
Sep 11 18:56:58.367854 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:56:58.367565    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-kbl8k"
Sep 11 18:56:58.376513 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:56:58.376486    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-dznv5"
Sep 11 18:56:58.379586 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:56:58.379564    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-kbl8k"
Sep 11 18:56:58.380759 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:56:58.380738    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x2h2b\" (UniqueName: \"kubernetes.io/projected/282da65e-1cbc-45c8-8ee2-566e7e650fd2-kube-api-access-x2h2b\") pod \"282da65e-1cbc-45c8-8ee2-566e7e650fd2\" (UID: \"282da65e-1cbc-45c8-8ee2-566e7e650fd2\") "
Sep 11 18:56:58.380846 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:56:58.380776    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/282da65e-1cbc-45c8-8ee2-566e7e650fd2-tls-cert\") pod \"282da65e-1cbc-45c8-8ee2-566e7e650fd2\" (UID: \"282da65e-1cbc-45c8-8ee2-566e7e650fd2\") "
Sep 11 18:56:58.380846 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:56:58.380826    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n69d8\" (UniqueName: \"kubernetes.io/projected/acaf3e61-d4ad-4daf-8fd2-a696d0dfc195-kube-api-access-n69d8\") pod \"acaf3e61-d4ad-4daf-8fd2-a696d0dfc195\" (UID: \"acaf3e61-d4ad-4daf-8fd2-a696d0dfc195\") "
Sep 11 18:56:58.383001 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:56:58.382969    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acaf3e61-d4ad-4daf-8fd2-a696d0dfc195-kube-api-access-n69d8" (OuterVolumeSpecName: "kube-api-access-n69d8") pod "acaf3e61-d4ad-4daf-8fd2-a696d0dfc195" (UID: "acaf3e61-d4ad-4daf-8fd2-a696d0dfc195"). InnerVolumeSpecName "kube-api-access-n69d8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:56:58.383082 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:56:58.383012    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/282da65e-1cbc-45c8-8ee2-566e7e650fd2-kube-api-access-x2h2b" (OuterVolumeSpecName: "kube-api-access-x2h2b") pod "282da65e-1cbc-45c8-8ee2-566e7e650fd2" (UID: "282da65e-1cbc-45c8-8ee2-566e7e650fd2"). InnerVolumeSpecName "kube-api-access-x2h2b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:56:58.383082 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:56:58.383048    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/282da65e-1cbc-45c8-8ee2-566e7e650fd2-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "282da65e-1cbc-45c8-8ee2-566e7e650fd2" (UID: "282da65e-1cbc-45c8-8ee2-566e7e650fd2"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:56:58.481783 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:56:58.481755    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n69d8\" (UniqueName: \"kubernetes.io/projected/acaf3e61-d4ad-4daf-8fd2-a696d0dfc195-kube-api-access-n69d8\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 18:56:58.481783 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:56:58.481778    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2h2b\" (UniqueName: \"kubernetes.io/projected/282da65e-1cbc-45c8-8ee2-566e7e650fd2-kube-api-access-x2h2b\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 18:56:58.481783 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:56:58.481787    2578 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/282da65e-1cbc-45c8-8ee2-566e7e650fd2-tls-cert\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 18:56:59.375672 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:56:59.375639    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-kbl8k"
Sep 11 18:56:59.376027 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:56:59.375638    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-dznv5"
Sep 11 18:56:59.400798 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:56:59.400774    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-dznv5"]
Sep 11 18:56:59.404474 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:56:59.404450    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-dznv5"]
Sep 11 18:56:59.423745 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:56:59.423722    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-kbl8k"]
Sep 11 18:56:59.427131 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:56:59.427111    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-kbl8k"]
Sep 11 18:57:00.395731 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:57:00.395701    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="282da65e-1cbc-45c8-8ee2-566e7e650fd2" path="/var/lib/kubelet/pods/282da65e-1cbc-45c8-8ee2-566e7e650fd2/volumes"
Sep 11 18:57:00.396105 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:57:00.395917    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="acaf3e61-d4ad-4daf-8fd2-a696d0dfc195" path="/var/lib/kubelet/pods/acaf3e61-d4ad-4daf-8fd2-a696d0dfc195/volumes"
Sep 11 18:57:00.683559 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:57:00.683490    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-j8zzz"]
Sep 11 18:57:00.683711 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:57:00.683680    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-j8zzz" podUID="2bdbe663-58e7-49a1-b4e0-cb5a8c581828" containerName="authorino" containerID="cri-o://2bd7a3e91f26ebbe1ff69cf19f1fcefd6d0e6a18d55943a09e547480e64ddb43" gracePeriod=30
Sep 11 18:57:00.955206 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:57:00.955186    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-j8zzz"
Sep 11 18:57:00.998195 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:57:00.998170    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6cslh\" (UniqueName: \"kubernetes.io/projected/2bdbe663-58e7-49a1-b4e0-cb5a8c581828-kube-api-access-6cslh\") pod \"2bdbe663-58e7-49a1-b4e0-cb5a8c581828\" (UID: \"2bdbe663-58e7-49a1-b4e0-cb5a8c581828\") "
Sep 11 18:57:01.000318 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:57:01.000294    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bdbe663-58e7-49a1-b4e0-cb5a8c581828-kube-api-access-6cslh" (OuterVolumeSpecName: "kube-api-access-6cslh") pod "2bdbe663-58e7-49a1-b4e0-cb5a8c581828" (UID: "2bdbe663-58e7-49a1-b4e0-cb5a8c581828"). InnerVolumeSpecName "kube-api-access-6cslh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:57:01.098930 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:57:01.098895    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cslh\" (UniqueName: \"kubernetes.io/projected/2bdbe663-58e7-49a1-b4e0-cb5a8c581828-kube-api-access-6cslh\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 18:57:01.384771 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:57:01.384683    2578 generic.go:356] "Generic (PLEG): container finished" podID="2bdbe663-58e7-49a1-b4e0-cb5a8c581828" containerID="2bd7a3e91f26ebbe1ff69cf19f1fcefd6d0e6a18d55943a09e547480e64ddb43" exitCode=0
Sep 11 18:57:01.384771 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:57:01.384722    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-j8zzz"
Sep 11 18:57:01.384994 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:57:01.384724    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-j8zzz" event={"ID":"2bdbe663-58e7-49a1-b4e0-cb5a8c581828","Type":"ContainerDied","Data":"2bd7a3e91f26ebbe1ff69cf19f1fcefd6d0e6a18d55943a09e547480e64ddb43"}
Sep 11 18:57:01.384994 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:57:01.384862    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-j8zzz" event={"ID":"2bdbe663-58e7-49a1-b4e0-cb5a8c581828","Type":"ContainerDied","Data":"159538790ea26352e94083d56487d85a639ad230397fa33eeed3abc1d1517c50"}
Sep 11 18:57:01.384994 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:57:01.384892    2578 scope.go:117] "RemoveContainer" containerID="2bd7a3e91f26ebbe1ff69cf19f1fcefd6d0e6a18d55943a09e547480e64ddb43"
Sep 11 18:57:01.393484 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:57:01.393467    2578 scope.go:117] "RemoveContainer" containerID="2bd7a3e91f26ebbe1ff69cf19f1fcefd6d0e6a18d55943a09e547480e64ddb43"
Sep 11 18:57:01.393775 ip-10-0-134-189 kubenswrapper[2578]: E0911 18:57:01.393757    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bd7a3e91f26ebbe1ff69cf19f1fcefd6d0e6a18d55943a09e547480e64ddb43\": container with ID starting with 2bd7a3e91f26ebbe1ff69cf19f1fcefd6d0e6a18d55943a09e547480e64ddb43 not found: ID does not exist" containerID="2bd7a3e91f26ebbe1ff69cf19f1fcefd6d0e6a18d55943a09e547480e64ddb43"
Sep 11 18:57:01.393835 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:57:01.393783    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bd7a3e91f26ebbe1ff69cf19f1fcefd6d0e6a18d55943a09e547480e64ddb43"} err="failed to get container status \"2bd7a3e91f26ebbe1ff69cf19f1fcefd6d0e6a18d55943a09e547480e64ddb43\": rpc error: code = NotFound desc = could not find container \"2bd7a3e91f26ebbe1ff69cf19f1fcefd6d0e6a18d55943a09e547480e64ddb43\": container with ID starting with 2bd7a3e91f26ebbe1ff69cf19f1fcefd6d0e6a18d55943a09e547480e64ddb43 not found: ID does not exist"
Sep 11 18:57:01.429546 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:57:01.429522    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-j8zzz"]
Sep 11 18:57:01.432371 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:57:01.432352    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-j8zzz"]
Sep 11 18:57:02.394811 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:57:02.394781    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2bdbe663-58e7-49a1-b4e0-cb5a8c581828" path="/var/lib/kubelet/pods/2bdbe663-58e7-49a1-b4e0-cb5a8c581828/volumes"
Sep 11 18:57:43.010085 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:57:43.010013    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-pm7kk"]
Sep 11 18:57:43.010525 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:57:43.010345    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2bdbe663-58e7-49a1-b4e0-cb5a8c581828" containerName="authorino"
Sep 11 18:57:43.010525 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:57:43.010358    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2bdbe663-58e7-49a1-b4e0-cb5a8c581828" containerName="authorino"
Sep 11 18:57:43.010525 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:57:43.010412    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2bdbe663-58e7-49a1-b4e0-cb5a8c581828" containerName="authorino"
Sep 11 18:57:43.012483 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:57:43.012467    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-pm7kk"
Sep 11 18:57:43.014369 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:57:43.014352    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 11 18:57:43.014441 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:57:43.014426    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-tdxv5\""
Sep 11 18:57:43.018409 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:57:43.018383    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-pm7kk"]
Sep 11 18:57:43.121060 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:57:43.121031    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1edf23d3-5888-4b32-9809-066efe3155c6-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-pm7kk\" (UID: \"1edf23d3-5888-4b32-9809-066efe3155c6\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-pm7kk"
Sep 11 18:57:43.121190 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:57:43.121081    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45lnr\" (UniqueName: \"kubernetes.io/projected/1edf23d3-5888-4b32-9809-066efe3155c6-kube-api-access-45lnr\") pod \"payload-processing-5b54b7fbb7-pm7kk\" (UID: \"1edf23d3-5888-4b32-9809-066efe3155c6\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-pm7kk"
Sep 11 18:57:43.222525 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:57:43.222499    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45lnr\" (UniqueName: \"kubernetes.io/projected/1edf23d3-5888-4b32-9809-066efe3155c6-kube-api-access-45lnr\") pod \"payload-processing-5b54b7fbb7-pm7kk\" (UID: \"1edf23d3-5888-4b32-9809-066efe3155c6\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-pm7kk"
Sep 11 18:57:43.222686 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:57:43.222567    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1edf23d3-5888-4b32-9809-066efe3155c6-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-pm7kk\" (UID: \"1edf23d3-5888-4b32-9809-066efe3155c6\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-pm7kk"
Sep 11 18:57:43.223241 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:57:43.223220    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1edf23d3-5888-4b32-9809-066efe3155c6-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-pm7kk\" (UID: \"1edf23d3-5888-4b32-9809-066efe3155c6\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-pm7kk"
Sep 11 18:57:43.230771 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:57:43.230751    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45lnr\" (UniqueName: \"kubernetes.io/projected/1edf23d3-5888-4b32-9809-066efe3155c6-kube-api-access-45lnr\") pod \"payload-processing-5b54b7fbb7-pm7kk\" (UID: \"1edf23d3-5888-4b32-9809-066efe3155c6\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-pm7kk"
Sep 11 18:57:43.323060 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:57:43.322988    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-pm7kk"
Sep 11 18:57:43.590691 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:57:43.590620    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-pm7kk"]
Sep 11 18:57:43.593713 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:57:43.593675    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1edf23d3_5888_4b32_9809_066efe3155c6.slice/crio-d7b70a61a14f81726857e8255171ba7fb8ffaea0a547330d03a5e3bd296658a5 WatchSource:0}: Error finding container d7b70a61a14f81726857e8255171ba7fb8ffaea0a547330d03a5e3bd296658a5: Status 404 returned error can't find the container with id d7b70a61a14f81726857e8255171ba7fb8ffaea0a547330d03a5e3bd296658a5
Sep 11 18:57:44.556415 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:57:44.556374    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-pm7kk" event={"ID":"1edf23d3-5888-4b32-9809-066efe3155c6","Type":"ContainerStarted","Data":"d7b70a61a14f81726857e8255171ba7fb8ffaea0a547330d03a5e3bd296658a5"}
Sep 11 18:57:46.567074 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:57:46.567032    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-pm7kk" event={"ID":"1edf23d3-5888-4b32-9809-066efe3155c6","Type":"ContainerStarted","Data":"c0191da77063f1ffbc3b4ad9bfc724e1fb62b9e319f90b179241a7a4d9640c73"}
Sep 11 18:57:46.567482 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:57:46.567137    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-pm7kk"
Sep 11 18:57:46.583312 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:57:46.583265    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-pm7kk" podStartSLOduration=2.314007562 podStartE2EDuration="4.583245463s" totalImagesPullingTime="2.269237901s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:57:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:57:46.581703256 +0000 UTC m=+542.659293827" watchObservedRunningTime="2026-09-11 18:57:46.583245463 +0000 UTC m=+542.660836019"
Sep 11 18:57:57.574154 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:57:57.574125    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-pm7kk"
Sep 11 18:58:31.028734 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:58:31.028693    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-t2k77"]
Sep 11 18:58:31.031943 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:58:31.031927    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t2k77"
Sep 11 18:58:31.034712 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:58:31.034686    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 11 18:58:31.035501 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:58:31.035452    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Sep 11 18:58:31.035501 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:58:31.035470    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-tgc84\""
Sep 11 18:58:31.035501 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:58:31.035470    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 11 18:58:31.043155 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:58:31.043133    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-t2k77"]
Sep 11 18:58:31.111288 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:58:31.111260    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45e92205-f8aa-46e3-98f2-f7d2b1e400f6-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-t2k77\" (UID: \"45e92205-f8aa-46e3-98f2-f7d2b1e400f6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t2k77"
Sep 11 18:58:31.111430 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:58:31.111319    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45e92205-f8aa-46e3-98f2-f7d2b1e400f6-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-t2k77\" (UID: \"45e92205-f8aa-46e3-98f2-f7d2b1e400f6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t2k77"
Sep 11 18:58:31.111430 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:58:31.111365    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2hsz\" (UniqueName: \"kubernetes.io/projected/45e92205-f8aa-46e3-98f2-f7d2b1e400f6-kube-api-access-n2hsz\") pod \"e2e-distinct-simulated-kserve-59975d69f9-t2k77\" (UID: \"45e92205-f8aa-46e3-98f2-f7d2b1e400f6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t2k77"
Sep 11 18:58:31.111430 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:58:31.111406    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45e92205-f8aa-46e3-98f2-f7d2b1e400f6-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-t2k77\" (UID: \"45e92205-f8aa-46e3-98f2-f7d2b1e400f6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t2k77"
Sep 11 18:58:31.111430 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:58:31.111430    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45e92205-f8aa-46e3-98f2-f7d2b1e400f6-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-t2k77\" (UID: \"45e92205-f8aa-46e3-98f2-f7d2b1e400f6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t2k77"
Sep 11 18:58:31.111566 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:58:31.111450    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45e92205-f8aa-46e3-98f2-f7d2b1e400f6-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-t2k77\" (UID: \"45e92205-f8aa-46e3-98f2-f7d2b1e400f6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t2k77"
Sep 11 18:58:31.212742 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:58:31.212710    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45e92205-f8aa-46e3-98f2-f7d2b1e400f6-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-t2k77\" (UID: \"45e92205-f8aa-46e3-98f2-f7d2b1e400f6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t2k77"
Sep 11 18:58:31.212904 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:58:31.212745    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2hsz\" (UniqueName: \"kubernetes.io/projected/45e92205-f8aa-46e3-98f2-f7d2b1e400f6-kube-api-access-n2hsz\") pod \"e2e-distinct-simulated-kserve-59975d69f9-t2k77\" (UID: \"45e92205-f8aa-46e3-98f2-f7d2b1e400f6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t2k77"
Sep 11 18:58:31.212904 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:58:31.212773    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45e92205-f8aa-46e3-98f2-f7d2b1e400f6-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-t2k77\" (UID: \"45e92205-f8aa-46e3-98f2-f7d2b1e400f6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t2k77"
Sep 11 18:58:31.212904 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:58:31.212794    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45e92205-f8aa-46e3-98f2-f7d2b1e400f6-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-t2k77\" (UID: \"45e92205-f8aa-46e3-98f2-f7d2b1e400f6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t2k77"
Sep 11 18:58:31.212904 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:58:31.212824    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45e92205-f8aa-46e3-98f2-f7d2b1e400f6-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-t2k77\" (UID: \"45e92205-f8aa-46e3-98f2-f7d2b1e400f6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t2k77"
Sep 11 18:58:31.213111 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:58:31.212943    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45e92205-f8aa-46e3-98f2-f7d2b1e400f6-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-t2k77\" (UID: \"45e92205-f8aa-46e3-98f2-f7d2b1e400f6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t2k77"
Sep 11 18:58:31.213171 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:58:31.213113    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45e92205-f8aa-46e3-98f2-f7d2b1e400f6-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-t2k77\" (UID: \"45e92205-f8aa-46e3-98f2-f7d2b1e400f6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t2k77"
Sep 11 18:58:31.213224 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:58:31.213197    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45e92205-f8aa-46e3-98f2-f7d2b1e400f6-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-t2k77\" (UID: \"45e92205-f8aa-46e3-98f2-f7d2b1e400f6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t2k77"
Sep 11 18:58:31.213293 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:58:31.213274    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45e92205-f8aa-46e3-98f2-f7d2b1e400f6-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-t2k77\" (UID: \"45e92205-f8aa-46e3-98f2-f7d2b1e400f6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t2k77"
Sep 11 18:58:31.215161 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:58:31.215138    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45e92205-f8aa-46e3-98f2-f7d2b1e400f6-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-t2k77\" (UID: \"45e92205-f8aa-46e3-98f2-f7d2b1e400f6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t2k77"
Sep 11 18:58:31.215554 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:58:31.215534    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45e92205-f8aa-46e3-98f2-f7d2b1e400f6-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-t2k77\" (UID: \"45e92205-f8aa-46e3-98f2-f7d2b1e400f6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t2k77"
Sep 11 18:58:31.220127 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:58:31.220106    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2hsz\" (UniqueName: \"kubernetes.io/projected/45e92205-f8aa-46e3-98f2-f7d2b1e400f6-kube-api-access-n2hsz\") pod \"e2e-distinct-simulated-kserve-59975d69f9-t2k77\" (UID: \"45e92205-f8aa-46e3-98f2-f7d2b1e400f6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t2k77"
Sep 11 18:58:31.342895 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:58:31.342800    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t2k77"
Sep 11 18:58:31.613900 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:58:31.613878    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-t2k77"]
Sep 11 18:58:31.616339 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:58:31.616303    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45e92205_f8aa_46e3_98f2_f7d2b1e400f6.slice/crio-5afab1c9e30036ea93ceebff20c703d095521dc052f05eafb706a5da9db0ed81 WatchSource:0}: Error finding container 5afab1c9e30036ea93ceebff20c703d095521dc052f05eafb706a5da9db0ed81: Status 404 returned error can't find the container with id 5afab1c9e30036ea93ceebff20c703d095521dc052f05eafb706a5da9db0ed81
Sep 11 18:58:31.761364 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:58:31.761332    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t2k77" event={"ID":"45e92205-f8aa-46e3-98f2-f7d2b1e400f6","Type":"ContainerStarted","Data":"5afab1c9e30036ea93ceebff20c703d095521dc052f05eafb706a5da9db0ed81"}
Sep 11 18:58:32.766582 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:58:32.766546    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t2k77" event={"ID":"45e92205-f8aa-46e3-98f2-f7d2b1e400f6","Type":"ContainerStarted","Data":"133f7f493bdef634b66697cd4ec6cfc4395de2454519d366cb2e3634fe4ee49c"}
Sep 11 18:58:32.784289 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:58:32.784245    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t2k77" podStartSLOduration=0.806026625 podStartE2EDuration="1.784228194s" totalImagesPullingTime="978.201569ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:58:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:58:32.782937464 +0000 UTC m=+588.860528054" watchObservedRunningTime="2026-09-11 18:58:32.784228194 +0000 UTC m=+588.861818749"
Sep 11 18:58:41.343935 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:58:41.343897    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t2k77"
Sep 11 18:58:41.344391 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:58:41.343965    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t2k77"
Sep 11 18:58:41.356364 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:58:41.356344    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t2k77"
Sep 11 18:58:41.821018 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:58:41.820991    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t2k77"
Sep 11 18:58:44.300044 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:58:44.300016    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-hsbsf_d0880abe-ec36-48e4-a511-ec663873fffa/console-operator/1.log"
Sep 11 18:58:44.300442 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:58:44.300392    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-hsbsf_d0880abe-ec36-48e4-a511-ec663873fffa/console-operator/1.log"
Sep 11 18:59:11.534803 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:59:11.534725    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-9v9gf"]
Sep 11 18:59:11.537662 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:59:11.537647    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9v9gf"
Sep 11 18:59:11.539773 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:59:11.539757    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 11 18:59:11.548758 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:59:11.548734    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-9v9gf"]
Sep 11 18:59:11.616771 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:59:11.616743    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4449df87-c83f-4043-b2d4-706f8c46ddb0-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9v9gf\" (UID: \"4449df87-c83f-4043-b2d4-706f8c46ddb0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9v9gf"
Sep 11 18:59:11.616921 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:59:11.616790    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rccvz\" (UniqueName: \"kubernetes.io/projected/4449df87-c83f-4043-b2d4-706f8c46ddb0-kube-api-access-rccvz\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9v9gf\" (UID: \"4449df87-c83f-4043-b2d4-706f8c46ddb0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9v9gf"
Sep 11 18:59:11.616921 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:59:11.616864    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4449df87-c83f-4043-b2d4-706f8c46ddb0-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9v9gf\" (UID: \"4449df87-c83f-4043-b2d4-706f8c46ddb0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9v9gf"
Sep 11 18:59:11.616921 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:59:11.616894    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4449df87-c83f-4043-b2d4-706f8c46ddb0-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9v9gf\" (UID: \"4449df87-c83f-4043-b2d4-706f8c46ddb0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9v9gf"
Sep 11 18:59:11.617070 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:59:11.616924    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4449df87-c83f-4043-b2d4-706f8c46ddb0-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9v9gf\" (UID: \"4449df87-c83f-4043-b2d4-706f8c46ddb0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9v9gf"
Sep 11 18:59:11.617070 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:59:11.616961    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4449df87-c83f-4043-b2d4-706f8c46ddb0-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9v9gf\" (UID: \"4449df87-c83f-4043-b2d4-706f8c46ddb0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9v9gf"
Sep 11 18:59:11.717952 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:59:11.717918    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4449df87-c83f-4043-b2d4-706f8c46ddb0-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9v9gf\" (UID: \"4449df87-c83f-4043-b2d4-706f8c46ddb0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9v9gf"
Sep 11 18:59:11.718144 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:59:11.717960    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rccvz\" (UniqueName: \"kubernetes.io/projected/4449df87-c83f-4043-b2d4-706f8c46ddb0-kube-api-access-rccvz\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9v9gf\" (UID: \"4449df87-c83f-4043-b2d4-706f8c46ddb0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9v9gf"
Sep 11 18:59:11.718144 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:59:11.718005    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4449df87-c83f-4043-b2d4-706f8c46ddb0-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9v9gf\" (UID: \"4449df87-c83f-4043-b2d4-706f8c46ddb0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9v9gf"
Sep 11 18:59:11.718144 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:59:11.718021    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4449df87-c83f-4043-b2d4-706f8c46ddb0-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9v9gf\" (UID: \"4449df87-c83f-4043-b2d4-706f8c46ddb0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9v9gf"
Sep 11 18:59:11.718144 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:59:11.718047    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4449df87-c83f-4043-b2d4-706f8c46ddb0-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9v9gf\" (UID: \"4449df87-c83f-4043-b2d4-706f8c46ddb0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9v9gf"
Sep 11 18:59:11.718373 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:59:11.718165    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4449df87-c83f-4043-b2d4-706f8c46ddb0-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9v9gf\" (UID: \"4449df87-c83f-4043-b2d4-706f8c46ddb0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9v9gf"
Sep 11 18:59:11.718373 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:59:11.718355    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4449df87-c83f-4043-b2d4-706f8c46ddb0-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9v9gf\" (UID: \"4449df87-c83f-4043-b2d4-706f8c46ddb0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9v9gf"
Sep 11 18:59:11.718482 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:59:11.718464    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4449df87-c83f-4043-b2d4-706f8c46ddb0-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9v9gf\" (UID: \"4449df87-c83f-4043-b2d4-706f8c46ddb0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9v9gf"
Sep 11 18:59:11.718533 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:59:11.718507    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4449df87-c83f-4043-b2d4-706f8c46ddb0-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9v9gf\" (UID: \"4449df87-c83f-4043-b2d4-706f8c46ddb0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9v9gf"
Sep 11 18:59:11.720454 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:59:11.720432    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4449df87-c83f-4043-b2d4-706f8c46ddb0-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9v9gf\" (UID: \"4449df87-c83f-4043-b2d4-706f8c46ddb0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9v9gf"
Sep 11 18:59:11.720806 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:59:11.720789    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4449df87-c83f-4043-b2d4-706f8c46ddb0-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9v9gf\" (UID: \"4449df87-c83f-4043-b2d4-706f8c46ddb0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9v9gf"
Sep 11 18:59:11.725772 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:59:11.725745    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rccvz\" (UniqueName: \"kubernetes.io/projected/4449df87-c83f-4043-b2d4-706f8c46ddb0-kube-api-access-rccvz\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9v9gf\" (UID: \"4449df87-c83f-4043-b2d4-706f8c46ddb0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9v9gf"
Sep 11 18:59:11.849767 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:59:11.849699    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9v9gf"
Sep 11 18:59:12.121860 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:59:12.121835    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-9v9gf"]
Sep 11 18:59:12.124113 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:59:12.124085    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4449df87_c83f_4043_b2d4_706f8c46ddb0.slice/crio-f965a10bb331f4bde58895d1c40aa70f00cc7d834391ca6be424c06ed0527212 WatchSource:0}: Error finding container f965a10bb331f4bde58895d1c40aa70f00cc7d834391ca6be424c06ed0527212: Status 404 returned error can't find the container with id f965a10bb331f4bde58895d1c40aa70f00cc7d834391ca6be424c06ed0527212
Sep 11 18:59:12.125442 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:59:12.125422    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 18:59:12.942829 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:59:12.942793    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9v9gf" event={"ID":"4449df87-c83f-4043-b2d4-706f8c46ddb0","Type":"ContainerStarted","Data":"e1494d57b96bd2a638a6edbef20c21120a2dd1a4e16c0cad1a72b145edfaf400"}
Sep 11 18:59:12.943184 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:59:12.942852    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9v9gf" event={"ID":"4449df87-c83f-4043-b2d4-706f8c46ddb0","Type":"ContainerStarted","Data":"f965a10bb331f4bde58895d1c40aa70f00cc7d834391ca6be424c06ed0527212"}
Sep 11 18:59:12.958699 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:59:12.958654    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9v9gf" podStartSLOduration=1.728415249 podStartE2EDuration="1.958638052s" totalImagesPullingTime="230.222803ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:59:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:59:12.957703573 +0000 UTC m=+629.035294139" watchObservedRunningTime="2026-09-11 18:59:12.958638052 +0000 UTC m=+629.036228607"
Sep 11 18:59:18.831321 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:59:18.831291    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-fpn8c"]
Sep 11 18:59:18.834701 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:59:18.834681    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fpn8c"
Sep 11 18:59:18.837711 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:59:18.837685    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 11 18:59:18.843460 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:59:18.843294    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-fpn8c"]
Sep 11 18:59:18.872695 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:59:18.872663    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glj74\" (UniqueName: \"kubernetes.io/projected/697b3bd3-5cca-43d1-9b05-ca5ba81a7e99-kube-api-access-glj74\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fpn8c\" (UID: \"697b3bd3-5cca-43d1-9b05-ca5ba81a7e99\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fpn8c"
Sep 11 18:59:18.872822 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:59:18.872707    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/697b3bd3-5cca-43d1-9b05-ca5ba81a7e99-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fpn8c\" (UID: \"697b3bd3-5cca-43d1-9b05-ca5ba81a7e99\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fpn8c"
Sep 11 18:59:18.872822 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:59:18.872763    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/697b3bd3-5cca-43d1-9b05-ca5ba81a7e99-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fpn8c\" (UID: \"697b3bd3-5cca-43d1-9b05-ca5ba81a7e99\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fpn8c"
Sep 11 18:59:18.872822 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:59:18.872797    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/697b3bd3-5cca-43d1-9b05-ca5ba81a7e99-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fpn8c\" (UID: \"697b3bd3-5cca-43d1-9b05-ca5ba81a7e99\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fpn8c"
Sep 11 18:59:18.872989 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:59:18.872856    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/697b3bd3-5cca-43d1-9b05-ca5ba81a7e99-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fpn8c\" (UID: \"697b3bd3-5cca-43d1-9b05-ca5ba81a7e99\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fpn8c"
Sep 11 18:59:18.872989 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:59:18.872898    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/697b3bd3-5cca-43d1-9b05-ca5ba81a7e99-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fpn8c\" (UID: \"697b3bd3-5cca-43d1-9b05-ca5ba81a7e99\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fpn8c"
Sep 11 18:59:18.973923 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:59:18.973878    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/697b3bd3-5cca-43d1-9b05-ca5ba81a7e99-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fpn8c\" (UID: \"697b3bd3-5cca-43d1-9b05-ca5ba81a7e99\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fpn8c"
Sep 11 18:59:18.973923 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:59:18.973926    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/697b3bd3-5cca-43d1-9b05-ca5ba81a7e99-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fpn8c\" (UID: \"697b3bd3-5cca-43d1-9b05-ca5ba81a7e99\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fpn8c"
Sep 11 18:59:18.974169 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:59:18.974043    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/697b3bd3-5cca-43d1-9b05-ca5ba81a7e99-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fpn8c\" (UID: \"697b3bd3-5cca-43d1-9b05-ca5ba81a7e99\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fpn8c"
Sep 11 18:59:18.974233 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:59:18.974171    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glj74\" (UniqueName: \"kubernetes.io/projected/697b3bd3-5cca-43d1-9b05-ca5ba81a7e99-kube-api-access-glj74\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fpn8c\" (UID: \"697b3bd3-5cca-43d1-9b05-ca5ba81a7e99\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fpn8c"
Sep 11 18:59:18.974233 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:59:18.974209    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/697b3bd3-5cca-43d1-9b05-ca5ba81a7e99-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fpn8c\" (UID: \"697b3bd3-5cca-43d1-9b05-ca5ba81a7e99\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fpn8c"
Sep 11 18:59:18.974330 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:59:18.974247    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/697b3bd3-5cca-43d1-9b05-ca5ba81a7e99-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fpn8c\" (UID: \"697b3bd3-5cca-43d1-9b05-ca5ba81a7e99\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fpn8c"
Sep 11 18:59:18.974330 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:59:18.974308    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/697b3bd3-5cca-43d1-9b05-ca5ba81a7e99-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fpn8c\" (UID: \"697b3bd3-5cca-43d1-9b05-ca5ba81a7e99\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fpn8c"
Sep 11 18:59:18.974436 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:59:18.974407    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/697b3bd3-5cca-43d1-9b05-ca5ba81a7e99-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fpn8c\" (UID: \"697b3bd3-5cca-43d1-9b05-ca5ba81a7e99\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fpn8c"
Sep 11 18:59:18.974562 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:59:18.974544    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/697b3bd3-5cca-43d1-9b05-ca5ba81a7e99-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fpn8c\" (UID: \"697b3bd3-5cca-43d1-9b05-ca5ba81a7e99\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fpn8c"
Sep 11 18:59:18.976416 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:59:18.976396    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/697b3bd3-5cca-43d1-9b05-ca5ba81a7e99-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fpn8c\" (UID: \"697b3bd3-5cca-43d1-9b05-ca5ba81a7e99\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fpn8c"
Sep 11 18:59:18.977240 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:59:18.977219    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/697b3bd3-5cca-43d1-9b05-ca5ba81a7e99-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fpn8c\" (UID: \"697b3bd3-5cca-43d1-9b05-ca5ba81a7e99\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fpn8c"
Sep 11 18:59:18.985189 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:59:18.985164    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-glj74\" (UniqueName: \"kubernetes.io/projected/697b3bd3-5cca-43d1-9b05-ca5ba81a7e99-kube-api-access-glj74\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fpn8c\" (UID: \"697b3bd3-5cca-43d1-9b05-ca5ba81a7e99\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fpn8c"
Sep 11 18:59:19.146577 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:59:19.146486    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fpn8c"
Sep 11 18:59:19.418749 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:59:19.418719    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-fpn8c"]
Sep 11 18:59:19.420724 ip-10-0-134-189 kubenswrapper[2578]: W0911 18:59:19.420698    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod697b3bd3_5cca_43d1_9b05_ca5ba81a7e99.slice/crio-435f9dbac81ae2d65f0beb7b05301041a0e969a1486afee101f1bb2b630cfa32 WatchSource:0}: Error finding container 435f9dbac81ae2d65f0beb7b05301041a0e969a1486afee101f1bb2b630cfa32: Status 404 returned error can't find the container with id 435f9dbac81ae2d65f0beb7b05301041a0e969a1486afee101f1bb2b630cfa32
Sep 11 18:59:19.976059 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:59:19.976027    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fpn8c" event={"ID":"697b3bd3-5cca-43d1-9b05-ca5ba81a7e99","Type":"ContainerStarted","Data":"5cc225e86738b54402082b9fb5d3fbc78e4938c139fdcc2a74eeca4cb69fb7cb"}
Sep 11 18:59:19.976431 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:59:19.976070    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fpn8c" event={"ID":"697b3bd3-5cca-43d1-9b05-ca5ba81a7e99","Type":"ContainerStarted","Data":"435f9dbac81ae2d65f0beb7b05301041a0e969a1486afee101f1bb2b630cfa32"}
Sep 11 18:59:19.993727 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:59:19.993668    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fpn8c" podStartSLOduration=1.780570216 podStartE2EDuration="1.993652395s" totalImagesPullingTime="213.082179ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:59:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:59:19.992300336 +0000 UTC m=+636.069890902" watchObservedRunningTime="2026-09-11 18:59:19.993652395 +0000 UTC m=+636.071242951"
Sep 11 18:59:21.850080 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:59:21.850051    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9v9gf"
Sep 11 18:59:21.850475 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:59:21.850107    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9v9gf"
Sep 11 18:59:21.862370 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:59:21.862350    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9v9gf"
Sep 11 18:59:21.995448 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:59:21.995418    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9v9gf"
Sep 11 18:59:29.147086 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:59:29.147048    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fpn8c"
Sep 11 18:59:29.147086 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:59:29.147096    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fpn8c"
Sep 11 18:59:29.159772 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:59:29.159749    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fpn8c"
Sep 11 18:59:30.028144 ip-10-0-134-189 kubenswrapper[2578]: I0911 18:59:30.028112    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fpn8c"
Sep 11 19:00:37.438189 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:37.438119    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-c1aebc-openshift-default-5c87857866-4s5ww"]
Sep 11 19:00:37.441563 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:37.441542    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c1aebc-openshift-default-5c87857866-4s5ww"
Sep 11 19:00:37.444302 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:37.444279    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-c1aebc-openshift-default-dockercfg-mblf9\""
Sep 11 19:00:37.452673 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:37.452652    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c1aebc-openshift-default-5c87857866-4s5ww"]
Sep 11 19:00:37.509324 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:37.509298    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9b6e770-8c4d-4531-88d9-ab04cb50effc-istio-data\") pod \"e2e-worker-w4-c1aebc-openshift-default-5c87857866-4s5ww\" (UID: \"f9b6e770-8c4d-4531-88d9-ab04cb50effc\") " pod="openshift-ingress/e2e-worker-w4-c1aebc-openshift-default-5c87857866-4s5ww"
Sep 11 19:00:37.509451 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:37.509348    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hk67v\" (UniqueName: \"kubernetes.io/projected/f9b6e770-8c4d-4531-88d9-ab04cb50effc-kube-api-access-hk67v\") pod \"e2e-worker-w4-c1aebc-openshift-default-5c87857866-4s5ww\" (UID: \"f9b6e770-8c4d-4531-88d9-ab04cb50effc\") " pod="openshift-ingress/e2e-worker-w4-c1aebc-openshift-default-5c87857866-4s5ww"
Sep 11 19:00:37.509451 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:37.509377    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9b6e770-8c4d-4531-88d9-ab04cb50effc-workload-certs\") pod \"e2e-worker-w4-c1aebc-openshift-default-5c87857866-4s5ww\" (UID: \"f9b6e770-8c4d-4531-88d9-ab04cb50effc\") " pod="openshift-ingress/e2e-worker-w4-c1aebc-openshift-default-5c87857866-4s5ww"
Sep 11 19:00:37.509451 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:37.509434    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9b6e770-8c4d-4531-88d9-ab04cb50effc-credential-socket\") pod \"e2e-worker-w4-c1aebc-openshift-default-5c87857866-4s5ww\" (UID: \"f9b6e770-8c4d-4531-88d9-ab04cb50effc\") " pod="openshift-ingress/e2e-worker-w4-c1aebc-openshift-default-5c87857866-4s5ww"
Sep 11 19:00:37.509618 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:37.509473    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9b6e770-8c4d-4531-88d9-ab04cb50effc-istiod-ca-cert\") pod \"e2e-worker-w4-c1aebc-openshift-default-5c87857866-4s5ww\" (UID: \"f9b6e770-8c4d-4531-88d9-ab04cb50effc\") " pod="openshift-ingress/e2e-worker-w4-c1aebc-openshift-default-5c87857866-4s5ww"
Sep 11 19:00:37.509618 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:37.509508    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9b6e770-8c4d-4531-88d9-ab04cb50effc-istio-token\") pod \"e2e-worker-w4-c1aebc-openshift-default-5c87857866-4s5ww\" (UID: \"f9b6e770-8c4d-4531-88d9-ab04cb50effc\") " pod="openshift-ingress/e2e-worker-w4-c1aebc-openshift-default-5c87857866-4s5ww"
Sep 11 19:00:37.509618 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:37.509578    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9b6e770-8c4d-4531-88d9-ab04cb50effc-workload-socket\") pod \"e2e-worker-w4-c1aebc-openshift-default-5c87857866-4s5ww\" (UID: \"f9b6e770-8c4d-4531-88d9-ab04cb50effc\") " pod="openshift-ingress/e2e-worker-w4-c1aebc-openshift-default-5c87857866-4s5ww"
Sep 11 19:00:37.509769 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:37.509631    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9b6e770-8c4d-4531-88d9-ab04cb50effc-istio-podinfo\") pod \"e2e-worker-w4-c1aebc-openshift-default-5c87857866-4s5ww\" (UID: \"f9b6e770-8c4d-4531-88d9-ab04cb50effc\") " pod="openshift-ingress/e2e-worker-w4-c1aebc-openshift-default-5c87857866-4s5ww"
Sep 11 19:00:37.509769 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:37.509686    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9b6e770-8c4d-4531-88d9-ab04cb50effc-istio-envoy\") pod \"e2e-worker-w4-c1aebc-openshift-default-5c87857866-4s5ww\" (UID: \"f9b6e770-8c4d-4531-88d9-ab04cb50effc\") " pod="openshift-ingress/e2e-worker-w4-c1aebc-openshift-default-5c87857866-4s5ww"
Sep 11 19:00:37.610339 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:37.610314    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9b6e770-8c4d-4531-88d9-ab04cb50effc-istio-envoy\") pod \"e2e-worker-w4-c1aebc-openshift-default-5c87857866-4s5ww\" (UID: \"f9b6e770-8c4d-4531-88d9-ab04cb50effc\") " pod="openshift-ingress/e2e-worker-w4-c1aebc-openshift-default-5c87857866-4s5ww"
Sep 11 19:00:37.610482 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:37.610356    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9b6e770-8c4d-4531-88d9-ab04cb50effc-istio-data\") pod \"e2e-worker-w4-c1aebc-openshift-default-5c87857866-4s5ww\" (UID: \"f9b6e770-8c4d-4531-88d9-ab04cb50effc\") " pod="openshift-ingress/e2e-worker-w4-c1aebc-openshift-default-5c87857866-4s5ww"
Sep 11 19:00:37.610482 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:37.610385    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk67v\" (UniqueName: \"kubernetes.io/projected/f9b6e770-8c4d-4531-88d9-ab04cb50effc-kube-api-access-hk67v\") pod \"e2e-worker-w4-c1aebc-openshift-default-5c87857866-4s5ww\" (UID: \"f9b6e770-8c4d-4531-88d9-ab04cb50effc\") " pod="openshift-ingress/e2e-worker-w4-c1aebc-openshift-default-5c87857866-4s5ww"
Sep 11 19:00:37.610482 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:37.610418    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9b6e770-8c4d-4531-88d9-ab04cb50effc-workload-certs\") pod \"e2e-worker-w4-c1aebc-openshift-default-5c87857866-4s5ww\" (UID: \"f9b6e770-8c4d-4531-88d9-ab04cb50effc\") " pod="openshift-ingress/e2e-worker-w4-c1aebc-openshift-default-5c87857866-4s5ww"
Sep 11 19:00:37.610482 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:37.610436    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9b6e770-8c4d-4531-88d9-ab04cb50effc-credential-socket\") pod \"e2e-worker-w4-c1aebc-openshift-default-5c87857866-4s5ww\" (UID: \"f9b6e770-8c4d-4531-88d9-ab04cb50effc\") " pod="openshift-ingress/e2e-worker-w4-c1aebc-openshift-default-5c87857866-4s5ww"
Sep 11 19:00:37.610482 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:37.610452    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9b6e770-8c4d-4531-88d9-ab04cb50effc-istiod-ca-cert\") pod \"e2e-worker-w4-c1aebc-openshift-default-5c87857866-4s5ww\" (UID: \"f9b6e770-8c4d-4531-88d9-ab04cb50effc\") " pod="openshift-ingress/e2e-worker-w4-c1aebc-openshift-default-5c87857866-4s5ww"
Sep 11 19:00:37.610757 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:37.610481    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9b6e770-8c4d-4531-88d9-ab04cb50effc-istio-token\") pod \"e2e-worker-w4-c1aebc-openshift-default-5c87857866-4s5ww\" (UID: \"f9b6e770-8c4d-4531-88d9-ab04cb50effc\") " pod="openshift-ingress/e2e-worker-w4-c1aebc-openshift-default-5c87857866-4s5ww"
Sep 11 19:00:37.610757 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:37.610514    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9b6e770-8c4d-4531-88d9-ab04cb50effc-workload-socket\") pod \"e2e-worker-w4-c1aebc-openshift-default-5c87857866-4s5ww\" (UID: \"f9b6e770-8c4d-4531-88d9-ab04cb50effc\") " pod="openshift-ingress/e2e-worker-w4-c1aebc-openshift-default-5c87857866-4s5ww"
Sep 11 19:00:37.610757 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:37.610538    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9b6e770-8c4d-4531-88d9-ab04cb50effc-istio-podinfo\") pod \"e2e-worker-w4-c1aebc-openshift-default-5c87857866-4s5ww\" (UID: \"f9b6e770-8c4d-4531-88d9-ab04cb50effc\") " pod="openshift-ingress/e2e-worker-w4-c1aebc-openshift-default-5c87857866-4s5ww"
Sep 11 19:00:37.610911 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:37.610789    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9b6e770-8c4d-4531-88d9-ab04cb50effc-istio-data\") pod \"e2e-worker-w4-c1aebc-openshift-default-5c87857866-4s5ww\" (UID: \"f9b6e770-8c4d-4531-88d9-ab04cb50effc\") " pod="openshift-ingress/e2e-worker-w4-c1aebc-openshift-default-5c87857866-4s5ww"
Sep 11 19:00:37.610911 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:37.610899    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9b6e770-8c4d-4531-88d9-ab04cb50effc-credential-socket\") pod \"e2e-worker-w4-c1aebc-openshift-default-5c87857866-4s5ww\" (UID: \"f9b6e770-8c4d-4531-88d9-ab04cb50effc\") " pod="openshift-ingress/e2e-worker-w4-c1aebc-openshift-default-5c87857866-4s5ww"
Sep 11 19:00:37.611071 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:37.611049    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9b6e770-8c4d-4531-88d9-ab04cb50effc-workload-socket\") pod \"e2e-worker-w4-c1aebc-openshift-default-5c87857866-4s5ww\" (UID: \"f9b6e770-8c4d-4531-88d9-ab04cb50effc\") " pod="openshift-ingress/e2e-worker-w4-c1aebc-openshift-default-5c87857866-4s5ww"
Sep 11 19:00:37.611161 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:37.611140    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9b6e770-8c4d-4531-88d9-ab04cb50effc-workload-certs\") pod \"e2e-worker-w4-c1aebc-openshift-default-5c87857866-4s5ww\" (UID: \"f9b6e770-8c4d-4531-88d9-ab04cb50effc\") " pod="openshift-ingress/e2e-worker-w4-c1aebc-openshift-default-5c87857866-4s5ww"
Sep 11 19:00:37.611246 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:37.611230    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9b6e770-8c4d-4531-88d9-ab04cb50effc-istiod-ca-cert\") pod \"e2e-worker-w4-c1aebc-openshift-default-5c87857866-4s5ww\" (UID: \"f9b6e770-8c4d-4531-88d9-ab04cb50effc\") " pod="openshift-ingress/e2e-worker-w4-c1aebc-openshift-default-5c87857866-4s5ww"
Sep 11 19:00:37.612736 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:37.612712    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9b6e770-8c4d-4531-88d9-ab04cb50effc-istio-envoy\") pod \"e2e-worker-w4-c1aebc-openshift-default-5c87857866-4s5ww\" (UID: \"f9b6e770-8c4d-4531-88d9-ab04cb50effc\") " pod="openshift-ingress/e2e-worker-w4-c1aebc-openshift-default-5c87857866-4s5ww"
Sep 11 19:00:37.613063 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:37.613045    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9b6e770-8c4d-4531-88d9-ab04cb50effc-istio-podinfo\") pod \"e2e-worker-w4-c1aebc-openshift-default-5c87857866-4s5ww\" (UID: \"f9b6e770-8c4d-4531-88d9-ab04cb50effc\") " pod="openshift-ingress/e2e-worker-w4-c1aebc-openshift-default-5c87857866-4s5ww"
Sep 11 19:00:37.618672 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:37.618649    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk67v\" (UniqueName: \"kubernetes.io/projected/f9b6e770-8c4d-4531-88d9-ab04cb50effc-kube-api-access-hk67v\") pod \"e2e-worker-w4-c1aebc-openshift-default-5c87857866-4s5ww\" (UID: \"f9b6e770-8c4d-4531-88d9-ab04cb50effc\") " pod="openshift-ingress/e2e-worker-w4-c1aebc-openshift-default-5c87857866-4s5ww"
Sep 11 19:00:37.618858 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:37.618841    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9b6e770-8c4d-4531-88d9-ab04cb50effc-istio-token\") pod \"e2e-worker-w4-c1aebc-openshift-default-5c87857866-4s5ww\" (UID: \"f9b6e770-8c4d-4531-88d9-ab04cb50effc\") " pod="openshift-ingress/e2e-worker-w4-c1aebc-openshift-default-5c87857866-4s5ww"
Sep 11 19:00:37.752851 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:37.752828    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c1aebc-openshift-default-5c87857866-4s5ww"
Sep 11 19:00:38.029362 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:38.029339    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c1aebc-openshift-default-5c87857866-4s5ww"]
Sep 11 19:00:38.031725 ip-10-0-134-189 kubenswrapper[2578]: W0911 19:00:38.031696    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9b6e770_8c4d_4531_88d9_ab04cb50effc.slice/crio-4ed77de082915928c36da19ec79b9940b800e51aeb27915cded683e36203a9af WatchSource:0}: Error finding container 4ed77de082915928c36da19ec79b9940b800e51aeb27915cded683e36203a9af: Status 404 returned error can't find the container with id 4ed77de082915928c36da19ec79b9940b800e51aeb27915cded683e36203a9af
Sep 11 19:00:38.033776 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:38.033743    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 19:00:38.033863 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:38.033807    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 19:00:38.033863 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:38.033834    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 19:00:38.307531 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:38.307438    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c1aebc-openshift-default-5c87857866-4s5ww" event={"ID":"f9b6e770-8c4d-4531-88d9-ab04cb50effc","Type":"ContainerStarted","Data":"236544bd6a2ada481215866aedad1fb4c3cee2a3555d6ec1e253198f982252b5"}
Sep 11 19:00:38.307531 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:38.307502    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c1aebc-openshift-default-5c87857866-4s5ww" event={"ID":"f9b6e770-8c4d-4531-88d9-ab04cb50effc","Type":"ContainerStarted","Data":"4ed77de082915928c36da19ec79b9940b800e51aeb27915cded683e36203a9af"}
Sep 11 19:00:38.329827 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:38.329754    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-c1aebc-openshift-default-5c87857866-4s5ww" podStartSLOduration=1.32972886 podStartE2EDuration="1.32972886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:00:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:00:38.328113598 +0000 UTC m=+714.405704214" watchObservedRunningTime="2026-09-11 19:00:38.32972886 +0000 UTC m=+714.407319415"
Sep 11 19:00:38.753320 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:38.753270    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-c1aebc-openshift-default-5c87857866-4s5ww"
Sep 11 19:00:38.969407 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:38.969372    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7797-053f84"]
Sep 11 19:00:38.972423 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:38.972405    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7797-053f84"
Sep 11 19:00:38.974540 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:38.974516    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-jxjbf\""
Sep 11 19:00:38.977642 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:38.977621    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7797-053f84"]
Sep 11 19:00:39.024036 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:39.023958    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5s6tp\" (UniqueName: \"kubernetes.io/projected/3569b0b6-ae00-4c8c-b5f9-4990831c691d-kube-api-access-5s6tp\") pod \"test-curl-7797-053f84\" (UID: \"3569b0b6-ae00-4c8c-b5f9-4990831c691d\") " pod="opendatahub/test-curl-7797-053f84"
Sep 11 19:00:39.125038 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:39.125007    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5s6tp\" (UniqueName: \"kubernetes.io/projected/3569b0b6-ae00-4c8c-b5f9-4990831c691d-kube-api-access-5s6tp\") pod \"test-curl-7797-053f84\" (UID: \"3569b0b6-ae00-4c8c-b5f9-4990831c691d\") " pod="opendatahub/test-curl-7797-053f84"
Sep 11 19:00:39.136518 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:39.136488    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5s6tp\" (UniqueName: \"kubernetes.io/projected/3569b0b6-ae00-4c8c-b5f9-4990831c691d-kube-api-access-5s6tp\") pod \"test-curl-7797-053f84\" (UID: \"3569b0b6-ae00-4c8c-b5f9-4990831c691d\") " pod="opendatahub/test-curl-7797-053f84"
Sep 11 19:00:39.283144 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:39.283070    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7797-053f84"
Sep 11 19:00:39.557838 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:39.557759    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7797-053f84"]
Sep 11 19:00:39.560678 ip-10-0-134-189 kubenswrapper[2578]: W0911 19:00:39.560650    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3569b0b6_ae00_4c8c_b5f9_4990831c691d.slice/crio-814fdf60152356d273ecee5113b0bff911a0526f002edc6c2483084bf33c8c46 WatchSource:0}: Error finding container 814fdf60152356d273ecee5113b0bff911a0526f002edc6c2483084bf33c8c46: Status 404 returned error can't find the container with id 814fdf60152356d273ecee5113b0bff911a0526f002edc6c2483084bf33c8c46
Sep 11 19:00:39.758579 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:39.758546    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-c1aebc-openshift-default-5c87857866-4s5ww"
Sep 11 19:00:40.317374 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:40.317334    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7797-053f84" event={"ID":"3569b0b6-ae00-4c8c-b5f9-4990831c691d","Type":"ContainerStarted","Data":"814fdf60152356d273ecee5113b0bff911a0526f002edc6c2483084bf33c8c46"}
Sep 11 19:00:40.317647 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:40.317632    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-c1aebc-openshift-default-5c87857866-4s5ww"
Sep 11 19:00:40.318800 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:40.318778    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-c1aebc-openshift-default-5c87857866-4s5ww"
Sep 11 19:00:41.323381 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:41.323340    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7797-053f84" event={"ID":"3569b0b6-ae00-4c8c-b5f9-4990831c691d","Type":"ContainerStarted","Data":"8b22b8ae5a2f1d30b2bdd057a59bccb24151ba8831e818addb202e71d98c465f"}
Sep 11 19:00:41.338729 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:41.338685    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7797-053f84" podStartSLOduration=2.624619417 podStartE2EDuration="3.338670775s" totalImagesPullingTime="714.051358ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:00:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:00:41.337760802 +0000 UTC m=+717.415351357" watchObservedRunningTime="2026-09-11 19:00:41.338670775 +0000 UTC m=+717.416261329"
Sep 11 19:00:41.919660 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:41.919626    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7797-053f84"]
Sep 11 19:00:41.924387 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:41.924362    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7797-053f84"]
Sep 11 19:00:42.034672 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:42.034579    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7797-5fbe22"]
Sep 11 19:00:42.034988 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:42.034970    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3569b0b6-ae00-4c8c-b5f9-4990831c691d" containerName="test-curl-7797-053f84"
Sep 11 19:00:42.035401 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:42.034990    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3569b0b6-ae00-4c8c-b5f9-4990831c691d" containerName="test-curl-7797-053f84"
Sep 11 19:00:42.035401 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:42.035092    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3569b0b6-ae00-4c8c-b5f9-4990831c691d" containerName="test-curl-7797-053f84"
Sep 11 19:00:42.038223 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:42.038203    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7797-5fbe22"
Sep 11 19:00:42.046025 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:42.045995    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7797-5fbe22"]
Sep 11 19:00:42.153684 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:42.153649    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qskj\" (UniqueName: \"kubernetes.io/projected/12b3df87-cbfa-411e-a919-dc0eb7b06970-kube-api-access-8qskj\") pod \"test-curl-7797-5fbe22\" (UID: \"12b3df87-cbfa-411e-a919-dc0eb7b06970\") " pod="opendatahub/test-curl-7797-5fbe22"
Sep 11 19:00:42.254323 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:42.254276    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qskj\" (UniqueName: \"kubernetes.io/projected/12b3df87-cbfa-411e-a919-dc0eb7b06970-kube-api-access-8qskj\") pod \"test-curl-7797-5fbe22\" (UID: \"12b3df87-cbfa-411e-a919-dc0eb7b06970\") " pod="opendatahub/test-curl-7797-5fbe22"
Sep 11 19:00:42.264227 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:42.264196    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qskj\" (UniqueName: \"kubernetes.io/projected/12b3df87-cbfa-411e-a919-dc0eb7b06970-kube-api-access-8qskj\") pod \"test-curl-7797-5fbe22\" (UID: \"12b3df87-cbfa-411e-a919-dc0eb7b06970\") " pod="opendatahub/test-curl-7797-5fbe22"
Sep 11 19:00:42.328292 ip-10-0-134-189 kubenswrapper[2578]: E0911 19:00:42.328223    2578 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7797-053f84" container="test-curl-7797-053f84"
Sep 11 19:00:42.352309 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:42.352283    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7797-5fbe22"
Sep 11 19:00:42.625734 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:42.625627    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7797-5fbe22"]
Sep 11 19:00:42.628042 ip-10-0-134-189 kubenswrapper[2578]: W0911 19:00:42.628013    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod12b3df87_cbfa_411e_a919_dc0eb7b06970.slice/crio-b4bf90b13e561298f1f6e3e15b95cad45a60c40bee85efa07fd7aecfec7a9d23 WatchSource:0}: Error finding container b4bf90b13e561298f1f6e3e15b95cad45a60c40bee85efa07fd7aecfec7a9d23: Status 404 returned error can't find the container with id b4bf90b13e561298f1f6e3e15b95cad45a60c40bee85efa07fd7aecfec7a9d23
Sep 11 19:00:43.334223 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:43.334190    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7797-5fbe22" event={"ID":"12b3df87-cbfa-411e-a919-dc0eb7b06970","Type":"ContainerStarted","Data":"7250e5fd8b254131880b2b73a681389913ccc11610d90bfd984a9960eca3e5c5"}
Sep 11 19:00:43.334728 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:43.334231    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7797-5fbe22" event={"ID":"12b3df87-cbfa-411e-a919-dc0eb7b06970","Type":"ContainerStarted","Data":"b4bf90b13e561298f1f6e3e15b95cad45a60c40bee85efa07fd7aecfec7a9d23"}
Sep 11 19:00:43.334728 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:43.334436    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7797-053f84" podUID="3569b0b6-ae00-4c8c-b5f9-4990831c691d" containerName="test-curl-7797-053f84" containerID="cri-o://8b22b8ae5a2f1d30b2bdd057a59bccb24151ba8831e818addb202e71d98c465f" gracePeriod=2
Sep 11 19:00:43.348794 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:43.348746    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7797-5fbe22" podStartSLOduration=1.34872981 podStartE2EDuration="1.34872981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:00:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:00:43.348578704 +0000 UTC m=+719.426169260" watchObservedRunningTime="2026-09-11 19:00:43.34872981 +0000 UTC m=+719.426320375"
Sep 11 19:00:43.917582 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:43.917545    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7797-5fbe22"]
Sep 11 19:00:43.919871 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:43.919847    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7797-5fbe22"]
Sep 11 19:00:44.338503 ip-10-0-134-189 kubenswrapper[2578]: E0911 19:00:44.338475    2578 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7797-5fbe22" container="test-curl-7797-5fbe22"
Sep 11 19:00:44.702342 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:44.702286    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-89e163-openshift-default-7774f9d49d-ddkg6"]
Sep 11 19:00:44.702792 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:44.702774    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12b3df87-cbfa-411e-a919-dc0eb7b06970" containerName="test-curl-7797-5fbe22"
Sep 11 19:00:44.702884 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:44.702795    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="12b3df87-cbfa-411e-a919-dc0eb7b06970" containerName="test-curl-7797-5fbe22"
Sep 11 19:00:44.702884 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:44.702870    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="12b3df87-cbfa-411e-a919-dc0eb7b06970" containerName="test-curl-7797-5fbe22"
Sep 11 19:00:44.706529 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:44.706505    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-89e163-openshift-default-7774f9d49d-ddkg6"
Sep 11 19:00:44.708836 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:44.708814    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-89e163-openshift-default-dockercfg-7wstq\""
Sep 11 19:00:44.722368 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:44.722331    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-89e163-openshift-default-7774f9d49d-ddkg6"]
Sep 11 19:00:44.775742 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:44.775696    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f145e25-8489-4f68-b919-c981f05e730b-workload-certs\") pod \"e2e-autores-89e163-openshift-default-7774f9d49d-ddkg6\" (UID: \"0f145e25-8489-4f68-b919-c981f05e730b\") " pod="openshift-ingress/e2e-autores-89e163-openshift-default-7774f9d49d-ddkg6"
Sep 11 19:00:44.775742 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:44.775742    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f145e25-8489-4f68-b919-c981f05e730b-istio-data\") pod \"e2e-autores-89e163-openshift-default-7774f9d49d-ddkg6\" (UID: \"0f145e25-8489-4f68-b919-c981f05e730b\") " pod="openshift-ingress/e2e-autores-89e163-openshift-default-7774f9d49d-ddkg6"
Sep 11 19:00:44.775940 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:44.775765    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f145e25-8489-4f68-b919-c981f05e730b-istio-podinfo\") pod \"e2e-autores-89e163-openshift-default-7774f9d49d-ddkg6\" (UID: \"0f145e25-8489-4f68-b919-c981f05e730b\") " pod="openshift-ingress/e2e-autores-89e163-openshift-default-7774f9d49d-ddkg6"
Sep 11 19:00:44.775940 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:44.775840    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f145e25-8489-4f68-b919-c981f05e730b-istio-token\") pod \"e2e-autores-89e163-openshift-default-7774f9d49d-ddkg6\" (UID: \"0f145e25-8489-4f68-b919-c981f05e730b\") " pod="openshift-ingress/e2e-autores-89e163-openshift-default-7774f9d49d-ddkg6"
Sep 11 19:00:44.775940 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:44.775867    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkzk7\" (UniqueName: \"kubernetes.io/projected/0f145e25-8489-4f68-b919-c981f05e730b-kube-api-access-wkzk7\") pod \"e2e-autores-89e163-openshift-default-7774f9d49d-ddkg6\" (UID: \"0f145e25-8489-4f68-b919-c981f05e730b\") " pod="openshift-ingress/e2e-autores-89e163-openshift-default-7774f9d49d-ddkg6"
Sep 11 19:00:44.775940 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:44.775886    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f145e25-8489-4f68-b919-c981f05e730b-istio-envoy\") pod \"e2e-autores-89e163-openshift-default-7774f9d49d-ddkg6\" (UID: \"0f145e25-8489-4f68-b919-c981f05e730b\") " pod="openshift-ingress/e2e-autores-89e163-openshift-default-7774f9d49d-ddkg6"
Sep 11 19:00:44.775940 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:44.775903    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f145e25-8489-4f68-b919-c981f05e730b-istiod-ca-cert\") pod \"e2e-autores-89e163-openshift-default-7774f9d49d-ddkg6\" (UID: \"0f145e25-8489-4f68-b919-c981f05e730b\") " pod="openshift-ingress/e2e-autores-89e163-openshift-default-7774f9d49d-ddkg6"
Sep 11 19:00:44.776102 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:44.775950    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f145e25-8489-4f68-b919-c981f05e730b-credential-socket\") pod \"e2e-autores-89e163-openshift-default-7774f9d49d-ddkg6\" (UID: \"0f145e25-8489-4f68-b919-c981f05e730b\") " pod="openshift-ingress/e2e-autores-89e163-openshift-default-7774f9d49d-ddkg6"
Sep 11 19:00:44.776102 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:44.775986    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f145e25-8489-4f68-b919-c981f05e730b-workload-socket\") pod \"e2e-autores-89e163-openshift-default-7774f9d49d-ddkg6\" (UID: \"0f145e25-8489-4f68-b919-c981f05e730b\") " pod="openshift-ingress/e2e-autores-89e163-openshift-default-7774f9d49d-ddkg6"
Sep 11 19:00:44.876725 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:44.876682    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f145e25-8489-4f68-b919-c981f05e730b-workload-certs\") pod \"e2e-autores-89e163-openshift-default-7774f9d49d-ddkg6\" (UID: \"0f145e25-8489-4f68-b919-c981f05e730b\") " pod="openshift-ingress/e2e-autores-89e163-openshift-default-7774f9d49d-ddkg6"
Sep 11 19:00:44.876725 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:44.876727    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f145e25-8489-4f68-b919-c981f05e730b-istio-data\") pod \"e2e-autores-89e163-openshift-default-7774f9d49d-ddkg6\" (UID: \"0f145e25-8489-4f68-b919-c981f05e730b\") " pod="openshift-ingress/e2e-autores-89e163-openshift-default-7774f9d49d-ddkg6"
Sep 11 19:00:44.876930 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:44.876752    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f145e25-8489-4f68-b919-c981f05e730b-istio-podinfo\") pod \"e2e-autores-89e163-openshift-default-7774f9d49d-ddkg6\" (UID: \"0f145e25-8489-4f68-b919-c981f05e730b\") " pod="openshift-ingress/e2e-autores-89e163-openshift-default-7774f9d49d-ddkg6"
Sep 11 19:00:44.876930 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:44.876792    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f145e25-8489-4f68-b919-c981f05e730b-istio-token\") pod \"e2e-autores-89e163-openshift-default-7774f9d49d-ddkg6\" (UID: \"0f145e25-8489-4f68-b919-c981f05e730b\") " pod="openshift-ingress/e2e-autores-89e163-openshift-default-7774f9d49d-ddkg6"
Sep 11 19:00:44.876930 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:44.876808    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkzk7\" (UniqueName: \"kubernetes.io/projected/0f145e25-8489-4f68-b919-c981f05e730b-kube-api-access-wkzk7\") pod \"e2e-autores-89e163-openshift-default-7774f9d49d-ddkg6\" (UID: \"0f145e25-8489-4f68-b919-c981f05e730b\") " pod="openshift-ingress/e2e-autores-89e163-openshift-default-7774f9d49d-ddkg6"
Sep 11 19:00:44.876930 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:44.876825    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f145e25-8489-4f68-b919-c981f05e730b-istio-envoy\") pod \"e2e-autores-89e163-openshift-default-7774f9d49d-ddkg6\" (UID: \"0f145e25-8489-4f68-b919-c981f05e730b\") " pod="openshift-ingress/e2e-autores-89e163-openshift-default-7774f9d49d-ddkg6"
Sep 11 19:00:44.877134 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:44.876997    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f145e25-8489-4f68-b919-c981f05e730b-istiod-ca-cert\") pod \"e2e-autores-89e163-openshift-default-7774f9d49d-ddkg6\" (UID: \"0f145e25-8489-4f68-b919-c981f05e730b\") " pod="openshift-ingress/e2e-autores-89e163-openshift-default-7774f9d49d-ddkg6"
Sep 11 19:00:44.877134 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:44.877064    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f145e25-8489-4f68-b919-c981f05e730b-credential-socket\") pod \"e2e-autores-89e163-openshift-default-7774f9d49d-ddkg6\" (UID: \"0f145e25-8489-4f68-b919-c981f05e730b\") " pod="openshift-ingress/e2e-autores-89e163-openshift-default-7774f9d49d-ddkg6"
Sep 11 19:00:44.877134 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:44.877108    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f145e25-8489-4f68-b919-c981f05e730b-workload-socket\") pod \"e2e-autores-89e163-openshift-default-7774f9d49d-ddkg6\" (UID: \"0f145e25-8489-4f68-b919-c981f05e730b\") " pod="openshift-ingress/e2e-autores-89e163-openshift-default-7774f9d49d-ddkg6"
Sep 11 19:00:44.877134 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:44.877120    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f145e25-8489-4f68-b919-c981f05e730b-workload-certs\") pod \"e2e-autores-89e163-openshift-default-7774f9d49d-ddkg6\" (UID: \"0f145e25-8489-4f68-b919-c981f05e730b\") " pod="openshift-ingress/e2e-autores-89e163-openshift-default-7774f9d49d-ddkg6"
Sep 11 19:00:44.877302 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:44.877136    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f145e25-8489-4f68-b919-c981f05e730b-istio-data\") pod \"e2e-autores-89e163-openshift-default-7774f9d49d-ddkg6\" (UID: \"0f145e25-8489-4f68-b919-c981f05e730b\") " pod="openshift-ingress/e2e-autores-89e163-openshift-default-7774f9d49d-ddkg6"
Sep 11 19:00:44.877366 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:44.877341    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f145e25-8489-4f68-b919-c981f05e730b-credential-socket\") pod \"e2e-autores-89e163-openshift-default-7774f9d49d-ddkg6\" (UID: \"0f145e25-8489-4f68-b919-c981f05e730b\") " pod="openshift-ingress/e2e-autores-89e163-openshift-default-7774f9d49d-ddkg6"
Sep 11 19:00:44.877417 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:44.877386    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f145e25-8489-4f68-b919-c981f05e730b-workload-socket\") pod \"e2e-autores-89e163-openshift-default-7774f9d49d-ddkg6\" (UID: \"0f145e25-8489-4f68-b919-c981f05e730b\") " pod="openshift-ingress/e2e-autores-89e163-openshift-default-7774f9d49d-ddkg6"
Sep 11 19:00:44.877807 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:44.877786    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f145e25-8489-4f68-b919-c981f05e730b-istiod-ca-cert\") pod \"e2e-autores-89e163-openshift-default-7774f9d49d-ddkg6\" (UID: \"0f145e25-8489-4f68-b919-c981f05e730b\") " pod="openshift-ingress/e2e-autores-89e163-openshift-default-7774f9d49d-ddkg6"
Sep 11 19:00:44.879317 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:44.879295    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f145e25-8489-4f68-b919-c981f05e730b-istio-envoy\") pod \"e2e-autores-89e163-openshift-default-7774f9d49d-ddkg6\" (UID: \"0f145e25-8489-4f68-b919-c981f05e730b\") " pod="openshift-ingress/e2e-autores-89e163-openshift-default-7774f9d49d-ddkg6"
Sep 11 19:00:44.879420 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:44.879400    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f145e25-8489-4f68-b919-c981f05e730b-istio-podinfo\") pod \"e2e-autores-89e163-openshift-default-7774f9d49d-ddkg6\" (UID: \"0f145e25-8489-4f68-b919-c981f05e730b\") " pod="openshift-ingress/e2e-autores-89e163-openshift-default-7774f9d49d-ddkg6"
Sep 11 19:00:44.889721 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:44.889692    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f145e25-8489-4f68-b919-c981f05e730b-istio-token\") pod \"e2e-autores-89e163-openshift-default-7774f9d49d-ddkg6\" (UID: \"0f145e25-8489-4f68-b919-c981f05e730b\") " pod="openshift-ingress/e2e-autores-89e163-openshift-default-7774f9d49d-ddkg6"
Sep 11 19:00:44.890295 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:44.890275    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkzk7\" (UniqueName: \"kubernetes.io/projected/0f145e25-8489-4f68-b919-c981f05e730b-kube-api-access-wkzk7\") pod \"e2e-autores-89e163-openshift-default-7774f9d49d-ddkg6\" (UID: \"0f145e25-8489-4f68-b919-c981f05e730b\") " pod="openshift-ingress/e2e-autores-89e163-openshift-default-7774f9d49d-ddkg6"
Sep 11 19:00:45.026435 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:45.026345    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-89e163-openshift-default-7774f9d49d-ddkg6"
Sep 11 19:00:45.302933 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:45.302910    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-89e163-openshift-default-7774f9d49d-ddkg6"]
Sep 11 19:00:45.304860 ip-10-0-134-189 kubenswrapper[2578]: W0911 19:00:45.304833    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f145e25_8489_4f68_b919_c981f05e730b.slice/crio-3d72b560947250254de23d018e71ebde4d407d8fea4cabf6701869228c36ff33 WatchSource:0}: Error finding container 3d72b560947250254de23d018e71ebde4d407d8fea4cabf6701869228c36ff33: Status 404 returned error can't find the container with id 3d72b560947250254de23d018e71ebde4d407d8fea4cabf6701869228c36ff33
Sep 11 19:00:45.306889 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:45.306857    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 19:00:45.306979 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:45.306918    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 19:00:45.306979 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:45.306946    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 19:00:45.344478 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:45.344454    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-89e163-openshift-default-7774f9d49d-ddkg6" event={"ID":"0f145e25-8489-4f68-b919-c981f05e730b","Type":"ContainerStarted","Data":"3d72b560947250254de23d018e71ebde4d407d8fea4cabf6701869228c36ff33"}
Sep 11 19:00:45.344791 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:45.344560    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7797-5fbe22" podUID="12b3df87-cbfa-411e-a919-dc0eb7b06970" containerName="test-curl-7797-5fbe22" containerID="cri-o://7250e5fd8b254131880b2b73a681389913ccc11610d90bfd984a9960eca3e5c5" gracePeriod=2
Sep 11 19:00:45.409118 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:45.408969    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c6c610-8cf45f4d7-lq42w"]
Sep 11 19:00:45.413480 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:45.413390    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c6c610-8cf45f4d7-lq42w"
Sep 11 19:00:45.416120 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:45.416091    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-c6c610\""
Sep 11 19:00:45.416120 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:45.416096    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-c6c610-dockercfg-jh2jc\""
Sep 11 19:00:45.427346 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:45.427256    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c6c610-8cf45f4d7-lq42w"]
Sep 11 19:00:45.482329 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:45.482293    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe0fddea-982f-4f3f-81bd-7e1a8c268157-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c6c610-8cf45f4d7-lq42w\" (UID: \"fe0fddea-982f-4f3f-81bd-7e1a8c268157\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c6c610-8cf45f4d7-lq42w"
Sep 11 19:00:45.482477 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:45.482440    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gc8j4\" (UniqueName: \"kubernetes.io/projected/fe0fddea-982f-4f3f-81bd-7e1a8c268157-kube-api-access-gc8j4\") pod \"payload-processing-e2e-inf-a-c6c610-8cf45f4d7-lq42w\" (UID: \"fe0fddea-982f-4f3f-81bd-7e1a8c268157\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c6c610-8cf45f4d7-lq42w"
Sep 11 19:00:45.519766 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:45.519737    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7797-053f84"
Sep 11 19:00:45.583699 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:45.583617    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5s6tp\" (UniqueName: \"kubernetes.io/projected/3569b0b6-ae00-4c8c-b5f9-4990831c691d-kube-api-access-5s6tp\") pod \"3569b0b6-ae00-4c8c-b5f9-4990831c691d\" (UID: \"3569b0b6-ae00-4c8c-b5f9-4990831c691d\") "
Sep 11 19:00:45.583872 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:45.583854    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe0fddea-982f-4f3f-81bd-7e1a8c268157-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c6c610-8cf45f4d7-lq42w\" (UID: \"fe0fddea-982f-4f3f-81bd-7e1a8c268157\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c6c610-8cf45f4d7-lq42w"
Sep 11 19:00:45.583974 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:45.583955    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gc8j4\" (UniqueName: \"kubernetes.io/projected/fe0fddea-982f-4f3f-81bd-7e1a8c268157-kube-api-access-gc8j4\") pod \"payload-processing-e2e-inf-a-c6c610-8cf45f4d7-lq42w\" (UID: \"fe0fddea-982f-4f3f-81bd-7e1a8c268157\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c6c610-8cf45f4d7-lq42w"
Sep 11 19:00:45.584671 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:45.584647    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe0fddea-982f-4f3f-81bd-7e1a8c268157-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c6c610-8cf45f4d7-lq42w\" (UID: \"fe0fddea-982f-4f3f-81bd-7e1a8c268157\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c6c610-8cf45f4d7-lq42w"
Sep 11 19:00:45.585982 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:45.585956    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3569b0b6-ae00-4c8c-b5f9-4990831c691d-kube-api-access-5s6tp" (OuterVolumeSpecName: "kube-api-access-5s6tp") pod "3569b0b6-ae00-4c8c-b5f9-4990831c691d" (UID: "3569b0b6-ae00-4c8c-b5f9-4990831c691d"). InnerVolumeSpecName "kube-api-access-5s6tp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:00:45.592001 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:45.591977    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gc8j4\" (UniqueName: \"kubernetes.io/projected/fe0fddea-982f-4f3f-81bd-7e1a8c268157-kube-api-access-gc8j4\") pod \"payload-processing-e2e-inf-a-c6c610-8cf45f4d7-lq42w\" (UID: \"fe0fddea-982f-4f3f-81bd-7e1a8c268157\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c6c610-8cf45f4d7-lq42w"
Sep 11 19:00:45.684733 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:45.684704    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5s6tp\" (UniqueName: \"kubernetes.io/projected/3569b0b6-ae00-4c8c-b5f9-4990831c691d-kube-api-access-5s6tp\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:00:45.726870 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:45.726828    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c6c610-8cf45f4d7-lq42w"
Sep 11 19:00:46.004114 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:46.004089    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c6c610-8cf45f4d7-lq42w"]
Sep 11 19:00:46.005839 ip-10-0-134-189 kubenswrapper[2578]: W0911 19:00:46.005808    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe0fddea_982f_4f3f_81bd_7e1a8c268157.slice/crio-6bdb42afab39d8b076ae1836f0a7457b2f58cd26b2d0087be85b140d77fc328b WatchSource:0}: Error finding container 6bdb42afab39d8b076ae1836f0a7457b2f58cd26b2d0087be85b140d77fc328b: Status 404 returned error can't find the container with id 6bdb42afab39d8b076ae1836f0a7457b2f58cd26b2d0087be85b140d77fc328b
Sep 11 19:00:46.350445 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:46.350413    2578 generic.go:356] "Generic (PLEG): container finished" podID="3569b0b6-ae00-4c8c-b5f9-4990831c691d" containerID="8b22b8ae5a2f1d30b2bdd057a59bccb24151ba8831e818addb202e71d98c465f" exitCode=137
Sep 11 19:00:46.350915 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:46.350466    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7797-053f84"
Sep 11 19:00:46.350915 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:46.350498    2578 scope.go:117] "RemoveContainer" containerID="8b22b8ae5a2f1d30b2bdd057a59bccb24151ba8831e818addb202e71d98c465f"
Sep 11 19:00:46.352351 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:46.352319    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-89e163-openshift-default-7774f9d49d-ddkg6" event={"ID":"0f145e25-8489-4f68-b919-c981f05e730b","Type":"ContainerStarted","Data":"76501c1fa721cf292975c7481206b43b935df0872994538afd6a02b8a80de777"}
Sep 11 19:00:46.354192 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:46.354170    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c6c610-8cf45f4d7-lq42w" event={"ID":"fe0fddea-982f-4f3f-81bd-7e1a8c268157","Type":"ContainerStarted","Data":"b83cd9fd8704d8ea0bbd8b88a9620afb04c6d18ca708e41220b37a37a4d6c25f"}
Sep 11 19:00:46.354282 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:46.354211    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c6c610-8cf45f4d7-lq42w" event={"ID":"fe0fddea-982f-4f3f-81bd-7e1a8c268157","Type":"ContainerStarted","Data":"6bdb42afab39d8b076ae1836f0a7457b2f58cd26b2d0087be85b140d77fc328b"}
Sep 11 19:00:46.354282 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:46.354236    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-c6c610-8cf45f4d7-lq42w"
Sep 11 19:00:46.359988 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:46.359968    2578 scope.go:117] "RemoveContainer" containerID="8b22b8ae5a2f1d30b2bdd057a59bccb24151ba8831e818addb202e71d98c465f"
Sep 11 19:00:46.360253 ip-10-0-134-189 kubenswrapper[2578]: E0911 19:00:46.360236    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b22b8ae5a2f1d30b2bdd057a59bccb24151ba8831e818addb202e71d98c465f\": container with ID starting with 8b22b8ae5a2f1d30b2bdd057a59bccb24151ba8831e818addb202e71d98c465f not found: ID does not exist" containerID="8b22b8ae5a2f1d30b2bdd057a59bccb24151ba8831e818addb202e71d98c465f"
Sep 11 19:00:46.360303 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:46.360261    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b22b8ae5a2f1d30b2bdd057a59bccb24151ba8831e818addb202e71d98c465f"} err="failed to get container status \"8b22b8ae5a2f1d30b2bdd057a59bccb24151ba8831e818addb202e71d98c465f\": rpc error: code = NotFound desc = could not find container \"8b22b8ae5a2f1d30b2bdd057a59bccb24151ba8831e818addb202e71d98c465f\": container with ID starting with 8b22b8ae5a2f1d30b2bdd057a59bccb24151ba8831e818addb202e71d98c465f not found: ID does not exist"
Sep 11 19:00:46.372527 ip-10-0-134-189 kubenswrapper[2578]: E0911 19:00:46.372489    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7797-053f84\" is forbidden: User \"system:node:ip-10-0-134-189.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-189.ec2.internal' and this object" podUID="3569b0b6-ae00-4c8c-b5f9-4990831c691d" pod="opendatahub/test-curl-7797-053f84"
Sep 11 19:00:46.373154 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:46.373116    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-89e163-openshift-default-7774f9d49d-ddkg6" podStartSLOduration=2.373103777 podStartE2EDuration="2.373103777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:00:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:00:46.370782962 +0000 UTC m=+722.448373516" watchObservedRunningTime="2026-09-11 19:00:46.373103777 +0000 UTC m=+722.450694385"
Sep 11 19:00:46.394458 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:46.394398    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-c6c610-8cf45f4d7-lq42w" podStartSLOduration=1.394377086 podStartE2EDuration="1.394377086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:00:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:00:46.393129188 +0000 UTC m=+722.470719756" watchObservedRunningTime="2026-09-11 19:00:46.394377086 +0000 UTC m=+722.471967648"
Sep 11 19:00:46.397992 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:46.397963    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3569b0b6-ae00-4c8c-b5f9-4990831c691d" path="/var/lib/kubelet/pods/3569b0b6-ae00-4c8c-b5f9-4990831c691d/volumes"
Sep 11 19:00:47.027564 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:47.027530    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-89e163-openshift-default-7774f9d49d-ddkg6"
Sep 11 19:00:47.032620 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:47.032567    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-89e163-openshift-default-7774f9d49d-ddkg6"
Sep 11 19:00:47.078229 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:47.078202    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-aff2b8e2-openshift-default-6997d749c6-wwp7k"]
Sep 11 19:00:47.083056 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:47.083034    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-aff2b8e2-openshift-default-6997d749c6-wwp7k"
Sep 11 19:00:47.085093 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:47.085074    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-aff2b8e2-openshift-default-dockercfg-pvlp9\""
Sep 11 19:00:47.106532 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:47.106500    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-aff2b8e2-openshift-default-6997d749c6-wwp7k"]
Sep 11 19:00:47.198769 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:47.198729    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bfd8b7a-2cd2-4121-aa13-110e6710a337-workload-socket\") pod \"e2e-ait-aff2b8e2-openshift-default-6997d749c6-wwp7k\" (UID: \"1bfd8b7a-2cd2-4121-aa13-110e6710a337\") " pod="openshift-ingress/e2e-ait-aff2b8e2-openshift-default-6997d749c6-wwp7k"
Sep 11 19:00:47.198769 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:47.198776    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bfd8b7a-2cd2-4121-aa13-110e6710a337-workload-certs\") pod \"e2e-ait-aff2b8e2-openshift-default-6997d749c6-wwp7k\" (UID: \"1bfd8b7a-2cd2-4121-aa13-110e6710a337\") " pod="openshift-ingress/e2e-ait-aff2b8e2-openshift-default-6997d749c6-wwp7k"
Sep 11 19:00:47.198982 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:47.198801    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnbc8\" (UniqueName: \"kubernetes.io/projected/1bfd8b7a-2cd2-4121-aa13-110e6710a337-kube-api-access-xnbc8\") pod \"e2e-ait-aff2b8e2-openshift-default-6997d749c6-wwp7k\" (UID: \"1bfd8b7a-2cd2-4121-aa13-110e6710a337\") " pod="openshift-ingress/e2e-ait-aff2b8e2-openshift-default-6997d749c6-wwp7k"
Sep 11 19:00:47.198982 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:47.198852    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bfd8b7a-2cd2-4121-aa13-110e6710a337-istio-data\") pod \"e2e-ait-aff2b8e2-openshift-default-6997d749c6-wwp7k\" (UID: \"1bfd8b7a-2cd2-4121-aa13-110e6710a337\") " pod="openshift-ingress/e2e-ait-aff2b8e2-openshift-default-6997d749c6-wwp7k"
Sep 11 19:00:47.198982 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:47.198876    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bfd8b7a-2cd2-4121-aa13-110e6710a337-istio-podinfo\") pod \"e2e-ait-aff2b8e2-openshift-default-6997d749c6-wwp7k\" (UID: \"1bfd8b7a-2cd2-4121-aa13-110e6710a337\") " pod="openshift-ingress/e2e-ait-aff2b8e2-openshift-default-6997d749c6-wwp7k"
Sep 11 19:00:47.198982 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:47.198902    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bfd8b7a-2cd2-4121-aa13-110e6710a337-istiod-ca-cert\") pod \"e2e-ait-aff2b8e2-openshift-default-6997d749c6-wwp7k\" (UID: \"1bfd8b7a-2cd2-4121-aa13-110e6710a337\") " pod="openshift-ingress/e2e-ait-aff2b8e2-openshift-default-6997d749c6-wwp7k"
Sep 11 19:00:47.198982 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:47.198940    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bfd8b7a-2cd2-4121-aa13-110e6710a337-istio-envoy\") pod \"e2e-ait-aff2b8e2-openshift-default-6997d749c6-wwp7k\" (UID: \"1bfd8b7a-2cd2-4121-aa13-110e6710a337\") " pod="openshift-ingress/e2e-ait-aff2b8e2-openshift-default-6997d749c6-wwp7k"
Sep 11 19:00:47.198982 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:47.198955    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bfd8b7a-2cd2-4121-aa13-110e6710a337-istio-token\") pod \"e2e-ait-aff2b8e2-openshift-default-6997d749c6-wwp7k\" (UID: \"1bfd8b7a-2cd2-4121-aa13-110e6710a337\") " pod="openshift-ingress/e2e-ait-aff2b8e2-openshift-default-6997d749c6-wwp7k"
Sep 11 19:00:47.198982 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:47.198972    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bfd8b7a-2cd2-4121-aa13-110e6710a337-credential-socket\") pod \"e2e-ait-aff2b8e2-openshift-default-6997d749c6-wwp7k\" (UID: \"1bfd8b7a-2cd2-4121-aa13-110e6710a337\") " pod="openshift-ingress/e2e-ait-aff2b8e2-openshift-default-6997d749c6-wwp7k"
Sep 11 19:00:47.300000 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:47.299898    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bfd8b7a-2cd2-4121-aa13-110e6710a337-istio-envoy\") pod \"e2e-ait-aff2b8e2-openshift-default-6997d749c6-wwp7k\" (UID: \"1bfd8b7a-2cd2-4121-aa13-110e6710a337\") " pod="openshift-ingress/e2e-ait-aff2b8e2-openshift-default-6997d749c6-wwp7k"
Sep 11 19:00:47.300757 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:47.300713    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bfd8b7a-2cd2-4121-aa13-110e6710a337-istio-token\") pod \"e2e-ait-aff2b8e2-openshift-default-6997d749c6-wwp7k\" (UID: \"1bfd8b7a-2cd2-4121-aa13-110e6710a337\") " pod="openshift-ingress/e2e-ait-aff2b8e2-openshift-default-6997d749c6-wwp7k"
Sep 11 19:00:47.301527 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:47.301490    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bfd8b7a-2cd2-4121-aa13-110e6710a337-credential-socket\") pod \"e2e-ait-aff2b8e2-openshift-default-6997d749c6-wwp7k\" (UID: \"1bfd8b7a-2cd2-4121-aa13-110e6710a337\") " pod="openshift-ingress/e2e-ait-aff2b8e2-openshift-default-6997d749c6-wwp7k"
Sep 11 19:00:47.301708 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:47.301649    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bfd8b7a-2cd2-4121-aa13-110e6710a337-workload-socket\") pod \"e2e-ait-aff2b8e2-openshift-default-6997d749c6-wwp7k\" (UID: \"1bfd8b7a-2cd2-4121-aa13-110e6710a337\") " pod="openshift-ingress/e2e-ait-aff2b8e2-openshift-default-6997d749c6-wwp7k"
Sep 11 19:00:47.301708 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:47.301697    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bfd8b7a-2cd2-4121-aa13-110e6710a337-workload-certs\") pod \"e2e-ait-aff2b8e2-openshift-default-6997d749c6-wwp7k\" (UID: \"1bfd8b7a-2cd2-4121-aa13-110e6710a337\") " pod="openshift-ingress/e2e-ait-aff2b8e2-openshift-default-6997d749c6-wwp7k"
Sep 11 19:00:47.301844 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:47.301748    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnbc8\" (UniqueName: \"kubernetes.io/projected/1bfd8b7a-2cd2-4121-aa13-110e6710a337-kube-api-access-xnbc8\") pod \"e2e-ait-aff2b8e2-openshift-default-6997d749c6-wwp7k\" (UID: \"1bfd8b7a-2cd2-4121-aa13-110e6710a337\") " pod="openshift-ingress/e2e-ait-aff2b8e2-openshift-default-6997d749c6-wwp7k"
Sep 11 19:00:47.301844 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:47.301800    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bfd8b7a-2cd2-4121-aa13-110e6710a337-istio-data\") pod \"e2e-ait-aff2b8e2-openshift-default-6997d749c6-wwp7k\" (UID: \"1bfd8b7a-2cd2-4121-aa13-110e6710a337\") " pod="openshift-ingress/e2e-ait-aff2b8e2-openshift-default-6997d749c6-wwp7k"
Sep 11 19:00:47.301844 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:47.301825    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bfd8b7a-2cd2-4121-aa13-110e6710a337-istio-podinfo\") pod \"e2e-ait-aff2b8e2-openshift-default-6997d749c6-wwp7k\" (UID: \"1bfd8b7a-2cd2-4121-aa13-110e6710a337\") " pod="openshift-ingress/e2e-ait-aff2b8e2-openshift-default-6997d749c6-wwp7k"
Sep 11 19:00:47.302006 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:47.301849    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bfd8b7a-2cd2-4121-aa13-110e6710a337-istiod-ca-cert\") pod \"e2e-ait-aff2b8e2-openshift-default-6997d749c6-wwp7k\" (UID: \"1bfd8b7a-2cd2-4121-aa13-110e6710a337\") " pod="openshift-ingress/e2e-ait-aff2b8e2-openshift-default-6997d749c6-wwp7k"
Sep 11 19:00:47.302006 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:47.301930    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bfd8b7a-2cd2-4121-aa13-110e6710a337-credential-socket\") pod \"e2e-ait-aff2b8e2-openshift-default-6997d749c6-wwp7k\" (UID: \"1bfd8b7a-2cd2-4121-aa13-110e6710a337\") " pod="openshift-ingress/e2e-ait-aff2b8e2-openshift-default-6997d749c6-wwp7k"
Sep 11 19:00:47.302500 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:47.302471    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bfd8b7a-2cd2-4121-aa13-110e6710a337-workload-socket\") pod \"e2e-ait-aff2b8e2-openshift-default-6997d749c6-wwp7k\" (UID: \"1bfd8b7a-2cd2-4121-aa13-110e6710a337\") " pod="openshift-ingress/e2e-ait-aff2b8e2-openshift-default-6997d749c6-wwp7k"
Sep 11 19:00:47.302615 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:47.302570    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bfd8b7a-2cd2-4121-aa13-110e6710a337-istiod-ca-cert\") pod \"e2e-ait-aff2b8e2-openshift-default-6997d749c6-wwp7k\" (UID: \"1bfd8b7a-2cd2-4121-aa13-110e6710a337\") " pod="openshift-ingress/e2e-ait-aff2b8e2-openshift-default-6997d749c6-wwp7k"
Sep 11 19:00:47.302802 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:47.302779    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bfd8b7a-2cd2-4121-aa13-110e6710a337-workload-certs\") pod \"e2e-ait-aff2b8e2-openshift-default-6997d749c6-wwp7k\" (UID: \"1bfd8b7a-2cd2-4121-aa13-110e6710a337\") " pod="openshift-ingress/e2e-ait-aff2b8e2-openshift-default-6997d749c6-wwp7k"
Sep 11 19:00:47.302902 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:47.302878    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bfd8b7a-2cd2-4121-aa13-110e6710a337-istio-data\") pod \"e2e-ait-aff2b8e2-openshift-default-6997d749c6-wwp7k\" (UID: \"1bfd8b7a-2cd2-4121-aa13-110e6710a337\") " pod="openshift-ingress/e2e-ait-aff2b8e2-openshift-default-6997d749c6-wwp7k"
Sep 11 19:00:47.303585 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:47.303557    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bfd8b7a-2cd2-4121-aa13-110e6710a337-istio-envoy\") pod \"e2e-ait-aff2b8e2-openshift-default-6997d749c6-wwp7k\" (UID: \"1bfd8b7a-2cd2-4121-aa13-110e6710a337\") " pod="openshift-ingress/e2e-ait-aff2b8e2-openshift-default-6997d749c6-wwp7k"
Sep 11 19:00:47.305829 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:47.305802    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bfd8b7a-2cd2-4121-aa13-110e6710a337-istio-podinfo\") pod \"e2e-ait-aff2b8e2-openshift-default-6997d749c6-wwp7k\" (UID: \"1bfd8b7a-2cd2-4121-aa13-110e6710a337\") " pod="openshift-ingress/e2e-ait-aff2b8e2-openshift-default-6997d749c6-wwp7k"
Sep 11 19:00:47.319704 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:47.319669    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnbc8\" (UniqueName: \"kubernetes.io/projected/1bfd8b7a-2cd2-4121-aa13-110e6710a337-kube-api-access-xnbc8\") pod \"e2e-ait-aff2b8e2-openshift-default-6997d749c6-wwp7k\" (UID: \"1bfd8b7a-2cd2-4121-aa13-110e6710a337\") " pod="openshift-ingress/e2e-ait-aff2b8e2-openshift-default-6997d749c6-wwp7k"
Sep 11 19:00:47.319875 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:47.319850    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bfd8b7a-2cd2-4121-aa13-110e6710a337-istio-token\") pod \"e2e-ait-aff2b8e2-openshift-default-6997d749c6-wwp7k\" (UID: \"1bfd8b7a-2cd2-4121-aa13-110e6710a337\") " pod="openshift-ingress/e2e-ait-aff2b8e2-openshift-default-6997d749c6-wwp7k"
Sep 11 19:00:47.361717 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:47.361687    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-89e163-openshift-default-7774f9d49d-ddkg6"
Sep 11 19:00:47.371184 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:47.371145    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-89e163-openshift-default-7774f9d49d-ddkg6"
Sep 11 19:00:47.392954 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:47.392926    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-aff2b8e2-openshift-default-6997d749c6-wwp7k"
Sep 11 19:00:47.585452 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:47.585405    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7797-5fbe22"
Sep 11 19:00:47.684937 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:47.684892    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-aff2b8e2-openshift-default-6997d749c6-wwp7k"]
Sep 11 19:00:47.688329 ip-10-0-134-189 kubenswrapper[2578]: W0911 19:00:47.688299    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bfd8b7a_2cd2_4121_aa13_110e6710a337.slice/crio-212d7f1ed57dfbc6277c1c7020d0c004523692b07c5df845f670b3313a699cf8 WatchSource:0}: Error finding container 212d7f1ed57dfbc6277c1c7020d0c004523692b07c5df845f670b3313a699cf8: Status 404 returned error can't find the container with id 212d7f1ed57dfbc6277c1c7020d0c004523692b07c5df845f670b3313a699cf8
Sep 11 19:00:47.690925 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:47.690897    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 19:00:47.691005 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:47.690968    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 19:00:47.691005 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:47.691003    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 19:00:47.706589 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:47.706568    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8qskj\" (UniqueName: \"kubernetes.io/projected/12b3df87-cbfa-411e-a919-dc0eb7b06970-kube-api-access-8qskj\") pod \"12b3df87-cbfa-411e-a919-dc0eb7b06970\" (UID: \"12b3df87-cbfa-411e-a919-dc0eb7b06970\") "
Sep 11 19:00:47.708894 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:47.708867    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12b3df87-cbfa-411e-a919-dc0eb7b06970-kube-api-access-8qskj" (OuterVolumeSpecName: "kube-api-access-8qskj") pod "12b3df87-cbfa-411e-a919-dc0eb7b06970" (UID: "12b3df87-cbfa-411e-a919-dc0eb7b06970"). InnerVolumeSpecName "kube-api-access-8qskj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:00:47.807560 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:47.807484    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qskj\" (UniqueName: \"kubernetes.io/projected/12b3df87-cbfa-411e-a919-dc0eb7b06970-kube-api-access-8qskj\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:00:48.369001 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:48.368954    2578 generic.go:356] "Generic (PLEG): container finished" podID="12b3df87-cbfa-411e-a919-dc0eb7b06970" containerID="7250e5fd8b254131880b2b73a681389913ccc11610d90bfd984a9960eca3e5c5" exitCode=137
Sep 11 19:00:48.369462 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:48.369065    2578 scope.go:117] "RemoveContainer" containerID="7250e5fd8b254131880b2b73a681389913ccc11610d90bfd984a9960eca3e5c5"
Sep 11 19:00:48.369462 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:48.369218    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7797-5fbe22"
Sep 11 19:00:48.373701 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:48.373667    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-aff2b8e2-openshift-default-6997d749c6-wwp7k" event={"ID":"1bfd8b7a-2cd2-4121-aa13-110e6710a337","Type":"ContainerStarted","Data":"e84a1a35cabc990a297c18bd74f362a2368efcb22e5c53d0e1acea79425c1933"}
Sep 11 19:00:48.373820 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:48.373757    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-aff2b8e2-openshift-default-6997d749c6-wwp7k" event={"ID":"1bfd8b7a-2cd2-4121-aa13-110e6710a337","Type":"ContainerStarted","Data":"212d7f1ed57dfbc6277c1c7020d0c004523692b07c5df845f670b3313a699cf8"}
Sep 11 19:00:48.390159 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:48.390134    2578 scope.go:117] "RemoveContainer" containerID="7250e5fd8b254131880b2b73a681389913ccc11610d90bfd984a9960eca3e5c5"
Sep 11 19:00:48.390566 ip-10-0-134-189 kubenswrapper[2578]: E0911 19:00:48.390516    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7250e5fd8b254131880b2b73a681389913ccc11610d90bfd984a9960eca3e5c5\": container with ID starting with 7250e5fd8b254131880b2b73a681389913ccc11610d90bfd984a9960eca3e5c5 not found: ID does not exist" containerID="7250e5fd8b254131880b2b73a681389913ccc11610d90bfd984a9960eca3e5c5"
Sep 11 19:00:48.390676 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:48.390576    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7250e5fd8b254131880b2b73a681389913ccc11610d90bfd984a9960eca3e5c5"} err="failed to get container status \"7250e5fd8b254131880b2b73a681389913ccc11610d90bfd984a9960eca3e5c5\": rpc error: code = NotFound desc = could not find container \"7250e5fd8b254131880b2b73a681389913ccc11610d90bfd984a9960eca3e5c5\": container with ID starting with 7250e5fd8b254131880b2b73a681389913ccc11610d90bfd984a9960eca3e5c5 not found: ID does not exist"
Sep 11 19:00:48.396930 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:48.396704    2578 patch_prober.go:28] interesting pod/e2e-ait-aff2b8e2-openshift-default-6997d749c6-wwp7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Sep 11 19:00:48.396930 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:48.396789    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-aff2b8e2-openshift-default-6997d749c6-wwp7k" podUID="1bfd8b7a-2cd2-4121-aa13-110e6710a337" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Sep 11 19:00:48.398026 ip-10-0-134-189 kubenswrapper[2578]: E0911 19:00:48.397993    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7797-5fbe22\" is forbidden: User \"system:node:ip-10-0-134-189.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-189.ec2.internal' and this object" podUID="12b3df87-cbfa-411e-a919-dc0eb7b06970" pod="opendatahub/test-curl-7797-5fbe22"
Sep 11 19:00:48.398831 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:48.398808    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12b3df87-cbfa-411e-a919-dc0eb7b06970" path="/var/lib/kubelet/pods/12b3df87-cbfa-411e-a919-dc0eb7b06970/volumes"
Sep 11 19:00:48.399044 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:48.399011    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-aff2b8e2-openshift-default-6997d749c6-wwp7k" podStartSLOduration=1.39899806 podStartE2EDuration="1.39899806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:00:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:00:48.396276579 +0000 UTC m=+724.473867153" watchObservedRunningTime="2026-09-11 19:00:48.39899806 +0000 UTC m=+724.476588665"
Sep 11 19:00:48.399097 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:48.399045    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-aff2b8e2-openshift-default-6997d749c6-wwp7k"
Sep 11 19:00:49.398016 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:49.397990    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-aff2b8e2-openshift-default-6997d749c6-wwp7k"
Sep 11 19:00:50.387509 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:50.387481    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-aff2b8e2-openshift-default-6997d749c6-wwp7k"
Sep 11 19:00:50.388533 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:50.388511    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-aff2b8e2-openshift-default-6997d749c6-wwp7k"
Sep 11 19:00:55.701911 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:55.701491    2578 patch_prober.go:28] interesting pod/e2e-worker-w4-c1aebc-openshift-default-5c87857866-4s5ww container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": read tcp 10.134.0.2:53668->10.134.0.31:15021: read: connection reset by peer" start-of-body=
Sep 11 19:00:55.701911 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:55.701569    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-c1aebc-openshift-default-5c87857866-4s5ww" podUID="f9b6e770-8c4d-4531-88d9-ab04cb50effc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": read tcp 10.134.0.2:53668->10.134.0.31:15021: read: connection reset by peer"
Sep 11 19:00:56.192198 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:56.192114    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-89e163-5cf97d8fcc-7qq94"]
Sep 11 19:00:56.194528 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:56.194506    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-89e163-5cf97d8fcc-7qq94"
Sep 11 19:00:56.197159 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:56.197136    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-89e163-dockercfg-mpb7t\""
Sep 11 19:00:56.197395 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:56.197380    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-89e163\""
Sep 11 19:00:56.211827 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:56.211805    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-89e163-5cf97d8fcc-7qq94"]
Sep 11 19:00:56.283738 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:56.283709    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2730645f-35cd-486a-b110-e4b2848593d0-plugins-config-volume\") pod \"payload-processing-e2e-autores-89e163-5cf97d8fcc-7qq94\" (UID: \"2730645f-35cd-486a-b110-e4b2848593d0\") " pod="openshift-ingress/payload-processing-e2e-autores-89e163-5cf97d8fcc-7qq94"
Sep 11 19:00:56.283880 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:56.283826    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zxx2\" (UniqueName: \"kubernetes.io/projected/2730645f-35cd-486a-b110-e4b2848593d0-kube-api-access-9zxx2\") pod \"payload-processing-e2e-autores-89e163-5cf97d8fcc-7qq94\" (UID: \"2730645f-35cd-486a-b110-e4b2848593d0\") " pod="openshift-ingress/payload-processing-e2e-autores-89e163-5cf97d8fcc-7qq94"
Sep 11 19:00:56.322508 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:56.322479    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-093f6c-openshift-default-b5fc96998-8cgj5"]
Sep 11 19:00:56.324838 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:56.324816    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-093f6c-openshift-default-b5fc96998-8cgj5"
Sep 11 19:00:56.327092 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:56.327071    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-093f6c-openshift-default-dockercfg-qkxtv\""
Sep 11 19:00:56.340949 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:56.340831    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-093f6c-openshift-default-b5fc96998-8cgj5"]
Sep 11 19:00:56.384339 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:56.384307    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2730645f-35cd-486a-b110-e4b2848593d0-plugins-config-volume\") pod \"payload-processing-e2e-autores-89e163-5cf97d8fcc-7qq94\" (UID: \"2730645f-35cd-486a-b110-e4b2848593d0\") " pod="openshift-ingress/payload-processing-e2e-autores-89e163-5cf97d8fcc-7qq94"
Sep 11 19:00:56.384496 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:56.384375    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zxx2\" (UniqueName: \"kubernetes.io/projected/2730645f-35cd-486a-b110-e4b2848593d0-kube-api-access-9zxx2\") pod \"payload-processing-e2e-autores-89e163-5cf97d8fcc-7qq94\" (UID: \"2730645f-35cd-486a-b110-e4b2848593d0\") " pod="openshift-ingress/payload-processing-e2e-autores-89e163-5cf97d8fcc-7qq94"
Sep 11 19:00:56.385032 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:56.385008    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2730645f-35cd-486a-b110-e4b2848593d0-plugins-config-volume\") pod \"payload-processing-e2e-autores-89e163-5cf97d8fcc-7qq94\" (UID: \"2730645f-35cd-486a-b110-e4b2848593d0\") " pod="openshift-ingress/payload-processing-e2e-autores-89e163-5cf97d8fcc-7qq94"
Sep 11 19:00:56.399930 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:56.399905    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zxx2\" (UniqueName: \"kubernetes.io/projected/2730645f-35cd-486a-b110-e4b2848593d0-kube-api-access-9zxx2\") pod \"payload-processing-e2e-autores-89e163-5cf97d8fcc-7qq94\" (UID: \"2730645f-35cd-486a-b110-e4b2848593d0\") " pod="openshift-ingress/payload-processing-e2e-autores-89e163-5cf97d8fcc-7qq94"
Sep 11 19:00:56.421148 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:56.421105    2578 generic.go:356] "Generic (PLEG): container finished" podID="f9b6e770-8c4d-4531-88d9-ab04cb50effc" containerID="236544bd6a2ada481215866aedad1fb4c3cee2a3555d6ec1e253198f982252b5" exitCode=0
Sep 11 19:00:56.421148 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:56.421151    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c1aebc-openshift-default-5c87857866-4s5ww" event={"ID":"f9b6e770-8c4d-4531-88d9-ab04cb50effc","Type":"ContainerDied","Data":"236544bd6a2ada481215866aedad1fb4c3cee2a3555d6ec1e253198f982252b5"}
Sep 11 19:00:56.421651 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:56.421631    2578 scope.go:117] "RemoveContainer" containerID="236544bd6a2ada481215866aedad1fb4c3cee2a3555d6ec1e253198f982252b5"
Sep 11 19:00:56.423107 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:56.423069    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 19:00:56.423208 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:56.423164    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 19:00:56.423257 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:56.423214    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 19:00:56.485111 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:56.485086    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2745e9fb-5682-4fdf-8394-d98e50ebd7a7-istio-envoy\") pod \"e2e-inf-b-093f6c-openshift-default-b5fc96998-8cgj5\" (UID: \"2745e9fb-5682-4fdf-8394-d98e50ebd7a7\") " pod="openshift-ingress/e2e-inf-b-093f6c-openshift-default-b5fc96998-8cgj5"
Sep 11 19:00:56.485238 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:56.485214    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2745e9fb-5682-4fdf-8394-d98e50ebd7a7-workload-socket\") pod \"e2e-inf-b-093f6c-openshift-default-b5fc96998-8cgj5\" (UID: \"2745e9fb-5682-4fdf-8394-d98e50ebd7a7\") " pod="openshift-ingress/e2e-inf-b-093f6c-openshift-default-b5fc96998-8cgj5"
Sep 11 19:00:56.485350 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:56.485286    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2745e9fb-5682-4fdf-8394-d98e50ebd7a7-istio-data\") pod \"e2e-inf-b-093f6c-openshift-default-b5fc96998-8cgj5\" (UID: \"2745e9fb-5682-4fdf-8394-d98e50ebd7a7\") " pod="openshift-ingress/e2e-inf-b-093f6c-openshift-default-b5fc96998-8cgj5"
Sep 11 19:00:56.485454 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:56.485403    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2745e9fb-5682-4fdf-8394-d98e50ebd7a7-istio-podinfo\") pod \"e2e-inf-b-093f6c-openshift-default-b5fc96998-8cgj5\" (UID: \"2745e9fb-5682-4fdf-8394-d98e50ebd7a7\") " pod="openshift-ingress/e2e-inf-b-093f6c-openshift-default-b5fc96998-8cgj5"
Sep 11 19:00:56.485499 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:56.485477    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2745e9fb-5682-4fdf-8394-d98e50ebd7a7-workload-certs\") pod \"e2e-inf-b-093f6c-openshift-default-b5fc96998-8cgj5\" (UID: \"2745e9fb-5682-4fdf-8394-d98e50ebd7a7\") " pod="openshift-ingress/e2e-inf-b-093f6c-openshift-default-b5fc96998-8cgj5"
Sep 11 19:00:56.485540 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:56.485511    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2745e9fb-5682-4fdf-8394-d98e50ebd7a7-istio-token\") pod \"e2e-inf-b-093f6c-openshift-default-b5fc96998-8cgj5\" (UID: \"2745e9fb-5682-4fdf-8394-d98e50ebd7a7\") " pod="openshift-ingress/e2e-inf-b-093f6c-openshift-default-b5fc96998-8cgj5"
Sep 11 19:00:56.485621 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:56.485541    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6g2w\" (UniqueName: \"kubernetes.io/projected/2745e9fb-5682-4fdf-8394-d98e50ebd7a7-kube-api-access-c6g2w\") pod \"e2e-inf-b-093f6c-openshift-default-b5fc96998-8cgj5\" (UID: \"2745e9fb-5682-4fdf-8394-d98e50ebd7a7\") " pod="openshift-ingress/e2e-inf-b-093f6c-openshift-default-b5fc96998-8cgj5"
Sep 11 19:00:56.485621 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:56.485570    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2745e9fb-5682-4fdf-8394-d98e50ebd7a7-istiod-ca-cert\") pod \"e2e-inf-b-093f6c-openshift-default-b5fc96998-8cgj5\" (UID: \"2745e9fb-5682-4fdf-8394-d98e50ebd7a7\") " pod="openshift-ingress/e2e-inf-b-093f6c-openshift-default-b5fc96998-8cgj5"
Sep 11 19:00:56.485728 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:56.485664    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2745e9fb-5682-4fdf-8394-d98e50ebd7a7-credential-socket\") pod \"e2e-inf-b-093f6c-openshift-default-b5fc96998-8cgj5\" (UID: \"2745e9fb-5682-4fdf-8394-d98e50ebd7a7\") " pod="openshift-ingress/e2e-inf-b-093f6c-openshift-default-b5fc96998-8cgj5"
Sep 11 19:00:56.504891 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:56.504866    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-89e163-5cf97d8fcc-7qq94"
Sep 11 19:00:56.586314 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:56.586274    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2745e9fb-5682-4fdf-8394-d98e50ebd7a7-workload-socket\") pod \"e2e-inf-b-093f6c-openshift-default-b5fc96998-8cgj5\" (UID: \"2745e9fb-5682-4fdf-8394-d98e50ebd7a7\") " pod="openshift-ingress/e2e-inf-b-093f6c-openshift-default-b5fc96998-8cgj5"
Sep 11 19:00:56.586472 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:56.586368    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2745e9fb-5682-4fdf-8394-d98e50ebd7a7-istio-data\") pod \"e2e-inf-b-093f6c-openshift-default-b5fc96998-8cgj5\" (UID: \"2745e9fb-5682-4fdf-8394-d98e50ebd7a7\") " pod="openshift-ingress/e2e-inf-b-093f6c-openshift-default-b5fc96998-8cgj5"
Sep 11 19:00:56.586472 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:56.586459    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2745e9fb-5682-4fdf-8394-d98e50ebd7a7-istio-podinfo\") pod \"e2e-inf-b-093f6c-openshift-default-b5fc96998-8cgj5\" (UID: \"2745e9fb-5682-4fdf-8394-d98e50ebd7a7\") " pod="openshift-ingress/e2e-inf-b-093f6c-openshift-default-b5fc96998-8cgj5"
Sep 11 19:00:56.586948 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:56.586889    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2745e9fb-5682-4fdf-8394-d98e50ebd7a7-workload-certs\") pod \"e2e-inf-b-093f6c-openshift-default-b5fc96998-8cgj5\" (UID: \"2745e9fb-5682-4fdf-8394-d98e50ebd7a7\") " pod="openshift-ingress/e2e-inf-b-093f6c-openshift-default-b5fc96998-8cgj5"
Sep 11 19:00:56.586948 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:56.586938    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2745e9fb-5682-4fdf-8394-d98e50ebd7a7-istio-token\") pod \"e2e-inf-b-093f6c-openshift-default-b5fc96998-8cgj5\" (UID: \"2745e9fb-5682-4fdf-8394-d98e50ebd7a7\") " pod="openshift-ingress/e2e-inf-b-093f6c-openshift-default-b5fc96998-8cgj5"
Sep 11 19:00:56.587082 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:56.586968    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6g2w\" (UniqueName: \"kubernetes.io/projected/2745e9fb-5682-4fdf-8394-d98e50ebd7a7-kube-api-access-c6g2w\") pod \"e2e-inf-b-093f6c-openshift-default-b5fc96998-8cgj5\" (UID: \"2745e9fb-5682-4fdf-8394-d98e50ebd7a7\") " pod="openshift-ingress/e2e-inf-b-093f6c-openshift-default-b5fc96998-8cgj5"
Sep 11 19:00:56.587082 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:56.587000    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2745e9fb-5682-4fdf-8394-d98e50ebd7a7-istiod-ca-cert\") pod \"e2e-inf-b-093f6c-openshift-default-b5fc96998-8cgj5\" (UID: \"2745e9fb-5682-4fdf-8394-d98e50ebd7a7\") " pod="openshift-ingress/e2e-inf-b-093f6c-openshift-default-b5fc96998-8cgj5"
Sep 11 19:00:56.587082 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:56.587036    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2745e9fb-5682-4fdf-8394-d98e50ebd7a7-credential-socket\") pod \"e2e-inf-b-093f6c-openshift-default-b5fc96998-8cgj5\" (UID: \"2745e9fb-5682-4fdf-8394-d98e50ebd7a7\") " pod="openshift-ingress/e2e-inf-b-093f6c-openshift-default-b5fc96998-8cgj5"
Sep 11 19:00:56.587082 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:56.587067    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2745e9fb-5682-4fdf-8394-d98e50ebd7a7-istio-envoy\") pod \"e2e-inf-b-093f6c-openshift-default-b5fc96998-8cgj5\" (UID: \"2745e9fb-5682-4fdf-8394-d98e50ebd7a7\") " pod="openshift-ingress/e2e-inf-b-093f6c-openshift-default-b5fc96998-8cgj5"
Sep 11 19:00:56.588121 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:56.586893    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2745e9fb-5682-4fdf-8394-d98e50ebd7a7-workload-socket\") pod \"e2e-inf-b-093f6c-openshift-default-b5fc96998-8cgj5\" (UID: \"2745e9fb-5682-4fdf-8394-d98e50ebd7a7\") " pod="openshift-ingress/e2e-inf-b-093f6c-openshift-default-b5fc96998-8cgj5"
Sep 11 19:00:56.588397 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:56.588246    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2745e9fb-5682-4fdf-8394-d98e50ebd7a7-istiod-ca-cert\") pod \"e2e-inf-b-093f6c-openshift-default-b5fc96998-8cgj5\" (UID: \"2745e9fb-5682-4fdf-8394-d98e50ebd7a7\") " pod="openshift-ingress/e2e-inf-b-093f6c-openshift-default-b5fc96998-8cgj5"
Sep 11 19:00:56.588397 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:56.588347    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2745e9fb-5682-4fdf-8394-d98e50ebd7a7-istio-data\") pod \"e2e-inf-b-093f6c-openshift-default-b5fc96998-8cgj5\" (UID: \"2745e9fb-5682-4fdf-8394-d98e50ebd7a7\") " pod="openshift-ingress/e2e-inf-b-093f6c-openshift-default-b5fc96998-8cgj5"
Sep 11 19:00:56.588711 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:56.588672    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2745e9fb-5682-4fdf-8394-d98e50ebd7a7-credential-socket\") pod \"e2e-inf-b-093f6c-openshift-default-b5fc96998-8cgj5\" (UID: \"2745e9fb-5682-4fdf-8394-d98e50ebd7a7\") " pod="openshift-ingress/e2e-inf-b-093f6c-openshift-default-b5fc96998-8cgj5"
Sep 11 19:00:56.588805 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:56.588715    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2745e9fb-5682-4fdf-8394-d98e50ebd7a7-workload-certs\") pod \"e2e-inf-b-093f6c-openshift-default-b5fc96998-8cgj5\" (UID: \"2745e9fb-5682-4fdf-8394-d98e50ebd7a7\") " pod="openshift-ingress/e2e-inf-b-093f6c-openshift-default-b5fc96998-8cgj5"
Sep 11 19:00:56.589803 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:56.589537    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2745e9fb-5682-4fdf-8394-d98e50ebd7a7-istio-podinfo\") pod \"e2e-inf-b-093f6c-openshift-default-b5fc96998-8cgj5\" (UID: \"2745e9fb-5682-4fdf-8394-d98e50ebd7a7\") " pod="openshift-ingress/e2e-inf-b-093f6c-openshift-default-b5fc96998-8cgj5"
Sep 11 19:00:56.591409 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:56.591387    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2745e9fb-5682-4fdf-8394-d98e50ebd7a7-istio-envoy\") pod \"e2e-inf-b-093f6c-openshift-default-b5fc96998-8cgj5\" (UID: \"2745e9fb-5682-4fdf-8394-d98e50ebd7a7\") " pod="openshift-ingress/e2e-inf-b-093f6c-openshift-default-b5fc96998-8cgj5"
Sep 11 19:00:56.597842 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:56.597822    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6g2w\" (UniqueName: \"kubernetes.io/projected/2745e9fb-5682-4fdf-8394-d98e50ebd7a7-kube-api-access-c6g2w\") pod \"e2e-inf-b-093f6c-openshift-default-b5fc96998-8cgj5\" (UID: \"2745e9fb-5682-4fdf-8394-d98e50ebd7a7\") " pod="openshift-ingress/e2e-inf-b-093f6c-openshift-default-b5fc96998-8cgj5"
Sep 11 19:00:56.598003 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:56.597982    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2745e9fb-5682-4fdf-8394-d98e50ebd7a7-istio-token\") pod \"e2e-inf-b-093f6c-openshift-default-b5fc96998-8cgj5\" (UID: \"2745e9fb-5682-4fdf-8394-d98e50ebd7a7\") " pod="openshift-ingress/e2e-inf-b-093f6c-openshift-default-b5fc96998-8cgj5"
Sep 11 19:00:56.635060 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:56.634840    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-093f6c-openshift-default-b5fc96998-8cgj5"
Sep 11 19:00:56.784867 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:56.784839    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-89e163-5cf97d8fcc-7qq94"]
Sep 11 19:00:56.790540 ip-10-0-134-189 kubenswrapper[2578]: W0911 19:00:56.790512    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2730645f_35cd_486a_b110_e4b2848593d0.slice/crio-f327d70b2b786b66390d7aca909cebea29d3effeb3068bd9790aba68a1dd35f2 WatchSource:0}: Error finding container f327d70b2b786b66390d7aca909cebea29d3effeb3068bd9790aba68a1dd35f2: Status 404 returned error can't find the container with id f327d70b2b786b66390d7aca909cebea29d3effeb3068bd9790aba68a1dd35f2
Sep 11 19:00:56.916899 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:56.915219    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-093f6c-openshift-default-b5fc96998-8cgj5"]
Sep 11 19:00:56.920414 ip-10-0-134-189 kubenswrapper[2578]: W0911 19:00:56.920380    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2745e9fb_5682_4fdf_8394_d98e50ebd7a7.slice/crio-64e99b9afcfa9770fa7d95f35597f14ca13ec2f4a70a7d487421bf8bbfe0e907 WatchSource:0}: Error finding container 64e99b9afcfa9770fa7d95f35597f14ca13ec2f4a70a7d487421bf8bbfe0e907: Status 404 returned error can't find the container with id 64e99b9afcfa9770fa7d95f35597f14ca13ec2f4a70a7d487421bf8bbfe0e907
Sep 11 19:00:56.923292 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:56.923257    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 19:00:56.923404 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:56.923334    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 19:00:56.923404 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:56.923376    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 19:00:57.364729 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:57.363787    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-c6c610-8cf45f4d7-lq42w"
Sep 11 19:00:57.428674 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:57.428637    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-89e163-5cf97d8fcc-7qq94" event={"ID":"2730645f-35cd-486a-b110-e4b2848593d0","Type":"ContainerStarted","Data":"2d027bfb9be4e7e2f5b2391aa7fe23c29b8707f2c08f9641709d71163f8fde49"}
Sep 11 19:00:57.428881 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:57.428863    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-89e163-5cf97d8fcc-7qq94"
Sep 11 19:00:57.429010 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:57.428965    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-89e163-5cf97d8fcc-7qq94" event={"ID":"2730645f-35cd-486a-b110-e4b2848593d0","Type":"ContainerStarted","Data":"f327d70b2b786b66390d7aca909cebea29d3effeb3068bd9790aba68a1dd35f2"}
Sep 11 19:00:57.431203 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:57.431174    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c1aebc-openshift-default-5c87857866-4s5ww" event={"ID":"f9b6e770-8c4d-4531-88d9-ab04cb50effc","Type":"ContainerStarted","Data":"6a30fd111fb853f849fa04f8a4479d325402f315685099084cac0e054f4d989b"}
Sep 11 19:00:57.433074 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:57.433047    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-093f6c-openshift-default-b5fc96998-8cgj5" event={"ID":"2745e9fb-5682-4fdf-8394-d98e50ebd7a7","Type":"ContainerStarted","Data":"3bc562a4e47e6d79a7524562f2b7cf0021605ea74d71141462f17be5665093ab"}
Sep 11 19:00:57.433196 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:57.433098    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-093f6c-openshift-default-b5fc96998-8cgj5" event={"ID":"2745e9fb-5682-4fdf-8394-d98e50ebd7a7","Type":"ContainerStarted","Data":"64e99b9afcfa9770fa7d95f35597f14ca13ec2f4a70a7d487421bf8bbfe0e907"}
Sep 11 19:00:57.450111 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:57.450041    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-89e163-5cf97d8fcc-7qq94" podStartSLOduration=1.450021642 podStartE2EDuration="1.450021642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:00:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:00:57.449754044 +0000 UTC m=+733.527344619" watchObservedRunningTime="2026-09-11 19:00:57.450021642 +0000 UTC m=+733.527612197"
Sep 11 19:00:57.507572 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:57.507496    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-093f6c-openshift-default-b5fc96998-8cgj5" podStartSLOduration=1.507473361 podStartE2EDuration="1.507473361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:00:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:00:57.506037747 +0000 UTC m=+733.583628317" watchObservedRunningTime="2026-09-11 19:00:57.507473361 +0000 UTC m=+733.585063916"
Sep 11 19:00:57.635750 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:57.635665    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-093f6c-openshift-default-b5fc96998-8cgj5"
Sep 11 19:00:57.658802 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:57.658761    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-093f6c-openshift-default-b5fc96998-8cgj5"
Sep 11 19:00:57.753262 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:57.753226    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-c1aebc-openshift-default-5c87857866-4s5ww"
Sep 11 19:00:57.753439 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:57.753358    2578 patch_prober.go:28] interesting pod/e2e-worker-w4-c1aebc-openshift-default-5c87857866-4s5ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body=
Sep 11 19:00:57.753439 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:57.753400    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c1aebc-openshift-default-5c87857866-4s5ww" podUID="f9b6e770-8c4d-4531-88d9-ab04cb50effc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused"
Sep 11 19:00:58.438176 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:58.438135    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-093f6c-openshift-default-b5fc96998-8cgj5"
Sep 11 19:00:58.439065 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:58.439046    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-093f6c-openshift-default-b5fc96998-8cgj5"
Sep 11 19:00:58.753700 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:58.753663    2578 patch_prober.go:28] interesting pod/e2e-worker-w4-c1aebc-openshift-default-5c87857866-4s5ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body=
Sep 11 19:00:58.753876 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:58.753728    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c1aebc-openshift-default-5c87857866-4s5ww" podUID="f9b6e770-8c4d-4531-88d9-ab04cb50effc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused"
Sep 11 19:00:59.753361 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:59.753326    2578 patch_prober.go:28] interesting pod/e2e-worker-w4-c1aebc-openshift-default-5c87857866-4s5ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body=
Sep 11 19:00:59.753754 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:00:59.753387    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c1aebc-openshift-default-5c87857866-4s5ww" podUID="f9b6e770-8c4d-4531-88d9-ab04cb50effc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused"
Sep 11 19:01:00.753378 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:00.753342    2578 patch_prober.go:28] interesting pod/e2e-worker-w4-c1aebc-openshift-default-5c87857866-4s5ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body=
Sep 11 19:01:00.753772 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:00.753399    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c1aebc-openshift-default-5c87857866-4s5ww" podUID="f9b6e770-8c4d-4531-88d9-ab04cb50effc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused"
Sep 11 19:01:01.754010 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:01.753974    2578 patch_prober.go:28] interesting pod/e2e-worker-w4-c1aebc-openshift-default-5c87857866-4s5ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body=
Sep 11 19:01:01.754363 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:01.754027    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c1aebc-openshift-default-5c87857866-4s5ww" podUID="f9b6e770-8c4d-4531-88d9-ab04cb50effc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused"
Sep 11 19:01:02.758662 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:02.758630    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-c1aebc-openshift-default-5c87857866-4s5ww"
Sep 11 19:01:02.759029 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:02.759014    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-c1aebc-openshift-default-5c87857866-4s5ww"
Sep 11 19:01:02.759979 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:02.759958    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-c1aebc-openshift-default-5c87857866-4s5ww"
Sep 11 19:01:08.439783 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:08.439749    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-89e163-5cf97d8fcc-7qq94"
Sep 11 19:01:15.542668 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:15.542630    2578 generic.go:356] "Generic (PLEG): container finished" podID="0f145e25-8489-4f68-b919-c981f05e730b" containerID="76501c1fa721cf292975c7481206b43b935df0872994538afd6a02b8a80de777" exitCode=0
Sep 11 19:01:15.543165 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:15.542705    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-89e163-openshift-default-7774f9d49d-ddkg6" event={"ID":"0f145e25-8489-4f68-b919-c981f05e730b","Type":"ContainerDied","Data":"76501c1fa721cf292975c7481206b43b935df0872994538afd6a02b8a80de777"}
Sep 11 19:01:15.543165 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:15.543145    2578 scope.go:117] "RemoveContainer" containerID="76501c1fa721cf292975c7481206b43b935df0872994538afd6a02b8a80de777"
Sep 11 19:01:15.544438 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:15.544408    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 19:01:15.544558 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:15.544440    2578 generic.go:356] "Generic (PLEG): container finished" podID="1bfd8b7a-2cd2-4121-aa13-110e6710a337" containerID="e84a1a35cabc990a297c18bd74f362a2368efcb22e5c53d0e1acea79425c1933" exitCode=0
Sep 11 19:01:15.544558 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:15.544475    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 19:01:15.544558 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:15.544495    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-aff2b8e2-openshift-default-6997d749c6-wwp7k" event={"ID":"1bfd8b7a-2cd2-4121-aa13-110e6710a337","Type":"ContainerDied","Data":"e84a1a35cabc990a297c18bd74f362a2368efcb22e5c53d0e1acea79425c1933"}
Sep 11 19:01:15.544558 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:15.544507    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 19:01:15.544919 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:15.544902    2578 scope.go:117] "RemoveContainer" containerID="e84a1a35cabc990a297c18bd74f362a2368efcb22e5c53d0e1acea79425c1933"
Sep 11 19:01:15.546011 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:15.545982    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 19:01:15.546109 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:15.546043    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 19:01:15.546168 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:15.546105    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 19:01:16.550833 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:16.550801    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-aff2b8e2-openshift-default-6997d749c6-wwp7k" event={"ID":"1bfd8b7a-2cd2-4121-aa13-110e6710a337","Type":"ContainerStarted","Data":"b5cf773ddc218287ff1aa7e70dbcd64e81ad0a8fabf0b6ff6c95a8c85b582d51"}
Sep 11 19:01:16.553455 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:16.553430    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-89e163-openshift-default-7774f9d49d-ddkg6" event={"ID":"0f145e25-8489-4f68-b919-c981f05e730b","Type":"ContainerStarted","Data":"5698c9420e53c244bcdf4d0268198b3033760d9196eba5d8dede32307b7de22c"}
Sep 11 19:01:17.028342 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:17.028292    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-89e163-openshift-default-7774f9d49d-ddkg6"
Sep 11 19:01:17.028747 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:17.028724    2578 patch_prober.go:28] interesting pod/e2e-autores-89e163-openshift-default-7774f9d49d-ddkg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Sep 11 19:01:17.028934 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:17.028903    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-89e163-openshift-default-7774f9d49d-ddkg6" podUID="0f145e25-8489-4f68-b919-c981f05e730b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Sep 11 19:01:17.361838 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:17.361745    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-89e163-openshift-default-7774f9d49d-ddkg6"
Sep 11 19:01:17.394696 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:17.394660    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-aff2b8e2-openshift-default-6997d749c6-wwp7k"
Sep 11 19:01:17.395071 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:17.394985    2578 patch_prober.go:28] interesting pod/e2e-ait-aff2b8e2-openshift-default-6997d749c6-wwp7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Sep 11 19:01:17.395311 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:17.395287    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-aff2b8e2-openshift-default-6997d749c6-wwp7k" podUID="1bfd8b7a-2cd2-4121-aa13-110e6710a337" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Sep 11 19:01:18.027465 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:18.027424    2578 patch_prober.go:28] interesting pod/e2e-autores-89e163-openshift-default-7774f9d49d-ddkg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Sep 11 19:01:18.028094 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:18.028061    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-89e163-openshift-default-7774f9d49d-ddkg6" podUID="0f145e25-8489-4f68-b919-c981f05e730b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Sep 11 19:01:18.393779 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:18.393687    2578 patch_prober.go:28] interesting pod/e2e-ait-aff2b8e2-openshift-default-6997d749c6-wwp7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Sep 11 19:01:18.394062 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:18.394032    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-aff2b8e2-openshift-default-6997d749c6-wwp7k" podUID="1bfd8b7a-2cd2-4121-aa13-110e6710a337" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Sep 11 19:01:19.027613 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:19.027505    2578 patch_prober.go:28] interesting pod/e2e-autores-89e163-openshift-default-7774f9d49d-ddkg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Sep 11 19:01:19.028363 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:19.027713    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-89e163-openshift-default-7774f9d49d-ddkg6" podUID="0f145e25-8489-4f68-b919-c981f05e730b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Sep 11 19:01:19.393896 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:19.393860    2578 patch_prober.go:28] interesting pod/e2e-ait-aff2b8e2-openshift-default-6997d749c6-wwp7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Sep 11 19:01:19.394177 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:19.394149    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-aff2b8e2-openshift-default-6997d749c6-wwp7k" podUID="1bfd8b7a-2cd2-4121-aa13-110e6710a337" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Sep 11 19:01:20.026946 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:20.026900    2578 patch_prober.go:28] interesting pod/e2e-autores-89e163-openshift-default-7774f9d49d-ddkg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Sep 11 19:01:20.027164 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:20.026968    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-89e163-openshift-default-7774f9d49d-ddkg6" podUID="0f145e25-8489-4f68-b919-c981f05e730b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Sep 11 19:01:20.389120 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:20.388212    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-aff2b8e2-openshift-default-6997d749c6-wwp7k"
Sep 11 19:01:20.397823 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:20.397769    2578 patch_prober.go:28] interesting pod/e2e-ait-aff2b8e2-openshift-default-6997d749c6-wwp7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Sep 11 19:01:20.398249 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:20.398214    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-aff2b8e2-openshift-default-6997d749c6-wwp7k" podUID="1bfd8b7a-2cd2-4121-aa13-110e6710a337" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Sep 11 19:01:21.027554 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:21.027516    2578 patch_prober.go:28] interesting pod/e2e-autores-89e163-openshift-default-7774f9d49d-ddkg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Sep 11 19:01:21.027765 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:21.027581    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-89e163-openshift-default-7774f9d49d-ddkg6" podUID="0f145e25-8489-4f68-b919-c981f05e730b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Sep 11 19:01:21.393982 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:21.393896    2578 patch_prober.go:28] interesting pod/e2e-ait-aff2b8e2-openshift-default-6997d749c6-wwp7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Sep 11 19:01:21.394941 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:21.394509    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-aff2b8e2-openshift-default-6997d749c6-wwp7k" podUID="1bfd8b7a-2cd2-4121-aa13-110e6710a337" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Sep 11 19:01:22.174392 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:22.174344    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-89e163-openshift-default-7774f9d49d-ddkg6"
Sep 11 19:01:22.176814 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:22.176739    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-89e163-openshift-default-7774f9d49d-ddkg6"
Sep 11 19:01:22.397876 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:22.397848    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-aff2b8e2-openshift-default-6997d749c6-wwp7k"
Sep 11 19:01:22.398806 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:22.398787    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-aff2b8e2-openshift-default-6997d749c6-wwp7k"
Sep 11 19:01:23.599076 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:23.599042    2578 generic.go:356] "Generic (PLEG): container finished" podID="2745e9fb-5682-4fdf-8394-d98e50ebd7a7" containerID="3bc562a4e47e6d79a7524562f2b7cf0021605ea74d71141462f17be5665093ab" exitCode=0
Sep 11 19:01:23.599504 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:23.599117    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-093f6c-openshift-default-b5fc96998-8cgj5" event={"ID":"2745e9fb-5682-4fdf-8394-d98e50ebd7a7","Type":"ContainerDied","Data":"3bc562a4e47e6d79a7524562f2b7cf0021605ea74d71141462f17be5665093ab"}
Sep 11 19:01:23.599568 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:23.599512    2578 scope.go:117] "RemoveContainer" containerID="3bc562a4e47e6d79a7524562f2b7cf0021605ea74d71141462f17be5665093ab"
Sep 11 19:01:23.600637 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:23.600609    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 19:01:23.600706 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:23.600678    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 19:01:23.600742 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:23.600711    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 19:01:24.606138 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:24.606104    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-093f6c-openshift-default-b5fc96998-8cgj5" event={"ID":"2745e9fb-5682-4fdf-8394-d98e50ebd7a7","Type":"ContainerStarted","Data":"858f77a5fadd28954d7c37746eed0d18af56c4ee0aed9e4b4b73220c5f0f7056"}
Sep 11 19:01:24.634990 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:24.634962    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-093f6c-openshift-default-b5fc96998-8cgj5"
Sep 11 19:01:24.635965 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:24.635937    2578 patch_prober.go:28] interesting pod/e2e-inf-b-093f6c-openshift-default-b5fc96998-8cgj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Sep 11 19:01:24.636114 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:24.636003    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-093f6c-openshift-default-b5fc96998-8cgj5" podUID="2745e9fb-5682-4fdf-8394-d98e50ebd7a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Sep 11 19:01:25.635820 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:25.635784    2578 patch_prober.go:28] interesting pod/e2e-inf-b-093f6c-openshift-default-b5fc96998-8cgj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Sep 11 19:01:25.636180 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:25.635841    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-093f6c-openshift-default-b5fc96998-8cgj5" podUID="2745e9fb-5682-4fdf-8394-d98e50ebd7a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Sep 11 19:01:26.220813 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:26.220783    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-c1aebc/simulator-w4-c1aebc-kserve-867479ffb6-76pm2"]
Sep 11 19:01:26.223082 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:26.223065    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c1aebc/simulator-w4-c1aebc-kserve-867479ffb6-76pm2"
Sep 11 19:01:26.225730 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:26.225707    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-c1aebc\"/\"kube-root-ca.crt\""
Sep 11 19:01:26.225839 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:26.225744    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-c1aebc\"/\"default-dockercfg-grsdl\""
Sep 11 19:01:26.225839 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:26.225757    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-c1aebc\"/\"simulator-w4-c1aebc-kserve-self-signed-certs\""
Sep 11 19:01:26.225839 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:26.225766    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-c1aebc\"/\"openshift-service-ca.crt\""
Sep 11 19:01:26.232081 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:26.232061    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c1aebc/simulator-w4-c1aebc-kserve-867479ffb6-76pm2"]
Sep 11 19:01:26.297547 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:26.297518    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84d9a7b6-0329-418e-adf5-03ac9a1aa33d-dshm\") pod \"simulator-w4-c1aebc-kserve-867479ffb6-76pm2\" (UID: \"84d9a7b6-0329-418e-adf5-03ac9a1aa33d\") " pod="e2e-models-e2e-worker-w4-c1aebc/simulator-w4-c1aebc-kserve-867479ffb6-76pm2"
Sep 11 19:01:26.297751 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:26.297560    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84d9a7b6-0329-418e-adf5-03ac9a1aa33d-tls-certs\") pod \"simulator-w4-c1aebc-kserve-867479ffb6-76pm2\" (UID: \"84d9a7b6-0329-418e-adf5-03ac9a1aa33d\") " pod="e2e-models-e2e-worker-w4-c1aebc/simulator-w4-c1aebc-kserve-867479ffb6-76pm2"
Sep 11 19:01:26.297751 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:26.297576    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84d9a7b6-0329-418e-adf5-03ac9a1aa33d-home\") pod \"simulator-w4-c1aebc-kserve-867479ffb6-76pm2\" (UID: \"84d9a7b6-0329-418e-adf5-03ac9a1aa33d\") " pod="e2e-models-e2e-worker-w4-c1aebc/simulator-w4-c1aebc-kserve-867479ffb6-76pm2"
Sep 11 19:01:26.297751 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:26.297668    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nplqg\" (UniqueName: \"kubernetes.io/projected/84d9a7b6-0329-418e-adf5-03ac9a1aa33d-kube-api-access-nplqg\") pod \"simulator-w4-c1aebc-kserve-867479ffb6-76pm2\" (UID: \"84d9a7b6-0329-418e-adf5-03ac9a1aa33d\") " pod="e2e-models-e2e-worker-w4-c1aebc/simulator-w4-c1aebc-kserve-867479ffb6-76pm2"
Sep 11 19:01:26.297751 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:26.297719    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84d9a7b6-0329-418e-adf5-03ac9a1aa33d-tmp-dir\") pod \"simulator-w4-c1aebc-kserve-867479ffb6-76pm2\" (UID: \"84d9a7b6-0329-418e-adf5-03ac9a1aa33d\") " pod="e2e-models-e2e-worker-w4-c1aebc/simulator-w4-c1aebc-kserve-867479ffb6-76pm2"
Sep 11 19:01:26.297900 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:26.297762    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84d9a7b6-0329-418e-adf5-03ac9a1aa33d-model-cache\") pod \"simulator-w4-c1aebc-kserve-867479ffb6-76pm2\" (UID: \"84d9a7b6-0329-418e-adf5-03ac9a1aa33d\") " pod="e2e-models-e2e-worker-w4-c1aebc/simulator-w4-c1aebc-kserve-867479ffb6-76pm2"
Sep 11 19:01:26.398883 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:26.398847    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84d9a7b6-0329-418e-adf5-03ac9a1aa33d-dshm\") pod \"simulator-w4-c1aebc-kserve-867479ffb6-76pm2\" (UID: \"84d9a7b6-0329-418e-adf5-03ac9a1aa33d\") " pod="e2e-models-e2e-worker-w4-c1aebc/simulator-w4-c1aebc-kserve-867479ffb6-76pm2"
Sep 11 19:01:26.399073 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:26.398893    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84d9a7b6-0329-418e-adf5-03ac9a1aa33d-tls-certs\") pod \"simulator-w4-c1aebc-kserve-867479ffb6-76pm2\" (UID: \"84d9a7b6-0329-418e-adf5-03ac9a1aa33d\") " pod="e2e-models-e2e-worker-w4-c1aebc/simulator-w4-c1aebc-kserve-867479ffb6-76pm2"
Sep 11 19:01:26.399073 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:26.398917    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84d9a7b6-0329-418e-adf5-03ac9a1aa33d-home\") pod \"simulator-w4-c1aebc-kserve-867479ffb6-76pm2\" (UID: \"84d9a7b6-0329-418e-adf5-03ac9a1aa33d\") " pod="e2e-models-e2e-worker-w4-c1aebc/simulator-w4-c1aebc-kserve-867479ffb6-76pm2"
Sep 11 19:01:26.399073 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:26.398961    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nplqg\" (UniqueName: \"kubernetes.io/projected/84d9a7b6-0329-418e-adf5-03ac9a1aa33d-kube-api-access-nplqg\") pod \"simulator-w4-c1aebc-kserve-867479ffb6-76pm2\" (UID: \"84d9a7b6-0329-418e-adf5-03ac9a1aa33d\") " pod="e2e-models-e2e-worker-w4-c1aebc/simulator-w4-c1aebc-kserve-867479ffb6-76pm2"
Sep 11 19:01:26.399073 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:26.398992    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84d9a7b6-0329-418e-adf5-03ac9a1aa33d-tmp-dir\") pod \"simulator-w4-c1aebc-kserve-867479ffb6-76pm2\" (UID: \"84d9a7b6-0329-418e-adf5-03ac9a1aa33d\") " pod="e2e-models-e2e-worker-w4-c1aebc/simulator-w4-c1aebc-kserve-867479ffb6-76pm2"
Sep 11 19:01:26.399073 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:26.399036    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84d9a7b6-0329-418e-adf5-03ac9a1aa33d-model-cache\") pod \"simulator-w4-c1aebc-kserve-867479ffb6-76pm2\" (UID: \"84d9a7b6-0329-418e-adf5-03ac9a1aa33d\") " pod="e2e-models-e2e-worker-w4-c1aebc/simulator-w4-c1aebc-kserve-867479ffb6-76pm2"
Sep 11 19:01:26.399381 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:26.399358    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84d9a7b6-0329-418e-adf5-03ac9a1aa33d-home\") pod \"simulator-w4-c1aebc-kserve-867479ffb6-76pm2\" (UID: \"84d9a7b6-0329-418e-adf5-03ac9a1aa33d\") " pod="e2e-models-e2e-worker-w4-c1aebc/simulator-w4-c1aebc-kserve-867479ffb6-76pm2"
Sep 11 19:01:26.399444 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:26.399400    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84d9a7b6-0329-418e-adf5-03ac9a1aa33d-model-cache\") pod \"simulator-w4-c1aebc-kserve-867479ffb6-76pm2\" (UID: \"84d9a7b6-0329-418e-adf5-03ac9a1aa33d\") " pod="e2e-models-e2e-worker-w4-c1aebc/simulator-w4-c1aebc-kserve-867479ffb6-76pm2"
Sep 11 19:01:26.399494 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:26.399476    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84d9a7b6-0329-418e-adf5-03ac9a1aa33d-tmp-dir\") pod \"simulator-w4-c1aebc-kserve-867479ffb6-76pm2\" (UID: \"84d9a7b6-0329-418e-adf5-03ac9a1aa33d\") " pod="e2e-models-e2e-worker-w4-c1aebc/simulator-w4-c1aebc-kserve-867479ffb6-76pm2"
Sep 11 19:01:26.401278 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:26.401256    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84d9a7b6-0329-418e-adf5-03ac9a1aa33d-dshm\") pod \"simulator-w4-c1aebc-kserve-867479ffb6-76pm2\" (UID: \"84d9a7b6-0329-418e-adf5-03ac9a1aa33d\") " pod="e2e-models-e2e-worker-w4-c1aebc/simulator-w4-c1aebc-kserve-867479ffb6-76pm2"
Sep 11 19:01:26.401530 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:26.401511    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84d9a7b6-0329-418e-adf5-03ac9a1aa33d-tls-certs\") pod \"simulator-w4-c1aebc-kserve-867479ffb6-76pm2\" (UID: \"84d9a7b6-0329-418e-adf5-03ac9a1aa33d\") " pod="e2e-models-e2e-worker-w4-c1aebc/simulator-w4-c1aebc-kserve-867479ffb6-76pm2"
Sep 11 19:01:26.406509 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:26.406488    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nplqg\" (UniqueName: \"kubernetes.io/projected/84d9a7b6-0329-418e-adf5-03ac9a1aa33d-kube-api-access-nplqg\") pod \"simulator-w4-c1aebc-kserve-867479ffb6-76pm2\" (UID: \"84d9a7b6-0329-418e-adf5-03ac9a1aa33d\") " pod="e2e-models-e2e-worker-w4-c1aebc/simulator-w4-c1aebc-kserve-867479ffb6-76pm2"
Sep 11 19:01:26.532668 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:26.532575    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c1aebc/simulator-w4-c1aebc-kserve-867479ffb6-76pm2"
Sep 11 19:01:26.636298 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:26.636223    2578 patch_prober.go:28] interesting pod/e2e-inf-b-093f6c-openshift-default-b5fc96998-8cgj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Sep 11 19:01:26.636298 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:26.636270    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-093f6c-openshift-default-b5fc96998-8cgj5" podUID="2745e9fb-5682-4fdf-8394-d98e50ebd7a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Sep 11 19:01:26.828683 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:26.828661    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c1aebc/simulator-w4-c1aebc-kserve-867479ffb6-76pm2"]
Sep 11 19:01:26.830862 ip-10-0-134-189 kubenswrapper[2578]: W0911 19:01:26.830829    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod84d9a7b6_0329_418e_adf5_03ac9a1aa33d.slice/crio-8fefb69c375204550b94e6a628407afffe8540446fd17fe4493e0b04611f888e WatchSource:0}: Error finding container 8fefb69c375204550b94e6a628407afffe8540446fd17fe4493e0b04611f888e: Status 404 returned error can't find the container with id 8fefb69c375204550b94e6a628407afffe8540446fd17fe4493e0b04611f888e
Sep 11 19:01:27.624619 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:27.624567    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c1aebc/simulator-w4-c1aebc-kserve-867479ffb6-76pm2" event={"ID":"84d9a7b6-0329-418e-adf5-03ac9a1aa33d","Type":"ContainerStarted","Data":"46e9e33c40887bcba635964cd81bf431bb6d3b3e3696fa58799eb2f193a600d9"}
Sep 11 19:01:27.624619 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:27.624625    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c1aebc/simulator-w4-c1aebc-kserve-867479ffb6-76pm2" event={"ID":"84d9a7b6-0329-418e-adf5-03ac9a1aa33d","Type":"ContainerStarted","Data":"8fefb69c375204550b94e6a628407afffe8540446fd17fe4493e0b04611f888e"}
Sep 11 19:01:27.635809 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:27.635781    2578 patch_prober.go:28] interesting pod/e2e-inf-b-093f6c-openshift-default-b5fc96998-8cgj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Sep 11 19:01:27.635927 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:27.635832    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-093f6c-openshift-default-b5fc96998-8cgj5" podUID="2745e9fb-5682-4fdf-8394-d98e50ebd7a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Sep 11 19:01:27.642502 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:27.642461    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-c1aebc/simulator-w4-c1aebc-kserve-867479ffb6-76pm2" podStartSLOduration=1.6424468349999999 podStartE2EDuration="1.642446835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:01:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:01:27.64033928 +0000 UTC m=+763.717929872" watchObservedRunningTime="2026-09-11 19:01:27.642446835 +0000 UTC m=+763.720037387"
Sep 11 19:01:28.438523 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:28.438488    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-093f6c-openshift-default-b5fc96998-8cgj5"
Sep 11 19:01:28.635965 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:28.635926    2578 patch_prober.go:28] interesting pod/e2e-inf-b-093f6c-openshift-default-b5fc96998-8cgj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Sep 11 19:01:28.636156 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:28.635993    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-093f6c-openshift-default-b5fc96998-8cgj5" podUID="2745e9fb-5682-4fdf-8394-d98e50ebd7a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Sep 11 19:01:29.635214 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:29.635179    2578 patch_prober.go:28] interesting pod/e2e-inf-b-093f6c-openshift-default-b5fc96998-8cgj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Sep 11 19:01:29.635628 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:29.635246    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-093f6c-openshift-default-b5fc96998-8cgj5" podUID="2745e9fb-5682-4fdf-8394-d98e50ebd7a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Sep 11 19:01:30.639942 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:30.639916    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-093f6c-openshift-default-b5fc96998-8cgj5"
Sep 11 19:01:30.640822 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:30.640803    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-093f6c-openshift-default-b5fc96998-8cgj5"
Sep 11 19:01:32.543008 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:32.542972    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-aff2b8e2-openshift-default-6997d749c6-wwp7k"]
Sep 11 19:01:32.543433 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:32.543180    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-aff2b8e2-openshift-default-6997d749c6-wwp7k" podUID="1bfd8b7a-2cd2-4121-aa13-110e6710a337" containerName="istio-proxy" containerID="cri-o://b5cf773ddc218287ff1aa7e70dbcd64e81ad0a8fabf0b6ff6c95a8c85b582d51" gracePeriod=30
Sep 11 19:01:36.532750 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:36.532711    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-c1aebc/simulator-w4-c1aebc-kserve-867479ffb6-76pm2"
Sep 11 19:01:36.533084 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:36.532759    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c1aebc/simulator-w4-c1aebc-kserve-867479ffb6-76pm2"
Sep 11 19:01:36.545058 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:36.545035    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-c1aebc/simulator-w4-c1aebc-kserve-867479ffb6-76pm2"
Sep 11 19:01:36.681118 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:36.681090    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c1aebc/simulator-w4-c1aebc-kserve-867479ffb6-76pm2"
Sep 11 19:01:37.400716 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:37.400673    2578 patch_prober.go:28] interesting pod/e2e-ait-aff2b8e2-openshift-default-6997d749c6-wwp7k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Sep 11 19:01:37.400912 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:37.400748    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-aff2b8e2-openshift-default-6997d749c6-wwp7k" podUID="1bfd8b7a-2cd2-4121-aa13-110e6710a337" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 19:01:37.677189 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:37.677157    2578 generic.go:356] "Generic (PLEG): container finished" podID="1bfd8b7a-2cd2-4121-aa13-110e6710a337" containerID="b5cf773ddc218287ff1aa7e70dbcd64e81ad0a8fabf0b6ff6c95a8c85b582d51" exitCode=0
Sep 11 19:01:37.677627 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:37.677231    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-aff2b8e2-openshift-default-6997d749c6-wwp7k" event={"ID":"1bfd8b7a-2cd2-4121-aa13-110e6710a337","Type":"ContainerDied","Data":"b5cf773ddc218287ff1aa7e70dbcd64e81ad0a8fabf0b6ff6c95a8c85b582d51"}
Sep 11 19:01:37.677627 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:37.677349    2578 scope.go:117] "RemoveContainer" containerID="e84a1a35cabc990a297c18bd74f362a2368efcb22e5c53d0e1acea79425c1933"
Sep 11 19:01:37.795913 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:37.795885    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-aff2b8e2-openshift-default-6997d749c6-wwp7k"
Sep 11 19:01:37.905627 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:37.905570    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bfd8b7a-2cd2-4121-aa13-110e6710a337-istio-podinfo\") pod \"1bfd8b7a-2cd2-4121-aa13-110e6710a337\" (UID: \"1bfd8b7a-2cd2-4121-aa13-110e6710a337\") "
Sep 11 19:01:37.905811 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:37.905650    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bfd8b7a-2cd2-4121-aa13-110e6710a337-istiod-ca-cert\") pod \"1bfd8b7a-2cd2-4121-aa13-110e6710a337\" (UID: \"1bfd8b7a-2cd2-4121-aa13-110e6710a337\") "
Sep 11 19:01:37.905811 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:37.905677    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bfd8b7a-2cd2-4121-aa13-110e6710a337-istio-token\") pod \"1bfd8b7a-2cd2-4121-aa13-110e6710a337\" (UID: \"1bfd8b7a-2cd2-4121-aa13-110e6710a337\") "
Sep 11 19:01:37.905811 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:37.905744    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xnbc8\" (UniqueName: \"kubernetes.io/projected/1bfd8b7a-2cd2-4121-aa13-110e6710a337-kube-api-access-xnbc8\") pod \"1bfd8b7a-2cd2-4121-aa13-110e6710a337\" (UID: \"1bfd8b7a-2cd2-4121-aa13-110e6710a337\") "
Sep 11 19:01:37.905811 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:37.905775    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bfd8b7a-2cd2-4121-aa13-110e6710a337-istio-envoy\") pod \"1bfd8b7a-2cd2-4121-aa13-110e6710a337\" (UID: \"1bfd8b7a-2cd2-4121-aa13-110e6710a337\") "
Sep 11 19:01:37.905811 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:37.905807    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bfd8b7a-2cd2-4121-aa13-110e6710a337-credential-socket\") pod \"1bfd8b7a-2cd2-4121-aa13-110e6710a337\" (UID: \"1bfd8b7a-2cd2-4121-aa13-110e6710a337\") "
Sep 11 19:01:37.906067 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:37.905858    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bfd8b7a-2cd2-4121-aa13-110e6710a337-workload-certs\") pod \"1bfd8b7a-2cd2-4121-aa13-110e6710a337\" (UID: \"1bfd8b7a-2cd2-4121-aa13-110e6710a337\") "
Sep 11 19:01:37.906067 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:37.905896    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bfd8b7a-2cd2-4121-aa13-110e6710a337-istio-data\") pod \"1bfd8b7a-2cd2-4121-aa13-110e6710a337\" (UID: \"1bfd8b7a-2cd2-4121-aa13-110e6710a337\") "
Sep 11 19:01:37.906067 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:37.905954    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bfd8b7a-2cd2-4121-aa13-110e6710a337-workload-socket\") pod \"1bfd8b7a-2cd2-4121-aa13-110e6710a337\" (UID: \"1bfd8b7a-2cd2-4121-aa13-110e6710a337\") "
Sep 11 19:01:37.906210 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:37.906091    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bfd8b7a-2cd2-4121-aa13-110e6710a337-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1bfd8b7a-2cd2-4121-aa13-110e6710a337" (UID: "1bfd8b7a-2cd2-4121-aa13-110e6710a337"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:01:37.906269 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:37.906238    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bfd8b7a-2cd2-4121-aa13-110e6710a337-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1bfd8b7a-2cd2-4121-aa13-110e6710a337" (UID: "1bfd8b7a-2cd2-4121-aa13-110e6710a337"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:01:37.906269 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:37.906101    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bfd8b7a-2cd2-4121-aa13-110e6710a337-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1bfd8b7a-2cd2-4121-aa13-110e6710a337" (UID: "1bfd8b7a-2cd2-4121-aa13-110e6710a337"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 19:01:37.906374 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:37.906322    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bfd8b7a-2cd2-4121-aa13-110e6710a337-istio-data" (OuterVolumeSpecName: "istio-data") pod "1bfd8b7a-2cd2-4121-aa13-110e6710a337" (UID: "1bfd8b7a-2cd2-4121-aa13-110e6710a337"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:01:37.906374 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:37.906337    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bfd8b7a-2cd2-4121-aa13-110e6710a337-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1bfd8b7a-2cd2-4121-aa13-110e6710a337" (UID: "1bfd8b7a-2cd2-4121-aa13-110e6710a337"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:01:37.906657 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:37.906637    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bfd8b7a-2cd2-4121-aa13-110e6710a337-istiod-ca-cert\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:01:37.906723 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:37.906663    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bfd8b7a-2cd2-4121-aa13-110e6710a337-credential-socket\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:01:37.906723 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:37.906679    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bfd8b7a-2cd2-4121-aa13-110e6710a337-workload-certs\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:01:37.906723 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:37.906691    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bfd8b7a-2cd2-4121-aa13-110e6710a337-istio-data\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:01:37.906723 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:37.906705    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bfd8b7a-2cd2-4121-aa13-110e6710a337-workload-socket\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:01:37.908204 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:37.908173    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bfd8b7a-2cd2-4121-aa13-110e6710a337-istio-token" (OuterVolumeSpecName: "istio-token") pod "1bfd8b7a-2cd2-4121-aa13-110e6710a337" (UID: "1bfd8b7a-2cd2-4121-aa13-110e6710a337"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:01:37.908204 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:37.908169    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bfd8b7a-2cd2-4121-aa13-110e6710a337-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1bfd8b7a-2cd2-4121-aa13-110e6710a337" (UID: "1bfd8b7a-2cd2-4121-aa13-110e6710a337"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:01:37.908365 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:37.908227    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bfd8b7a-2cd2-4121-aa13-110e6710a337-kube-api-access-xnbc8" (OuterVolumeSpecName: "kube-api-access-xnbc8") pod "1bfd8b7a-2cd2-4121-aa13-110e6710a337" (UID: "1bfd8b7a-2cd2-4121-aa13-110e6710a337"). InnerVolumeSpecName "kube-api-access-xnbc8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:01:37.908365 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:37.908281    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1bfd8b7a-2cd2-4121-aa13-110e6710a337-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1bfd8b7a-2cd2-4121-aa13-110e6710a337" (UID: "1bfd8b7a-2cd2-4121-aa13-110e6710a337"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 19:01:38.007640 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:38.007583    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bfd8b7a-2cd2-4121-aa13-110e6710a337-istio-podinfo\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:01:38.007640 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:38.007634    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bfd8b7a-2cd2-4121-aa13-110e6710a337-istio-token\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:01:38.007843 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:38.007648    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnbc8\" (UniqueName: \"kubernetes.io/projected/1bfd8b7a-2cd2-4121-aa13-110e6710a337-kube-api-access-xnbc8\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:01:38.007843 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:38.007663    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bfd8b7a-2cd2-4121-aa13-110e6710a337-istio-envoy\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:01:38.568583 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:38.568548    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c158570a-5786998d6b-v874h"]
Sep 11 19:01:38.568920 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:38.568908    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bfd8b7a-2cd2-4121-aa13-110e6710a337" containerName="istio-proxy"
Sep 11 19:01:38.568973 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:38.568922    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bfd8b7a-2cd2-4121-aa13-110e6710a337" containerName="istio-proxy"
Sep 11 19:01:38.569008 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:38.568998    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1bfd8b7a-2cd2-4121-aa13-110e6710a337" containerName="istio-proxy"
Sep 11 19:01:38.569008 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:38.569006    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1bfd8b7a-2cd2-4121-aa13-110e6710a337" containerName="istio-proxy"
Sep 11 19:01:38.570923 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:38.570907    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c158570a-5786998d6b-v874h"
Sep 11 19:01:38.572947 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:38.572923    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-c158570a-dockercfg-snq79\""
Sep 11 19:01:38.573135 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:38.573115    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-c158570a\""
Sep 11 19:01:38.581249 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:38.581228    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c158570a-5786998d6b-v874h"]
Sep 11 19:01:38.684876 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:38.684791    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-aff2b8e2-openshift-default-6997d749c6-wwp7k"
Sep 11 19:01:38.684876 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:38.684836    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-aff2b8e2-openshift-default-6997d749c6-wwp7k" event={"ID":"1bfd8b7a-2cd2-4121-aa13-110e6710a337","Type":"ContainerDied","Data":"212d7f1ed57dfbc6277c1c7020d0c004523692b07c5df845f670b3313a699cf8"}
Sep 11 19:01:38.685308 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:38.684893    2578 scope.go:117] "RemoveContainer" containerID="b5cf773ddc218287ff1aa7e70dbcd64e81ad0a8fabf0b6ff6c95a8c85b582d51"
Sep 11 19:01:38.699037 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:38.699013    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-aff2b8e2-openshift-default-6997d749c6-wwp7k"]
Sep 11 19:01:38.703347 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:38.703326    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-aff2b8e2-openshift-default-6997d749c6-wwp7k"]
Sep 11 19:01:38.713865 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:38.713846    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dcbb91c-dac8-4e6f-8c44-5cc114f83737-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-c158570a-5786998d6b-v874h\" (UID: \"4dcbb91c-dac8-4e6f-8c44-5cc114f83737\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c158570a-5786998d6b-v874h"
Sep 11 19:01:38.713976 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:38.713960    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvrxs\" (UniqueName: \"kubernetes.io/projected/4dcbb91c-dac8-4e6f-8c44-5cc114f83737-kube-api-access-kvrxs\") pod \"payload-pre-processing-e2e-migrate-c158570a-5786998d6b-v874h\" (UID: \"4dcbb91c-dac8-4e6f-8c44-5cc114f83737\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c158570a-5786998d6b-v874h"
Sep 11 19:01:38.814966 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:38.814938    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvrxs\" (UniqueName: \"kubernetes.io/projected/4dcbb91c-dac8-4e6f-8c44-5cc114f83737-kube-api-access-kvrxs\") pod \"payload-pre-processing-e2e-migrate-c158570a-5786998d6b-v874h\" (UID: \"4dcbb91c-dac8-4e6f-8c44-5cc114f83737\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c158570a-5786998d6b-v874h"
Sep 11 19:01:38.815085 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:38.814990    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dcbb91c-dac8-4e6f-8c44-5cc114f83737-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-c158570a-5786998d6b-v874h\" (UID: \"4dcbb91c-dac8-4e6f-8c44-5cc114f83737\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c158570a-5786998d6b-v874h"
Sep 11 19:01:38.815652 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:38.815633    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dcbb91c-dac8-4e6f-8c44-5cc114f83737-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-c158570a-5786998d6b-v874h\" (UID: \"4dcbb91c-dac8-4e6f-8c44-5cc114f83737\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c158570a-5786998d6b-v874h"
Sep 11 19:01:38.825134 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:38.825107    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvrxs\" (UniqueName: \"kubernetes.io/projected/4dcbb91c-dac8-4e6f-8c44-5cc114f83737-kube-api-access-kvrxs\") pod \"payload-pre-processing-e2e-migrate-c158570a-5786998d6b-v874h\" (UID: \"4dcbb91c-dac8-4e6f-8c44-5cc114f83737\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c158570a-5786998d6b-v874h"
Sep 11 19:01:38.881888 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:38.881867    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c158570a-5786998d6b-v874h"
Sep 11 19:01:39.166422 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:39.166398    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c158570a-5786998d6b-v874h"]
Sep 11 19:01:39.168579 ip-10-0-134-189 kubenswrapper[2578]: W0911 19:01:39.168555    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dcbb91c_dac8_4e6f_8c44_5cc114f83737.slice/crio-801cd8dceea98680806f13d8b3df5e4526eff8473fc787308513b0780c12ee46 WatchSource:0}: Error finding container 801cd8dceea98680806f13d8b3df5e4526eff8473fc787308513b0780c12ee46: Status 404 returned error can't find the container with id 801cd8dceea98680806f13d8b3df5e4526eff8473fc787308513b0780c12ee46
Sep 11 19:01:39.692499 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:39.692458    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c158570a-5786998d6b-v874h" event={"ID":"4dcbb91c-dac8-4e6f-8c44-5cc114f83737","Type":"ContainerStarted","Data":"79d7e90f107c7d745179558308d6e76b7c95a483c24776099bd82a9d2bfcb892"}
Sep 11 19:01:39.692887 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:39.692516    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c158570a-5786998d6b-v874h" event={"ID":"4dcbb91c-dac8-4e6f-8c44-5cc114f83737","Type":"ContainerStarted","Data":"801cd8dceea98680806f13d8b3df5e4526eff8473fc787308513b0780c12ee46"}
Sep 11 19:01:39.692887 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:39.692619    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c158570a-5786998d6b-v874h"
Sep 11 19:01:39.708848 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:39.708799    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c158570a-5786998d6b-v874h" podStartSLOduration=1.7087822959999999 podStartE2EDuration="1.708782296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:01:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:01:39.707125135 +0000 UTC m=+775.784715701" watchObservedRunningTime="2026-09-11 19:01:39.708782296 +0000 UTC m=+775.786372850"
Sep 11 19:01:40.397360 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:40.397324    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1bfd8b7a-2cd2-4121-aa13-110e6710a337" path="/var/lib/kubelet/pods/1bfd8b7a-2cd2-4121-aa13-110e6710a337/volumes"
Sep 11 19:01:40.410441 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:40.410417    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-62499f/simulator-w0-62499f-kserve-6f58d77b77-h7rvf"]
Sep 11 19:01:40.410776 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:40.410761    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bfd8b7a-2cd2-4121-aa13-110e6710a337" containerName="istio-proxy"
Sep 11 19:01:40.410833 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:40.410777    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bfd8b7a-2cd2-4121-aa13-110e6710a337" containerName="istio-proxy"
Sep 11 19:01:40.417879 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:40.417845    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-62499f/simulator-w0-62499f-kserve-6f58d77b77-h7rvf"
Sep 11 19:01:40.420495 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:40.420470    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-62499f/simulator-w0-62499f-kserve-6f58d77b77-h7rvf"]
Sep 11 19:01:40.420637 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:40.420588    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-62499f\"/\"default-dockercfg-mg452\""
Sep 11 19:01:40.420706 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:40.420641    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-62499f\"/\"openshift-service-ca.crt\""
Sep 11 19:01:40.420762 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:40.420729    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-62499f\"/\"kube-root-ca.crt\""
Sep 11 19:01:40.420919 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:40.420901    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-62499f\"/\"simulator-w0-62499f-kserve-self-signed-certs\""
Sep 11 19:01:40.532378 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:40.532352    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsghq\" (UniqueName: \"kubernetes.io/projected/64cba910-3466-40e1-bfc1-f5169c831845-kube-api-access-rsghq\") pod \"simulator-w0-62499f-kserve-6f58d77b77-h7rvf\" (UID: \"64cba910-3466-40e1-bfc1-f5169c831845\") " pod="e2e-models-e2e-worker-w0-62499f/simulator-w0-62499f-kserve-6f58d77b77-h7rvf"
Sep 11 19:01:40.532521 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:40.532384    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64cba910-3466-40e1-bfc1-f5169c831845-home\") pod \"simulator-w0-62499f-kserve-6f58d77b77-h7rvf\" (UID: \"64cba910-3466-40e1-bfc1-f5169c831845\") " pod="e2e-models-e2e-worker-w0-62499f/simulator-w0-62499f-kserve-6f58d77b77-h7rvf"
Sep 11 19:01:40.532521 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:40.532432    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64cba910-3466-40e1-bfc1-f5169c831845-model-cache\") pod \"simulator-w0-62499f-kserve-6f58d77b77-h7rvf\" (UID: \"64cba910-3466-40e1-bfc1-f5169c831845\") " pod="e2e-models-e2e-worker-w0-62499f/simulator-w0-62499f-kserve-6f58d77b77-h7rvf"
Sep 11 19:01:40.532521 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:40.532480    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64cba910-3466-40e1-bfc1-f5169c831845-dshm\") pod \"simulator-w0-62499f-kserve-6f58d77b77-h7rvf\" (UID: \"64cba910-3466-40e1-bfc1-f5169c831845\") " pod="e2e-models-e2e-worker-w0-62499f/simulator-w0-62499f-kserve-6f58d77b77-h7rvf"
Sep 11 19:01:40.532658 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:40.532529    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64cba910-3466-40e1-bfc1-f5169c831845-tmp-dir\") pod \"simulator-w0-62499f-kserve-6f58d77b77-h7rvf\" (UID: \"64cba910-3466-40e1-bfc1-f5169c831845\") " pod="e2e-models-e2e-worker-w0-62499f/simulator-w0-62499f-kserve-6f58d77b77-h7rvf"
Sep 11 19:01:40.532658 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:40.532549    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64cba910-3466-40e1-bfc1-f5169c831845-tls-certs\") pod \"simulator-w0-62499f-kserve-6f58d77b77-h7rvf\" (UID: \"64cba910-3466-40e1-bfc1-f5169c831845\") " pod="e2e-models-e2e-worker-w0-62499f/simulator-w0-62499f-kserve-6f58d77b77-h7rvf"
Sep 11 19:01:40.633747 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:40.633712    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64cba910-3466-40e1-bfc1-f5169c831845-home\") pod \"simulator-w0-62499f-kserve-6f58d77b77-h7rvf\" (UID: \"64cba910-3466-40e1-bfc1-f5169c831845\") " pod="e2e-models-e2e-worker-w0-62499f/simulator-w0-62499f-kserve-6f58d77b77-h7rvf"
Sep 11 19:01:40.633904 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:40.633782    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64cba910-3466-40e1-bfc1-f5169c831845-model-cache\") pod \"simulator-w0-62499f-kserve-6f58d77b77-h7rvf\" (UID: \"64cba910-3466-40e1-bfc1-f5169c831845\") " pod="e2e-models-e2e-worker-w0-62499f/simulator-w0-62499f-kserve-6f58d77b77-h7rvf"
Sep 11 19:01:40.633904 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:40.633818    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64cba910-3466-40e1-bfc1-f5169c831845-dshm\") pod \"simulator-w0-62499f-kserve-6f58d77b77-h7rvf\" (UID: \"64cba910-3466-40e1-bfc1-f5169c831845\") " pod="e2e-models-e2e-worker-w0-62499f/simulator-w0-62499f-kserve-6f58d77b77-h7rvf"
Sep 11 19:01:40.634016 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:40.633958    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64cba910-3466-40e1-bfc1-f5169c831845-tmp-dir\") pod \"simulator-w0-62499f-kserve-6f58d77b77-h7rvf\" (UID: \"64cba910-3466-40e1-bfc1-f5169c831845\") " pod="e2e-models-e2e-worker-w0-62499f/simulator-w0-62499f-kserve-6f58d77b77-h7rvf"
Sep 11 19:01:40.634016 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:40.633983    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64cba910-3466-40e1-bfc1-f5169c831845-tls-certs\") pod \"simulator-w0-62499f-kserve-6f58d77b77-h7rvf\" (UID: \"64cba910-3466-40e1-bfc1-f5169c831845\") " pod="e2e-models-e2e-worker-w0-62499f/simulator-w0-62499f-kserve-6f58d77b77-h7rvf"
Sep 11 19:01:40.634120 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:40.634025    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsghq\" (UniqueName: \"kubernetes.io/projected/64cba910-3466-40e1-bfc1-f5169c831845-kube-api-access-rsghq\") pod \"simulator-w0-62499f-kserve-6f58d77b77-h7rvf\" (UID: \"64cba910-3466-40e1-bfc1-f5169c831845\") " pod="e2e-models-e2e-worker-w0-62499f/simulator-w0-62499f-kserve-6f58d77b77-h7rvf"
Sep 11 19:01:40.634120 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:40.634093    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64cba910-3466-40e1-bfc1-f5169c831845-home\") pod \"simulator-w0-62499f-kserve-6f58d77b77-h7rvf\" (UID: \"64cba910-3466-40e1-bfc1-f5169c831845\") " pod="e2e-models-e2e-worker-w0-62499f/simulator-w0-62499f-kserve-6f58d77b77-h7rvf"
Sep 11 19:01:40.634245 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:40.634226    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64cba910-3466-40e1-bfc1-f5169c831845-model-cache\") pod \"simulator-w0-62499f-kserve-6f58d77b77-h7rvf\" (UID: \"64cba910-3466-40e1-bfc1-f5169c831845\") " pod="e2e-models-e2e-worker-w0-62499f/simulator-w0-62499f-kserve-6f58d77b77-h7rvf"
Sep 11 19:01:40.634284 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:40.634267    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64cba910-3466-40e1-bfc1-f5169c831845-tmp-dir\") pod \"simulator-w0-62499f-kserve-6f58d77b77-h7rvf\" (UID: \"64cba910-3466-40e1-bfc1-f5169c831845\") " pod="e2e-models-e2e-worker-w0-62499f/simulator-w0-62499f-kserve-6f58d77b77-h7rvf"
Sep 11 19:01:40.636177 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:40.636156    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64cba910-3466-40e1-bfc1-f5169c831845-dshm\") pod \"simulator-w0-62499f-kserve-6f58d77b77-h7rvf\" (UID: \"64cba910-3466-40e1-bfc1-f5169c831845\") " pod="e2e-models-e2e-worker-w0-62499f/simulator-w0-62499f-kserve-6f58d77b77-h7rvf"
Sep 11 19:01:40.636423 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:40.636407    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64cba910-3466-40e1-bfc1-f5169c831845-tls-certs\") pod \"simulator-w0-62499f-kserve-6f58d77b77-h7rvf\" (UID: \"64cba910-3466-40e1-bfc1-f5169c831845\") " pod="e2e-models-e2e-worker-w0-62499f/simulator-w0-62499f-kserve-6f58d77b77-h7rvf"
Sep 11 19:01:40.641360 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:40.641336    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsghq\" (UniqueName: \"kubernetes.io/projected/64cba910-3466-40e1-bfc1-f5169c831845-kube-api-access-rsghq\") pod \"simulator-w0-62499f-kserve-6f58d77b77-h7rvf\" (UID: \"64cba910-3466-40e1-bfc1-f5169c831845\") " pod="e2e-models-e2e-worker-w0-62499f/simulator-w0-62499f-kserve-6f58d77b77-h7rvf"
Sep 11 19:01:40.728290 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:40.728270    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-62499f/simulator-w0-62499f-kserve-6f58d77b77-h7rvf"
Sep 11 19:01:41.018872 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:41.018846    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-62499f/simulator-w0-62499f-kserve-6f58d77b77-h7rvf"]
Sep 11 19:01:41.021303 ip-10-0-134-189 kubenswrapper[2578]: W0911 19:01:41.021277    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod64cba910_3466_40e1_bfc1_f5169c831845.slice/crio-fe3c0dea8c84ec21f33493f7e0086dd53adbf6b591fb13bb6093b63901b2514e WatchSource:0}: Error finding container fe3c0dea8c84ec21f33493f7e0086dd53adbf6b591fb13bb6093b63901b2514e: Status 404 returned error can't find the container with id fe3c0dea8c84ec21f33493f7e0086dd53adbf6b591fb13bb6093b63901b2514e
Sep 11 19:01:41.705334 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:41.705295    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-62499f/simulator-w0-62499f-kserve-6f58d77b77-h7rvf" event={"ID":"64cba910-3466-40e1-bfc1-f5169c831845","Type":"ContainerStarted","Data":"4dd2ac6e2972631acd0e508a7360751268470bf1c808b1ec06018a5bc3a84ced"}
Sep 11 19:01:41.705516 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:41.705344    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-62499f/simulator-w0-62499f-kserve-6f58d77b77-h7rvf" event={"ID":"64cba910-3466-40e1-bfc1-f5169c831845","Type":"ContainerStarted","Data":"fe3c0dea8c84ec21f33493f7e0086dd53adbf6b591fb13bb6093b63901b2514e"}
Sep 11 19:01:41.722097 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:41.722054    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-62499f/simulator-w0-62499f-kserve-6f58d77b77-h7rvf" podStartSLOduration=1.7220383689999998 podStartE2EDuration="1.722038369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:01:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:01:41.719854968 +0000 UTC m=+777.797445523" watchObservedRunningTime="2026-09-11 19:01:41.722038369 +0000 UTC m=+777.799628920"
Sep 11 19:01:50.700408 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:50.700375    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c158570a-5786998d6b-v874h"
Sep 11 19:01:50.728441 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:50.728413    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-62499f/simulator-w0-62499f-kserve-6f58d77b77-h7rvf"
Sep 11 19:01:50.728617 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:50.728450    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-62499f/simulator-w0-62499f-kserve-6f58d77b77-h7rvf"
Sep 11 19:01:50.741772 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:50.741740    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-62499f/simulator-w0-62499f-kserve-6f58d77b77-h7rvf"
Sep 11 19:01:50.759043 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:01:50.759015    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-62499f/simulator-w0-62499f-kserve-6f58d77b77-h7rvf"
Sep 11 19:02:00.922546 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:00.922517    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c158570a-d8gf4"]
Sep 11 19:02:00.926131 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:00.926110    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c158570a-d8gf4"
Sep 11 19:02:00.928417 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:00.928395    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 19:02:00.928516 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:00.928426    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ln295\""
Sep 11 19:02:00.933897 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:00.933876    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c158570a-d8gf4"]
Sep 11 19:02:01.020398 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:01.020370    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5daa25eb-358a-4f3f-96a0-88532220a9f8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c158570a-d8gf4\" (UID: \"5daa25eb-358a-4f3f-96a0-88532220a9f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c158570a-d8gf4"
Sep 11 19:02:01.121421 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:01.121386    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5daa25eb-358a-4f3f-96a0-88532220a9f8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c158570a-d8gf4\" (UID: \"5daa25eb-358a-4f3f-96a0-88532220a9f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c158570a-d8gf4"
Sep 11 19:02:01.122039 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:01.122020    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5daa25eb-358a-4f3f-96a0-88532220a9f8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c158570a-d8gf4\" (UID: \"5daa25eb-358a-4f3f-96a0-88532220a9f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c158570a-d8gf4"
Sep 11 19:02:01.238452 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:01.238424    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c158570a-d8gf4"
Sep 11 19:02:01.505549 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:01.505528    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c158570a-d8gf4"]
Sep 11 19:02:01.507530 ip-10-0-134-189 kubenswrapper[2578]: W0911 19:02:01.507492    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5daa25eb_358a_4f3f_96a0_88532220a9f8.slice/crio-f1d877ea8365c95b51c46c0828e849986d0ba3811fc1ababe7373f188c855628 WatchSource:0}: Error finding container f1d877ea8365c95b51c46c0828e849986d0ba3811fc1ababe7373f188c855628: Status 404 returned error can't find the container with id f1d877ea8365c95b51c46c0828e849986d0ba3811fc1ababe7373f188c855628
Sep 11 19:02:01.799106 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:01.799022    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c158570a-d8gf4" event={"ID":"5daa25eb-358a-4f3f-96a0-88532220a9f8","Type":"ContainerStarted","Data":"f1d877ea8365c95b51c46c0828e849986d0ba3811fc1ababe7373f188c855628"}
Sep 11 19:02:05.819715 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:05.819630    2578 generic.go:356] "Generic (PLEG): container finished" podID="5daa25eb-358a-4f3f-96a0-88532220a9f8" containerID="ab95400cf93d4f634d06a12cb2bb9bbfce2fc9152df07e45c2e08411a93c54fc" exitCode=0
Sep 11 19:02:05.820063 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:05.819711    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c158570a-d8gf4" event={"ID":"5daa25eb-358a-4f3f-96a0-88532220a9f8","Type":"ContainerDied","Data":"ab95400cf93d4f634d06a12cb2bb9bbfce2fc9152df07e45c2e08411a93c54fc"}
Sep 11 19:02:06.951272 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:06.951250    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c158570a-d8gf4"
Sep 11 19:02:06.976138 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:06.976113    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5daa25eb-358a-4f3f-96a0-88532220a9f8-maas-api-service-ca\") pod \"5daa25eb-358a-4f3f-96a0-88532220a9f8\" (UID: \"5daa25eb-358a-4f3f-96a0-88532220a9f8\") "
Sep 11 19:02:06.976433 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:06.976411    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5daa25eb-358a-4f3f-96a0-88532220a9f8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5daa25eb-358a-4f3f-96a0-88532220a9f8" (UID: "5daa25eb-358a-4f3f-96a0-88532220a9f8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 19:02:07.077342 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:07.077278    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5daa25eb-358a-4f3f-96a0-88532220a9f8-maas-api-service-ca\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:02:07.830547 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:07.830518    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c158570a-d8gf4"
Sep 11 19:02:07.830735 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:07.830558    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c158570a-d8gf4" event={"ID":"5daa25eb-358a-4f3f-96a0-88532220a9f8","Type":"ContainerDied","Data":"f1d877ea8365c95b51c46c0828e849986d0ba3811fc1ababe7373f188c855628"}
Sep 11 19:02:07.830735 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:07.830639    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f1d877ea8365c95b51c46c0828e849986d0ba3811fc1ababe7373f188c855628"
Sep 11 19:02:17.812636 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:17.812577    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c158570a-5786998d6b-v874h"]
Sep 11 19:02:17.813188 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:17.812851    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c158570a-5786998d6b-v874h" podUID="4dcbb91c-dac8-4e6f-8c44-5cc114f83737" containerName="payload-pre-processing" containerID="cri-o://79d7e90f107c7d745179558308d6e76b7c95a483c24776099bd82a9d2bfcb892" gracePeriod=30
Sep 11 19:02:17.983122 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:17.983081    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c158570a-d8gf4"]
Sep 11 19:02:17.986691 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:17.986668    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c158570a-d8gf4"]
Sep 11 19:02:18.057496 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:18.057457    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c158570a-5786998d6b-v874h"
Sep 11 19:02:18.176282 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:18.176200    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kvrxs\" (UniqueName: \"kubernetes.io/projected/4dcbb91c-dac8-4e6f-8c44-5cc114f83737-kube-api-access-kvrxs\") pod \"4dcbb91c-dac8-4e6f-8c44-5cc114f83737\" (UID: \"4dcbb91c-dac8-4e6f-8c44-5cc114f83737\") "
Sep 11 19:02:18.176427 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:18.176282    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dcbb91c-dac8-4e6f-8c44-5cc114f83737-plugins-config-volume\") pod \"4dcbb91c-dac8-4e6f-8c44-5cc114f83737\" (UID: \"4dcbb91c-dac8-4e6f-8c44-5cc114f83737\") "
Sep 11 19:02:18.176635 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:18.176587    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dcbb91c-dac8-4e6f-8c44-5cc114f83737-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4dcbb91c-dac8-4e6f-8c44-5cc114f83737" (UID: "4dcbb91c-dac8-4e6f-8c44-5cc114f83737"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 19:02:18.178496 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:18.178478    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dcbb91c-dac8-4e6f-8c44-5cc114f83737-kube-api-access-kvrxs" (OuterVolumeSpecName: "kube-api-access-kvrxs") pod "4dcbb91c-dac8-4e6f-8c44-5cc114f83737" (UID: "4dcbb91c-dac8-4e6f-8c44-5cc114f83737"). InnerVolumeSpecName "kube-api-access-kvrxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:02:18.277360 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:18.277327    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dcbb91c-dac8-4e6f-8c44-5cc114f83737-plugins-config-volume\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:02:18.277360 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:18.277354    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvrxs\" (UniqueName: \"kubernetes.io/projected/4dcbb91c-dac8-4e6f-8c44-5cc114f83737-kube-api-access-kvrxs\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:02:18.396720 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:18.396690    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5daa25eb-358a-4f3f-96a0-88532220a9f8" path="/var/lib/kubelet/pods/5daa25eb-358a-4f3f-96a0-88532220a9f8/volumes"
Sep 11 19:02:18.846411 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:18.846379    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-f0250e49-openshift-default-664bb69d6-p626z"]
Sep 11 19:02:18.846837 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:18.846753    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5daa25eb-358a-4f3f-96a0-88532220a9f8" containerName="revoke-keys"
Sep 11 19:02:18.846837 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:18.846766    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5daa25eb-358a-4f3f-96a0-88532220a9f8" containerName="revoke-keys"
Sep 11 19:02:18.846837 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:18.846800    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4dcbb91c-dac8-4e6f-8c44-5cc114f83737" containerName="payload-pre-processing"
Sep 11 19:02:18.846837 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:18.846807    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4dcbb91c-dac8-4e6f-8c44-5cc114f83737" containerName="payload-pre-processing"
Sep 11 19:02:18.846998 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:18.846863    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4dcbb91c-dac8-4e6f-8c44-5cc114f83737" containerName="payload-pre-processing"
Sep 11 19:02:18.846998 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:18.846874    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5daa25eb-358a-4f3f-96a0-88532220a9f8" containerName="revoke-keys"
Sep 11 19:02:18.848815 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:18.848797    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f0250e49-openshift-default-664bb69d6-p626z"
Sep 11 19:02:18.850987 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:18.850968    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-f0250e49-openshift-default-dockercfg-kskjt\""
Sep 11 19:02:18.865804 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:18.865782    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f0250e49-openshift-default-664bb69d6-p626z"]
Sep 11 19:02:18.886269 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:18.886238    2578 generic.go:356] "Generic (PLEG): container finished" podID="4dcbb91c-dac8-4e6f-8c44-5cc114f83737" containerID="79d7e90f107c7d745179558308d6e76b7c95a483c24776099bd82a9d2bfcb892" exitCode=0
Sep 11 19:02:18.886456 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:18.886332    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c158570a-5786998d6b-v874h"
Sep 11 19:02:18.886713 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:18.886350    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c158570a-5786998d6b-v874h" event={"ID":"4dcbb91c-dac8-4e6f-8c44-5cc114f83737","Type":"ContainerDied","Data":"79d7e90f107c7d745179558308d6e76b7c95a483c24776099bd82a9d2bfcb892"}
Sep 11 19:02:18.886848 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:18.886755    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c158570a-5786998d6b-v874h" event={"ID":"4dcbb91c-dac8-4e6f-8c44-5cc114f83737","Type":"ContainerDied","Data":"801cd8dceea98680806f13d8b3df5e4526eff8473fc787308513b0780c12ee46"}
Sep 11 19:02:18.886848 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:18.886824    2578 scope.go:117] "RemoveContainer" containerID="79d7e90f107c7d745179558308d6e76b7c95a483c24776099bd82a9d2bfcb892"
Sep 11 19:02:18.895634 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:18.895613    2578 scope.go:117] "RemoveContainer" containerID="79d7e90f107c7d745179558308d6e76b7c95a483c24776099bd82a9d2bfcb892"
Sep 11 19:02:18.895934 ip-10-0-134-189 kubenswrapper[2578]: E0911 19:02:18.895911    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79d7e90f107c7d745179558308d6e76b7c95a483c24776099bd82a9d2bfcb892\": container with ID starting with 79d7e90f107c7d745179558308d6e76b7c95a483c24776099bd82a9d2bfcb892 not found: ID does not exist" containerID="79d7e90f107c7d745179558308d6e76b7c95a483c24776099bd82a9d2bfcb892"
Sep 11 19:02:18.895986 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:18.895943    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79d7e90f107c7d745179558308d6e76b7c95a483c24776099bd82a9d2bfcb892"} err="failed to get container status \"79d7e90f107c7d745179558308d6e76b7c95a483c24776099bd82a9d2bfcb892\": rpc error: code = NotFound desc = could not find container \"79d7e90f107c7d745179558308d6e76b7c95a483c24776099bd82a9d2bfcb892\": container with ID starting with 79d7e90f107c7d745179558308d6e76b7c95a483c24776099bd82a9d2bfcb892 not found: ID does not exist"
Sep 11 19:02:18.902095 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:18.902073    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c158570a-5786998d6b-v874h"]
Sep 11 19:02:18.908138 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:18.908119    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c158570a-5786998d6b-v874h"]
Sep 11 19:02:18.983058 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:18.983024    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fd0dce6-a2cb-4c83-bb92-35309ea2b40e-istio-podinfo\") pod \"e2e-ait-f0250e49-openshift-default-664bb69d6-p626z\" (UID: \"7fd0dce6-a2cb-4c83-bb92-35309ea2b40e\") " pod="openshift-ingress/e2e-ait-f0250e49-openshift-default-664bb69d6-p626z"
Sep 11 19:02:18.983547 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:18.983069    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7wxr\" (UniqueName: \"kubernetes.io/projected/7fd0dce6-a2cb-4c83-bb92-35309ea2b40e-kube-api-access-d7wxr\") pod \"e2e-ait-f0250e49-openshift-default-664bb69d6-p626z\" (UID: \"7fd0dce6-a2cb-4c83-bb92-35309ea2b40e\") " pod="openshift-ingress/e2e-ait-f0250e49-openshift-default-664bb69d6-p626z"
Sep 11 19:02:18.983734 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:18.983586    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fd0dce6-a2cb-4c83-bb92-35309ea2b40e-istiod-ca-cert\") pod \"e2e-ait-f0250e49-openshift-default-664bb69d6-p626z\" (UID: \"7fd0dce6-a2cb-4c83-bb92-35309ea2b40e\") " pod="openshift-ingress/e2e-ait-f0250e49-openshift-default-664bb69d6-p626z"
Sep 11 19:02:18.983734 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:18.983685    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fd0dce6-a2cb-4c83-bb92-35309ea2b40e-workload-socket\") pod \"e2e-ait-f0250e49-openshift-default-664bb69d6-p626z\" (UID: \"7fd0dce6-a2cb-4c83-bb92-35309ea2b40e\") " pod="openshift-ingress/e2e-ait-f0250e49-openshift-default-664bb69d6-p626z"
Sep 11 19:02:18.983850 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:18.983789    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fd0dce6-a2cb-4c83-bb92-35309ea2b40e-istio-envoy\") pod \"e2e-ait-f0250e49-openshift-default-664bb69d6-p626z\" (UID: \"7fd0dce6-a2cb-4c83-bb92-35309ea2b40e\") " pod="openshift-ingress/e2e-ait-f0250e49-openshift-default-664bb69d6-p626z"
Sep 11 19:02:18.983951 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:18.983925    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fd0dce6-a2cb-4c83-bb92-35309ea2b40e-istio-data\") pod \"e2e-ait-f0250e49-openshift-default-664bb69d6-p626z\" (UID: \"7fd0dce6-a2cb-4c83-bb92-35309ea2b40e\") " pod="openshift-ingress/e2e-ait-f0250e49-openshift-default-664bb69d6-p626z"
Sep 11 19:02:18.984026 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:18.983969    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fd0dce6-a2cb-4c83-bb92-35309ea2b40e-workload-certs\") pod \"e2e-ait-f0250e49-openshift-default-664bb69d6-p626z\" (UID: \"7fd0dce6-a2cb-4c83-bb92-35309ea2b40e\") " pod="openshift-ingress/e2e-ait-f0250e49-openshift-default-664bb69d6-p626z"
Sep 11 19:02:18.984026 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:18.984012    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fd0dce6-a2cb-4c83-bb92-35309ea2b40e-credential-socket\") pod \"e2e-ait-f0250e49-openshift-default-664bb69d6-p626z\" (UID: \"7fd0dce6-a2cb-4c83-bb92-35309ea2b40e\") " pod="openshift-ingress/e2e-ait-f0250e49-openshift-default-664bb69d6-p626z"
Sep 11 19:02:18.984116 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:18.984052    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fd0dce6-a2cb-4c83-bb92-35309ea2b40e-istio-token\") pod \"e2e-ait-f0250e49-openshift-default-664bb69d6-p626z\" (UID: \"7fd0dce6-a2cb-4c83-bb92-35309ea2b40e\") " pod="openshift-ingress/e2e-ait-f0250e49-openshift-default-664bb69d6-p626z"
Sep 11 19:02:19.084748 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:19.084714    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fd0dce6-a2cb-4c83-bb92-35309ea2b40e-istio-data\") pod \"e2e-ait-f0250e49-openshift-default-664bb69d6-p626z\" (UID: \"7fd0dce6-a2cb-4c83-bb92-35309ea2b40e\") " pod="openshift-ingress/e2e-ait-f0250e49-openshift-default-664bb69d6-p626z"
Sep 11 19:02:19.084908 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:19.084754    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fd0dce6-a2cb-4c83-bb92-35309ea2b40e-workload-certs\") pod \"e2e-ait-f0250e49-openshift-default-664bb69d6-p626z\" (UID: \"7fd0dce6-a2cb-4c83-bb92-35309ea2b40e\") " pod="openshift-ingress/e2e-ait-f0250e49-openshift-default-664bb69d6-p626z"
Sep 11 19:02:19.084908 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:19.084789    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fd0dce6-a2cb-4c83-bb92-35309ea2b40e-credential-socket\") pod \"e2e-ait-f0250e49-openshift-default-664bb69d6-p626z\" (UID: \"7fd0dce6-a2cb-4c83-bb92-35309ea2b40e\") " pod="openshift-ingress/e2e-ait-f0250e49-openshift-default-664bb69d6-p626z"
Sep 11 19:02:19.084908 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:19.084820    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fd0dce6-a2cb-4c83-bb92-35309ea2b40e-istio-token\") pod \"e2e-ait-f0250e49-openshift-default-664bb69d6-p626z\" (UID: \"7fd0dce6-a2cb-4c83-bb92-35309ea2b40e\") " pod="openshift-ingress/e2e-ait-f0250e49-openshift-default-664bb69d6-p626z"
Sep 11 19:02:19.085026 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:19.084970    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fd0dce6-a2cb-4c83-bb92-35309ea2b40e-istio-podinfo\") pod \"e2e-ait-f0250e49-openshift-default-664bb69d6-p626z\" (UID: \"7fd0dce6-a2cb-4c83-bb92-35309ea2b40e\") " pod="openshift-ingress/e2e-ait-f0250e49-openshift-default-664bb69d6-p626z"
Sep 11 19:02:19.085026 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:19.085020    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7wxr\" (UniqueName: \"kubernetes.io/projected/7fd0dce6-a2cb-4c83-bb92-35309ea2b40e-kube-api-access-d7wxr\") pod \"e2e-ait-f0250e49-openshift-default-664bb69d6-p626z\" (UID: \"7fd0dce6-a2cb-4c83-bb92-35309ea2b40e\") " pod="openshift-ingress/e2e-ait-f0250e49-openshift-default-664bb69d6-p626z"
Sep 11 19:02:19.085124 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:19.085046    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fd0dce6-a2cb-4c83-bb92-35309ea2b40e-istiod-ca-cert\") pod \"e2e-ait-f0250e49-openshift-default-664bb69d6-p626z\" (UID: \"7fd0dce6-a2cb-4c83-bb92-35309ea2b40e\") " pod="openshift-ingress/e2e-ait-f0250e49-openshift-default-664bb69d6-p626z"
Sep 11 19:02:19.085124 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:19.085097    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fd0dce6-a2cb-4c83-bb92-35309ea2b40e-workload-socket\") pod \"e2e-ait-f0250e49-openshift-default-664bb69d6-p626z\" (UID: \"7fd0dce6-a2cb-4c83-bb92-35309ea2b40e\") " pod="openshift-ingress/e2e-ait-f0250e49-openshift-default-664bb69d6-p626z"
Sep 11 19:02:19.085218 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:19.085125    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fd0dce6-a2cb-4c83-bb92-35309ea2b40e-istio-envoy\") pod \"e2e-ait-f0250e49-openshift-default-664bb69d6-p626z\" (UID: \"7fd0dce6-a2cb-4c83-bb92-35309ea2b40e\") " pod="openshift-ingress/e2e-ait-f0250e49-openshift-default-664bb69d6-p626z"
Sep 11 19:02:19.085218 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:19.085165    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fd0dce6-a2cb-4c83-bb92-35309ea2b40e-istio-data\") pod \"e2e-ait-f0250e49-openshift-default-664bb69d6-p626z\" (UID: \"7fd0dce6-a2cb-4c83-bb92-35309ea2b40e\") " pod="openshift-ingress/e2e-ait-f0250e49-openshift-default-664bb69d6-p626z"
Sep 11 19:02:19.085318 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:19.085215    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fd0dce6-a2cb-4c83-bb92-35309ea2b40e-workload-certs\") pod \"e2e-ait-f0250e49-openshift-default-664bb69d6-p626z\" (UID: \"7fd0dce6-a2cb-4c83-bb92-35309ea2b40e\") " pod="openshift-ingress/e2e-ait-f0250e49-openshift-default-664bb69d6-p626z"
Sep 11 19:02:19.085318 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:19.085272    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fd0dce6-a2cb-4c83-bb92-35309ea2b40e-credential-socket\") pod \"e2e-ait-f0250e49-openshift-default-664bb69d6-p626z\" (UID: \"7fd0dce6-a2cb-4c83-bb92-35309ea2b40e\") " pod="openshift-ingress/e2e-ait-f0250e49-openshift-default-664bb69d6-p626z"
Sep 11 19:02:19.085467 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:19.085444    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fd0dce6-a2cb-4c83-bb92-35309ea2b40e-workload-socket\") pod \"e2e-ait-f0250e49-openshift-default-664bb69d6-p626z\" (UID: \"7fd0dce6-a2cb-4c83-bb92-35309ea2b40e\") " pod="openshift-ingress/e2e-ait-f0250e49-openshift-default-664bb69d6-p626z"
Sep 11 19:02:19.085730 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:19.085711    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fd0dce6-a2cb-4c83-bb92-35309ea2b40e-istiod-ca-cert\") pod \"e2e-ait-f0250e49-openshift-default-664bb69d6-p626z\" (UID: \"7fd0dce6-a2cb-4c83-bb92-35309ea2b40e\") " pod="openshift-ingress/e2e-ait-f0250e49-openshift-default-664bb69d6-p626z"
Sep 11 19:02:19.087358 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:19.087318    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fd0dce6-a2cb-4c83-bb92-35309ea2b40e-istio-envoy\") pod \"e2e-ait-f0250e49-openshift-default-664bb69d6-p626z\" (UID: \"7fd0dce6-a2cb-4c83-bb92-35309ea2b40e\") " pod="openshift-ingress/e2e-ait-f0250e49-openshift-default-664bb69d6-p626z"
Sep 11 19:02:19.087632 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:19.087613    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fd0dce6-a2cb-4c83-bb92-35309ea2b40e-istio-podinfo\") pod \"e2e-ait-f0250e49-openshift-default-664bb69d6-p626z\" (UID: \"7fd0dce6-a2cb-4c83-bb92-35309ea2b40e\") " pod="openshift-ingress/e2e-ait-f0250e49-openshift-default-664bb69d6-p626z"
Sep 11 19:02:19.094794 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:19.094758    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7wxr\" (UniqueName: \"kubernetes.io/projected/7fd0dce6-a2cb-4c83-bb92-35309ea2b40e-kube-api-access-d7wxr\") pod \"e2e-ait-f0250e49-openshift-default-664bb69d6-p626z\" (UID: \"7fd0dce6-a2cb-4c83-bb92-35309ea2b40e\") " pod="openshift-ingress/e2e-ait-f0250e49-openshift-default-664bb69d6-p626z"
Sep 11 19:02:19.095228 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:19.095209    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fd0dce6-a2cb-4c83-bb92-35309ea2b40e-istio-token\") pod \"e2e-ait-f0250e49-openshift-default-664bb69d6-p626z\" (UID: \"7fd0dce6-a2cb-4c83-bb92-35309ea2b40e\") " pod="openshift-ingress/e2e-ait-f0250e49-openshift-default-664bb69d6-p626z"
Sep 11 19:02:19.158474 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:19.158385    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f0250e49-openshift-default-664bb69d6-p626z"
Sep 11 19:02:19.433152 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:19.433129    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f0250e49-openshift-default-664bb69d6-p626z"]
Sep 11 19:02:19.434956 ip-10-0-134-189 kubenswrapper[2578]: W0911 19:02:19.434928    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fd0dce6_a2cb_4c83_bb92_35309ea2b40e.slice/crio-427b4c2e429d142ebb4316c6aadc604c441a999e0e86986c11d647c5cd563d38 WatchSource:0}: Error finding container 427b4c2e429d142ebb4316c6aadc604c441a999e0e86986c11d647c5cd563d38: Status 404 returned error can't find the container with id 427b4c2e429d142ebb4316c6aadc604c441a999e0e86986c11d647c5cd563d38
Sep 11 19:02:19.437468 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:19.437436    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 19:02:19.437543 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:19.437501    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 19:02:19.437543 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:19.437527    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 19:02:19.894990 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:19.894949    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f0250e49-openshift-default-664bb69d6-p626z" event={"ID":"7fd0dce6-a2cb-4c83-bb92-35309ea2b40e","Type":"ContainerStarted","Data":"96e95add969e6fd890ae7a0cffc2815daab26275ea299a0372d39e8eb3cf5dfa"}
Sep 11 19:02:19.895370 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:19.894999    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f0250e49-openshift-default-664bb69d6-p626z" event={"ID":"7fd0dce6-a2cb-4c83-bb92-35309ea2b40e","Type":"ContainerStarted","Data":"427b4c2e429d142ebb4316c6aadc604c441a999e0e86986c11d647c5cd563d38"}
Sep 11 19:02:19.914721 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:19.914461    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-f0250e49-openshift-default-664bb69d6-p626z" podStartSLOduration=1.914441762 podStartE2EDuration="1.914441762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:02:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:02:19.914073984 +0000 UTC m=+815.991664553" watchObservedRunningTime="2026-09-11 19:02:19.914441762 +0000 UTC m=+815.992032317"
Sep 11 19:02:20.159420 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:20.159353    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-f0250e49-openshift-default-664bb69d6-p626z"
Sep 11 19:02:20.165629 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:20.165584    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-f0250e49-openshift-default-664bb69d6-p626z"
Sep 11 19:02:20.397409 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:20.397374    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4dcbb91c-dac8-4e6f-8c44-5cc114f83737" path="/var/lib/kubelet/pods/4dcbb91c-dac8-4e6f-8c44-5cc114f83737/volumes"
Sep 11 19:02:20.899690 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:20.899664    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f0250e49-openshift-default-664bb69d6-p626z"
Sep 11 19:02:20.900590 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:20.900566    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-f0250e49-openshift-default-664bb69d6-p626z"
Sep 11 19:02:21.627203 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:21.627170    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-89e163/autores-model-89e163-kserve-696c5f94f9-m8lk7"]
Sep 11 19:02:21.629679 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:21.629664    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-89e163/autores-model-89e163-kserve-696c5f94f9-m8lk7"
Sep 11 19:02:21.632876 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:21.632857    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-89e163\"/\"autores-model-89e163-kserve-self-signed-certs\""
Sep 11 19:02:21.632876 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:21.632864    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-89e163\"/\"kube-root-ca.crt\""
Sep 11 19:02:21.633085 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:21.632863    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-89e163\"/\"default-dockercfg-f7nh5\""
Sep 11 19:02:21.633085 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:21.632931    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-89e163\"/\"openshift-service-ca.crt\""
Sep 11 19:02:21.640327 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:21.640308    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-89e163/autores-model-89e163-kserve-696c5f94f9-m8lk7"]
Sep 11 19:02:21.711572 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:21.711547    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa7ca741-597a-4af5-8665-ad042b545512-dshm\") pod \"autores-model-89e163-kserve-696c5f94f9-m8lk7\" (UID: \"aa7ca741-597a-4af5-8665-ad042b545512\") " pod="ai-tenant-e2e-autores-89e163/autores-model-89e163-kserve-696c5f94f9-m8lk7"
Sep 11 19:02:21.711718 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:21.711624    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k56tj\" (UniqueName: \"kubernetes.io/projected/aa7ca741-597a-4af5-8665-ad042b545512-kube-api-access-k56tj\") pod \"autores-model-89e163-kserve-696c5f94f9-m8lk7\" (UID: \"aa7ca741-597a-4af5-8665-ad042b545512\") " pod="ai-tenant-e2e-autores-89e163/autores-model-89e163-kserve-696c5f94f9-m8lk7"
Sep 11 19:02:21.711762 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:21.711711    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa7ca741-597a-4af5-8665-ad042b545512-tmp-dir\") pod \"autores-model-89e163-kserve-696c5f94f9-m8lk7\" (UID: \"aa7ca741-597a-4af5-8665-ad042b545512\") " pod="ai-tenant-e2e-autores-89e163/autores-model-89e163-kserve-696c5f94f9-m8lk7"
Sep 11 19:02:21.711762 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:21.711751    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa7ca741-597a-4af5-8665-ad042b545512-home\") pod \"autores-model-89e163-kserve-696c5f94f9-m8lk7\" (UID: \"aa7ca741-597a-4af5-8665-ad042b545512\") " pod="ai-tenant-e2e-autores-89e163/autores-model-89e163-kserve-696c5f94f9-m8lk7"
Sep 11 19:02:21.711840 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:21.711781    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa7ca741-597a-4af5-8665-ad042b545512-model-cache\") pod \"autores-model-89e163-kserve-696c5f94f9-m8lk7\" (UID: \"aa7ca741-597a-4af5-8665-ad042b545512\") " pod="ai-tenant-e2e-autores-89e163/autores-model-89e163-kserve-696c5f94f9-m8lk7"
Sep 11 19:02:21.711840 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:21.711808    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa7ca741-597a-4af5-8665-ad042b545512-tls-certs\") pod \"autores-model-89e163-kserve-696c5f94f9-m8lk7\" (UID: \"aa7ca741-597a-4af5-8665-ad042b545512\") " pod="ai-tenant-e2e-autores-89e163/autores-model-89e163-kserve-696c5f94f9-m8lk7"
Sep 11 19:02:21.812901 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:21.812874    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa7ca741-597a-4af5-8665-ad042b545512-tmp-dir\") pod \"autores-model-89e163-kserve-696c5f94f9-m8lk7\" (UID: \"aa7ca741-597a-4af5-8665-ad042b545512\") " pod="ai-tenant-e2e-autores-89e163/autores-model-89e163-kserve-696c5f94f9-m8lk7"
Sep 11 19:02:21.813044 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:21.812919    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa7ca741-597a-4af5-8665-ad042b545512-home\") pod \"autores-model-89e163-kserve-696c5f94f9-m8lk7\" (UID: \"aa7ca741-597a-4af5-8665-ad042b545512\") " pod="ai-tenant-e2e-autores-89e163/autores-model-89e163-kserve-696c5f94f9-m8lk7"
Sep 11 19:02:21.813044 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:21.812945    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa7ca741-597a-4af5-8665-ad042b545512-model-cache\") pod \"autores-model-89e163-kserve-696c5f94f9-m8lk7\" (UID: \"aa7ca741-597a-4af5-8665-ad042b545512\") " pod="ai-tenant-e2e-autores-89e163/autores-model-89e163-kserve-696c5f94f9-m8lk7"
Sep 11 19:02:21.813044 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:21.812979    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa7ca741-597a-4af5-8665-ad042b545512-tls-certs\") pod \"autores-model-89e163-kserve-696c5f94f9-m8lk7\" (UID: \"aa7ca741-597a-4af5-8665-ad042b545512\") " pod="ai-tenant-e2e-autores-89e163/autores-model-89e163-kserve-696c5f94f9-m8lk7"
Sep 11 19:02:21.813044 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:21.813039    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa7ca741-597a-4af5-8665-ad042b545512-dshm\") pod \"autores-model-89e163-kserve-696c5f94f9-m8lk7\" (UID: \"aa7ca741-597a-4af5-8665-ad042b545512\") " pod="ai-tenant-e2e-autores-89e163/autores-model-89e163-kserve-696c5f94f9-m8lk7"
Sep 11 19:02:21.813227 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:21.813093    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k56tj\" (UniqueName: \"kubernetes.io/projected/aa7ca741-597a-4af5-8665-ad042b545512-kube-api-access-k56tj\") pod \"autores-model-89e163-kserve-696c5f94f9-m8lk7\" (UID: \"aa7ca741-597a-4af5-8665-ad042b545512\") " pod="ai-tenant-e2e-autores-89e163/autores-model-89e163-kserve-696c5f94f9-m8lk7"
Sep 11 19:02:21.813352 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:21.813325    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa7ca741-597a-4af5-8665-ad042b545512-home\") pod \"autores-model-89e163-kserve-696c5f94f9-m8lk7\" (UID: \"aa7ca741-597a-4af5-8665-ad042b545512\") " pod="ai-tenant-e2e-autores-89e163/autores-model-89e163-kserve-696c5f94f9-m8lk7"
Sep 11 19:02:21.813429 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:21.813364    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa7ca741-597a-4af5-8665-ad042b545512-tmp-dir\") pod \"autores-model-89e163-kserve-696c5f94f9-m8lk7\" (UID: \"aa7ca741-597a-4af5-8665-ad042b545512\") " pod="ai-tenant-e2e-autores-89e163/autores-model-89e163-kserve-696c5f94f9-m8lk7"
Sep 11 19:02:21.813480 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:21.813423    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa7ca741-597a-4af5-8665-ad042b545512-model-cache\") pod \"autores-model-89e163-kserve-696c5f94f9-m8lk7\" (UID: \"aa7ca741-597a-4af5-8665-ad042b545512\") " pod="ai-tenant-e2e-autores-89e163/autores-model-89e163-kserve-696c5f94f9-m8lk7"
Sep 11 19:02:21.815444 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:21.815426    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa7ca741-597a-4af5-8665-ad042b545512-dshm\") pod \"autores-model-89e163-kserve-696c5f94f9-m8lk7\" (UID: \"aa7ca741-597a-4af5-8665-ad042b545512\") " pod="ai-tenant-e2e-autores-89e163/autores-model-89e163-kserve-696c5f94f9-m8lk7"
Sep 11 19:02:21.815635 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:21.815585    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa7ca741-597a-4af5-8665-ad042b545512-tls-certs\") pod \"autores-model-89e163-kserve-696c5f94f9-m8lk7\" (UID: \"aa7ca741-597a-4af5-8665-ad042b545512\") " pod="ai-tenant-e2e-autores-89e163/autores-model-89e163-kserve-696c5f94f9-m8lk7"
Sep 11 19:02:21.820721 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:21.820700    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k56tj\" (UniqueName: \"kubernetes.io/projected/aa7ca741-597a-4af5-8665-ad042b545512-kube-api-access-k56tj\") pod \"autores-model-89e163-kserve-696c5f94f9-m8lk7\" (UID: \"aa7ca741-597a-4af5-8665-ad042b545512\") " pod="ai-tenant-e2e-autores-89e163/autores-model-89e163-kserve-696c5f94f9-m8lk7"
Sep 11 19:02:21.939344 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:21.939263    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-89e163/autores-model-89e163-kserve-696c5f94f9-m8lk7"
Sep 11 19:02:22.210173 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:22.210149    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-89e163/autores-model-89e163-kserve-696c5f94f9-m8lk7"]
Sep 11 19:02:22.212391 ip-10-0-134-189 kubenswrapper[2578]: W0911 19:02:22.212364    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaa7ca741_597a_4af5_8665_ad042b545512.slice/crio-f1589fc0bdcb719a04acbcd6047f0be19b704bc1c2444bab717d0c7c59dcb826 WatchSource:0}: Error finding container f1589fc0bdcb719a04acbcd6047f0be19b704bc1c2444bab717d0c7c59dcb826: Status 404 returned error can't find the container with id f1589fc0bdcb719a04acbcd6047f0be19b704bc1c2444bab717d0c7c59dcb826
Sep 11 19:02:22.913957 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:22.913923    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-89e163/autores-model-89e163-kserve-696c5f94f9-m8lk7" event={"ID":"aa7ca741-597a-4af5-8665-ad042b545512","Type":"ContainerStarted","Data":"f2377f837c0bf154b0a166f273cdf4d5233d456a321ec03996f5b0d3a7c1462b"}
Sep 11 19:02:22.913957 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:22.913965    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-89e163/autores-model-89e163-kserve-696c5f94f9-m8lk7" event={"ID":"aa7ca741-597a-4af5-8665-ad042b545512","Type":"ContainerStarted","Data":"f1589fc0bdcb719a04acbcd6047f0be19b704bc1c2444bab717d0c7c59dcb826"}
Sep 11 19:02:22.933229 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:22.933166    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-89e163/autores-model-89e163-kserve-696c5f94f9-m8lk7" podStartSLOduration=1.933145839 podStartE2EDuration="1.933145839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:02:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:02:22.932084644 +0000 UTC m=+819.009675243" watchObservedRunningTime="2026-09-11 19:02:22.933145839 +0000 UTC m=+819.010736409"
Sep 11 19:02:31.940299 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:31.940265    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-89e163/autores-model-89e163-kserve-696c5f94f9-m8lk7"
Sep 11 19:02:31.940769 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:31.940314    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-89e163/autores-model-89e163-kserve-696c5f94f9-m8lk7"
Sep 11 19:02:31.952400 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:31.952377    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-89e163/autores-model-89e163-kserve-696c5f94f9-m8lk7"
Sep 11 19:02:31.969950 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:31.969919    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-89e163/autores-model-89e163-kserve-696c5f94f9-m8lk7"
Sep 11 19:02:38.914390 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:38.914356    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-c6c610/test-model-c6c610-kserve-5bfd568d7f-5mm5x"]
Sep 11 19:02:38.916668 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:38.916652    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c6c610/test-model-c6c610-kserve-5bfd568d7f-5mm5x"
Sep 11 19:02:38.918614 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:38.918567    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-c6c610\"/\"kube-root-ca.crt\""
Sep 11 19:02:38.918758 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:38.918685    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-c6c610\"/\"openshift-service-ca.crt\""
Sep 11 19:02:38.919308 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:38.919291    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-c6c610\"/\"test-model-c6c610-kserve-self-signed-certs\""
Sep 11 19:02:38.919415 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:38.919297    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-c6c610\"/\"default-dockercfg-phqtv\""
Sep 11 19:02:38.927443 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:38.927423    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-c6c610/test-model-c6c610-kserve-5bfd568d7f-5mm5x"]
Sep 11 19:02:38.992172 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:38.992138    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da1f5b99-5812-438f-a59b-dbd45c9ca454-home\") pod \"test-model-c6c610-kserve-5bfd568d7f-5mm5x\" (UID: \"da1f5b99-5812-438f-a59b-dbd45c9ca454\") " pod="ai-tenant-e2e-inf-a-c6c610/test-model-c6c610-kserve-5bfd568d7f-5mm5x"
Sep 11 19:02:38.992326 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:38.992225    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da1f5b99-5812-438f-a59b-dbd45c9ca454-tmp-dir\") pod \"test-model-c6c610-kserve-5bfd568d7f-5mm5x\" (UID: \"da1f5b99-5812-438f-a59b-dbd45c9ca454\") " pod="ai-tenant-e2e-inf-a-c6c610/test-model-c6c610-kserve-5bfd568d7f-5mm5x"
Sep 11 19:02:38.992326 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:38.992257    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da1f5b99-5812-438f-a59b-dbd45c9ca454-dshm\") pod \"test-model-c6c610-kserve-5bfd568d7f-5mm5x\" (UID: \"da1f5b99-5812-438f-a59b-dbd45c9ca454\") " pod="ai-tenant-e2e-inf-a-c6c610/test-model-c6c610-kserve-5bfd568d7f-5mm5x"
Sep 11 19:02:38.992326 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:38.992280    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da1f5b99-5812-438f-a59b-dbd45c9ca454-model-cache\") pod \"test-model-c6c610-kserve-5bfd568d7f-5mm5x\" (UID: \"da1f5b99-5812-438f-a59b-dbd45c9ca454\") " pod="ai-tenant-e2e-inf-a-c6c610/test-model-c6c610-kserve-5bfd568d7f-5mm5x"
Sep 11 19:02:38.992460 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:38.992361    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da1f5b99-5812-438f-a59b-dbd45c9ca454-tls-certs\") pod \"test-model-c6c610-kserve-5bfd568d7f-5mm5x\" (UID: \"da1f5b99-5812-438f-a59b-dbd45c9ca454\") " pod="ai-tenant-e2e-inf-a-c6c610/test-model-c6c610-kserve-5bfd568d7f-5mm5x"
Sep 11 19:02:38.992460 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:38.992403    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmt26\" (UniqueName: \"kubernetes.io/projected/da1f5b99-5812-438f-a59b-dbd45c9ca454-kube-api-access-nmt26\") pod \"test-model-c6c610-kserve-5bfd568d7f-5mm5x\" (UID: \"da1f5b99-5812-438f-a59b-dbd45c9ca454\") " pod="ai-tenant-e2e-inf-a-c6c610/test-model-c6c610-kserve-5bfd568d7f-5mm5x"
Sep 11 19:02:39.093276 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:39.093245    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da1f5b99-5812-438f-a59b-dbd45c9ca454-tmp-dir\") pod \"test-model-c6c610-kserve-5bfd568d7f-5mm5x\" (UID: \"da1f5b99-5812-438f-a59b-dbd45c9ca454\") " pod="ai-tenant-e2e-inf-a-c6c610/test-model-c6c610-kserve-5bfd568d7f-5mm5x"
Sep 11 19:02:39.093446 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:39.093289    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da1f5b99-5812-438f-a59b-dbd45c9ca454-dshm\") pod \"test-model-c6c610-kserve-5bfd568d7f-5mm5x\" (UID: \"da1f5b99-5812-438f-a59b-dbd45c9ca454\") " pod="ai-tenant-e2e-inf-a-c6c610/test-model-c6c610-kserve-5bfd568d7f-5mm5x"
Sep 11 19:02:39.093446 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:39.093311    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da1f5b99-5812-438f-a59b-dbd45c9ca454-model-cache\") pod \"test-model-c6c610-kserve-5bfd568d7f-5mm5x\" (UID: \"da1f5b99-5812-438f-a59b-dbd45c9ca454\") " pod="ai-tenant-e2e-inf-a-c6c610/test-model-c6c610-kserve-5bfd568d7f-5mm5x"
Sep 11 19:02:39.093446 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:39.093347    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da1f5b99-5812-438f-a59b-dbd45c9ca454-tls-certs\") pod \"test-model-c6c610-kserve-5bfd568d7f-5mm5x\" (UID: \"da1f5b99-5812-438f-a59b-dbd45c9ca454\") " pod="ai-tenant-e2e-inf-a-c6c610/test-model-c6c610-kserve-5bfd568d7f-5mm5x"
Sep 11 19:02:39.093446 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:39.093381    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmt26\" (UniqueName: \"kubernetes.io/projected/da1f5b99-5812-438f-a59b-dbd45c9ca454-kube-api-access-nmt26\") pod \"test-model-c6c610-kserve-5bfd568d7f-5mm5x\" (UID: \"da1f5b99-5812-438f-a59b-dbd45c9ca454\") " pod="ai-tenant-e2e-inf-a-c6c610/test-model-c6c610-kserve-5bfd568d7f-5mm5x"
Sep 11 19:02:39.093446 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:39.093432    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da1f5b99-5812-438f-a59b-dbd45c9ca454-home\") pod \"test-model-c6c610-kserve-5bfd568d7f-5mm5x\" (UID: \"da1f5b99-5812-438f-a59b-dbd45c9ca454\") " pod="ai-tenant-e2e-inf-a-c6c610/test-model-c6c610-kserve-5bfd568d7f-5mm5x"
Sep 11 19:02:39.093822 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:39.093795    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da1f5b99-5812-438f-a59b-dbd45c9ca454-model-cache\") pod \"test-model-c6c610-kserve-5bfd568d7f-5mm5x\" (UID: \"da1f5b99-5812-438f-a59b-dbd45c9ca454\") " pod="ai-tenant-e2e-inf-a-c6c610/test-model-c6c610-kserve-5bfd568d7f-5mm5x"
Sep 11 19:02:39.093937 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:39.093918    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da1f5b99-5812-438f-a59b-dbd45c9ca454-tmp-dir\") pod \"test-model-c6c610-kserve-5bfd568d7f-5mm5x\" (UID: \"da1f5b99-5812-438f-a59b-dbd45c9ca454\") " pod="ai-tenant-e2e-inf-a-c6c610/test-model-c6c610-kserve-5bfd568d7f-5mm5x"
Sep 11 19:02:39.094071 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:39.094051    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da1f5b99-5812-438f-a59b-dbd45c9ca454-home\") pod \"test-model-c6c610-kserve-5bfd568d7f-5mm5x\" (UID: \"da1f5b99-5812-438f-a59b-dbd45c9ca454\") " pod="ai-tenant-e2e-inf-a-c6c610/test-model-c6c610-kserve-5bfd568d7f-5mm5x"
Sep 11 19:02:39.095652 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:39.095631    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da1f5b99-5812-438f-a59b-dbd45c9ca454-dshm\") pod \"test-model-c6c610-kserve-5bfd568d7f-5mm5x\" (UID: \"da1f5b99-5812-438f-a59b-dbd45c9ca454\") " pod="ai-tenant-e2e-inf-a-c6c610/test-model-c6c610-kserve-5bfd568d7f-5mm5x"
Sep 11 19:02:39.096256 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:39.096237    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da1f5b99-5812-438f-a59b-dbd45c9ca454-tls-certs\") pod \"test-model-c6c610-kserve-5bfd568d7f-5mm5x\" (UID: \"da1f5b99-5812-438f-a59b-dbd45c9ca454\") " pod="ai-tenant-e2e-inf-a-c6c610/test-model-c6c610-kserve-5bfd568d7f-5mm5x"
Sep 11 19:02:39.101391 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:39.101365    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmt26\" (UniqueName: \"kubernetes.io/projected/da1f5b99-5812-438f-a59b-dbd45c9ca454-kube-api-access-nmt26\") pod \"test-model-c6c610-kserve-5bfd568d7f-5mm5x\" (UID: \"da1f5b99-5812-438f-a59b-dbd45c9ca454\") " pod="ai-tenant-e2e-inf-a-c6c610/test-model-c6c610-kserve-5bfd568d7f-5mm5x"
Sep 11 19:02:39.226533 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:39.226499    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c6c610/test-model-c6c610-kserve-5bfd568d7f-5mm5x"
Sep 11 19:02:39.518965 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:39.518941    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-c6c610/test-model-c6c610-kserve-5bfd568d7f-5mm5x"]
Sep 11 19:02:39.520718 ip-10-0-134-189 kubenswrapper[2578]: W0911 19:02:39.520689    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podda1f5b99_5812_438f_a59b_dbd45c9ca454.slice/crio-9304ea7da662078de5446b3ea552ae6db7b6d06cac0f1cd0bad22779dfff9c23 WatchSource:0}: Error finding container 9304ea7da662078de5446b3ea552ae6db7b6d06cac0f1cd0bad22779dfff9c23: Status 404 returned error can't find the container with id 9304ea7da662078de5446b3ea552ae6db7b6d06cac0f1cd0bad22779dfff9c23
Sep 11 19:02:40.002849 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:40.002812    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c6c610/test-model-c6c610-kserve-5bfd568d7f-5mm5x" event={"ID":"da1f5b99-5812-438f-a59b-dbd45c9ca454","Type":"ContainerStarted","Data":"9b766c5f8f3f7f4b0275ea21fa2ccea56b698d3e35e3becf868053d9e2827f8a"}
Sep 11 19:02:40.002849 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:40.002853    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c6c610/test-model-c6c610-kserve-5bfd568d7f-5mm5x" event={"ID":"da1f5b99-5812-438f-a59b-dbd45c9ca454","Type":"ContainerStarted","Data":"9304ea7da662078de5446b3ea552ae6db7b6d06cac0f1cd0bad22779dfff9c23"}
Sep 11 19:02:40.017884 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:40.017816    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-c6c610/test-model-c6c610-kserve-5bfd568d7f-5mm5x" podStartSLOduration=2.017800238 podStartE2EDuration="2.017800238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:02:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:02:40.017414084 +0000 UTC m=+836.095004655" watchObservedRunningTime="2026-09-11 19:02:40.017800238 +0000 UTC m=+836.095390792"
Sep 11 19:02:45.030002 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:45.029967    2578 generic.go:356] "Generic (PLEG): container finished" podID="7fd0dce6-a2cb-4c83-bb92-35309ea2b40e" containerID="96e95add969e6fd890ae7a0cffc2815daab26275ea299a0372d39e8eb3cf5dfa" exitCode=0
Sep 11 19:02:45.030395 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:45.030010    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f0250e49-openshift-default-664bb69d6-p626z" event={"ID":"7fd0dce6-a2cb-4c83-bb92-35309ea2b40e","Type":"ContainerDied","Data":"96e95add969e6fd890ae7a0cffc2815daab26275ea299a0372d39e8eb3cf5dfa"}
Sep 11 19:02:45.030448 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:45.030433    2578 scope.go:117] "RemoveContainer" containerID="96e95add969e6fd890ae7a0cffc2815daab26275ea299a0372d39e8eb3cf5dfa"
Sep 11 19:02:45.031519 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:45.031488    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 19:02:45.031573 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:45.031559    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 19:02:45.031636 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:45.031619    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 19:02:46.037230 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:46.037202    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f0250e49-openshift-default-664bb69d6-p626z" event={"ID":"7fd0dce6-a2cb-4c83-bb92-35309ea2b40e","Type":"ContainerStarted","Data":"7735bb19bb142432b065879ca30545bdf9338ece4a734790916bc7966c77445f"}
Sep 11 19:02:46.160566 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:46.160147    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-f0250e49-openshift-default-664bb69d6-p626z"
Sep 11 19:02:46.160566 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:46.160278    2578 patch_prober.go:28] interesting pod/e2e-ait-f0250e49-openshift-default-664bb69d6-p626z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Sep 11 19:02:46.160566 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:46.160322    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f0250e49-openshift-default-664bb69d6-p626z" podUID="7fd0dce6-a2cb-4c83-bb92-35309ea2b40e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Sep 11 19:02:46.933307 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:46.933274    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0250e49-tmd6h"]
Sep 11 19:02:46.936092 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:46.936075    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0250e49-tmd6h"
Sep 11 19:02:46.938271 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:46.938239    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 19:02:46.938489 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:46.938477    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ln295\""
Sep 11 19:02:46.943632 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:46.943609    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0250e49-tmd6h"]
Sep 11 19:02:46.983623 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:46.981135    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/575396b2-df84-428b-8274-701b9852c87f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f0250e49-tmd6h\" (UID: \"575396b2-df84-428b-8274-701b9852c87f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0250e49-tmd6h"
Sep 11 19:02:47.084042 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:47.084013    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/575396b2-df84-428b-8274-701b9852c87f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f0250e49-tmd6h\" (UID: \"575396b2-df84-428b-8274-701b9852c87f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0250e49-tmd6h"
Sep 11 19:02:47.084705 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:47.084684    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/575396b2-df84-428b-8274-701b9852c87f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f0250e49-tmd6h\" (UID: \"575396b2-df84-428b-8274-701b9852c87f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0250e49-tmd6h"
Sep 11 19:02:47.159647 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:47.159572    2578 patch_prober.go:28] interesting pod/e2e-ait-f0250e49-openshift-default-664bb69d6-p626z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Sep 11 19:02:47.160346 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:47.159913    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f0250e49-openshift-default-664bb69d6-p626z" podUID="7fd0dce6-a2cb-4c83-bb92-35309ea2b40e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Sep 11 19:02:47.247474 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:47.247443    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0250e49-tmd6h"
Sep 11 19:02:47.532385 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:47.532305    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0250e49-tmd6h"]
Sep 11 19:02:47.535159 ip-10-0-134-189 kubenswrapper[2578]: W0911 19:02:47.535127    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod575396b2_df84_428b_8274_701b9852c87f.slice/crio-41e6438ceac0cc99fd50fd6663179dc5a157ff14b5b28518d6200dc25e665e5d WatchSource:0}: Error finding container 41e6438ceac0cc99fd50fd6663179dc5a157ff14b5b28518d6200dc25e665e5d: Status 404 returned error can't find the container with id 41e6438ceac0cc99fd50fd6663179dc5a157ff14b5b28518d6200dc25e665e5d
Sep 11 19:02:48.054374 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:48.054336    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0250e49-tmd6h" event={"ID":"575396b2-df84-428b-8274-701b9852c87f","Type":"ContainerStarted","Data":"41e6438ceac0cc99fd50fd6663179dc5a157ff14b5b28518d6200dc25e665e5d"}
Sep 11 19:02:48.159509 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:48.159395    2578 patch_prober.go:28] interesting pod/e2e-ait-f0250e49-openshift-default-664bb69d6-p626z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Sep 11 19:02:48.159509 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:48.159485    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f0250e49-openshift-default-664bb69d6-p626z" podUID="7fd0dce6-a2cb-4c83-bb92-35309ea2b40e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Sep 11 19:02:49.061676 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:49.061635    2578 generic.go:356] "Generic (PLEG): container finished" podID="575396b2-df84-428b-8274-701b9852c87f" containerID="9f6ce7c1bc63cb265bf0bb4a833cb3585e2c96027d77be50948afd31d03d7af2" exitCode=0
Sep 11 19:02:49.061930 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:49.061740    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0250e49-tmd6h" event={"ID":"575396b2-df84-428b-8274-701b9852c87f","Type":"ContainerDied","Data":"9f6ce7c1bc63cb265bf0bb4a833cb3585e2c96027d77be50948afd31d03d7af2"}
Sep 11 19:02:49.158953 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:49.158909    2578 patch_prober.go:28] interesting pod/e2e-ait-f0250e49-openshift-default-664bb69d6-p626z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Sep 11 19:02:49.159152 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:49.158994    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f0250e49-openshift-default-664bb69d6-p626z" podUID="7fd0dce6-a2cb-4c83-bb92-35309ea2b40e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Sep 11 19:02:49.227342 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:49.227309    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-c6c610/test-model-c6c610-kserve-5bfd568d7f-5mm5x"
Sep 11 19:02:49.227806 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:49.227371    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-c6c610/test-model-c6c610-kserve-5bfd568d7f-5mm5x"
Sep 11 19:02:49.239745 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:49.239721    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-c6c610/test-model-c6c610-kserve-5bfd568d7f-5mm5x"
Sep 11 19:02:50.095389 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:50.095364    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-c6c610/test-model-c6c610-kserve-5bfd568d7f-5mm5x"
Sep 11 19:02:50.166736 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:50.166706    2578 patch_prober.go:28] interesting pod/e2e-ait-f0250e49-openshift-default-664bb69d6-p626z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Sep 11 19:02:50.167004 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:50.166975    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f0250e49-openshift-default-664bb69d6-p626z" podUID="7fd0dce6-a2cb-4c83-bb92-35309ea2b40e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Sep 11 19:02:50.241613 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:50.241580    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0250e49-tmd6h"
Sep 11 19:02:50.315269 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:50.315241    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/575396b2-df84-428b-8274-701b9852c87f-maas-api-service-ca\") pod \"575396b2-df84-428b-8274-701b9852c87f\" (UID: \"575396b2-df84-428b-8274-701b9852c87f\") "
Sep 11 19:02:50.315552 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:50.315531    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/575396b2-df84-428b-8274-701b9852c87f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "575396b2-df84-428b-8274-701b9852c87f" (UID: "575396b2-df84-428b-8274-701b9852c87f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 19:02:50.416357 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:50.416332    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/575396b2-df84-428b-8274-701b9852c87f-maas-api-service-ca\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:02:50.899614 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:50.899554    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f0250e49-openshift-default-664bb69d6-p626z"
Sep 11 19:02:51.080180 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:51.080147    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0250e49-tmd6h"
Sep 11 19:02:51.080382 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:51.080182    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0250e49-tmd6h" event={"ID":"575396b2-df84-428b-8274-701b9852c87f","Type":"ContainerDied","Data":"41e6438ceac0cc99fd50fd6663179dc5a157ff14b5b28518d6200dc25e665e5d"}
Sep 11 19:02:51.080382 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:51.080244    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="41e6438ceac0cc99fd50fd6663179dc5a157ff14b5b28518d6200dc25e665e5d"
Sep 11 19:02:51.162912 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:51.162841    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-f0250e49-openshift-default-664bb69d6-p626z"
Sep 11 19:02:51.163673 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:51.163657    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-f0250e49-openshift-default-664bb69d6-p626z"
Sep 11 19:02:52.929764 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:52.929732    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0250e49-tmd6h"]
Sep 11 19:02:52.939150 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:52.939121    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0250e49-tmd6h"]
Sep 11 19:02:53.993665 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:53.993631    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f0250e49-openshift-default-664bb69d6-p626z"]
Sep 11 19:02:53.994103 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:53.993903    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-f0250e49-openshift-default-664bb69d6-p626z" podUID="7fd0dce6-a2cb-4c83-bb92-35309ea2b40e" containerName="istio-proxy" containerID="cri-o://7735bb19bb142432b065879ca30545bdf9338ece4a734790916bc7966c77445f" gracePeriod=30
Sep 11 19:02:54.396465 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:54.396391    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="575396b2-df84-428b-8274-701b9852c87f" path="/var/lib/kubelet/pods/575396b2-df84-428b-8274-701b9852c87f/volumes"
Sep 11 19:02:59.125504 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:59.125475    2578 generic.go:356] "Generic (PLEG): container finished" podID="7fd0dce6-a2cb-4c83-bb92-35309ea2b40e" containerID="7735bb19bb142432b065879ca30545bdf9338ece4a734790916bc7966c77445f" exitCode=0
Sep 11 19:02:59.125912 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:59.125549    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f0250e49-openshift-default-664bb69d6-p626z" event={"ID":"7fd0dce6-a2cb-4c83-bb92-35309ea2b40e","Type":"ContainerDied","Data":"7735bb19bb142432b065879ca30545bdf9338ece4a734790916bc7966c77445f"}
Sep 11 19:02:59.125912 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:59.125676    2578 scope.go:117] "RemoveContainer" containerID="96e95add969e6fd890ae7a0cffc2815daab26275ea299a0372d39e8eb3cf5dfa"
Sep 11 19:02:59.253941 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:59.253912    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f0250e49-openshift-default-664bb69d6-p626z"
Sep 11 19:02:59.408485 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:59.408402    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d7wxr\" (UniqueName: \"kubernetes.io/projected/7fd0dce6-a2cb-4c83-bb92-35309ea2b40e-kube-api-access-d7wxr\") pod \"7fd0dce6-a2cb-4c83-bb92-35309ea2b40e\" (UID: \"7fd0dce6-a2cb-4c83-bb92-35309ea2b40e\") "
Sep 11 19:02:59.408485 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:59.408479    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fd0dce6-a2cb-4c83-bb92-35309ea2b40e-workload-certs\") pod \"7fd0dce6-a2cb-4c83-bb92-35309ea2b40e\" (UID: \"7fd0dce6-a2cb-4c83-bb92-35309ea2b40e\") "
Sep 11 19:02:59.408749 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:59.408503    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fd0dce6-a2cb-4c83-bb92-35309ea2b40e-istio-podinfo\") pod \"7fd0dce6-a2cb-4c83-bb92-35309ea2b40e\" (UID: \"7fd0dce6-a2cb-4c83-bb92-35309ea2b40e\") "
Sep 11 19:02:59.408749 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:59.408525    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fd0dce6-a2cb-4c83-bb92-35309ea2b40e-istio-envoy\") pod \"7fd0dce6-a2cb-4c83-bb92-35309ea2b40e\" (UID: \"7fd0dce6-a2cb-4c83-bb92-35309ea2b40e\") "
Sep 11 19:02:59.408749 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:59.408541    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fd0dce6-a2cb-4c83-bb92-35309ea2b40e-workload-socket\") pod \"7fd0dce6-a2cb-4c83-bb92-35309ea2b40e\" (UID: \"7fd0dce6-a2cb-4c83-bb92-35309ea2b40e\") "
Sep 11 19:02:59.408749 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:59.408677    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fd0dce6-a2cb-4c83-bb92-35309ea2b40e-istiod-ca-cert\") pod \"7fd0dce6-a2cb-4c83-bb92-35309ea2b40e\" (UID: \"7fd0dce6-a2cb-4c83-bb92-35309ea2b40e\") "
Sep 11 19:02:59.408749 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:59.408723    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fd0dce6-a2cb-4c83-bb92-35309ea2b40e-istio-token\") pod \"7fd0dce6-a2cb-4c83-bb92-35309ea2b40e\" (UID: \"7fd0dce6-a2cb-4c83-bb92-35309ea2b40e\") "
Sep 11 19:02:59.408749 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:59.408748    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fd0dce6-a2cb-4c83-bb92-35309ea2b40e-istio-data\") pod \"7fd0dce6-a2cb-4c83-bb92-35309ea2b40e\" (UID: \"7fd0dce6-a2cb-4c83-bb92-35309ea2b40e\") "
Sep 11 19:02:59.409058 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:59.408857    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fd0dce6-a2cb-4c83-bb92-35309ea2b40e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7fd0dce6-a2cb-4c83-bb92-35309ea2b40e" (UID: "7fd0dce6-a2cb-4c83-bb92-35309ea2b40e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:02:59.409058 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:59.408875    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fd0dce6-a2cb-4c83-bb92-35309ea2b40e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7fd0dce6-a2cb-4c83-bb92-35309ea2b40e" (UID: "7fd0dce6-a2cb-4c83-bb92-35309ea2b40e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:02:59.409058 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:59.408926    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fd0dce6-a2cb-4c83-bb92-35309ea2b40e-credential-socket\") pod \"7fd0dce6-a2cb-4c83-bb92-35309ea2b40e\" (UID: \"7fd0dce6-a2cb-4c83-bb92-35309ea2b40e\") "
Sep 11 19:02:59.409058 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:59.408973    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fd0dce6-a2cb-4c83-bb92-35309ea2b40e-istio-data" (OuterVolumeSpecName: "istio-data") pod "7fd0dce6-a2cb-4c83-bb92-35309ea2b40e" (UID: "7fd0dce6-a2cb-4c83-bb92-35309ea2b40e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:02:59.409058 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:59.409013    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fd0dce6-a2cb-4c83-bb92-35309ea2b40e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7fd0dce6-a2cb-4c83-bb92-35309ea2b40e" (UID: "7fd0dce6-a2cb-4c83-bb92-35309ea2b40e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 19:02:59.409464 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:59.409155    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fd0dce6-a2cb-4c83-bb92-35309ea2b40e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7fd0dce6-a2cb-4c83-bb92-35309ea2b40e" (UID: "7fd0dce6-a2cb-4c83-bb92-35309ea2b40e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:02:59.409464 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:59.409267    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fd0dce6-a2cb-4c83-bb92-35309ea2b40e-workload-socket\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:02:59.409464 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:59.409282    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fd0dce6-a2cb-4c83-bb92-35309ea2b40e-istiod-ca-cert\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:02:59.409464 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:59.409294    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fd0dce6-a2cb-4c83-bb92-35309ea2b40e-istio-data\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:02:59.409464 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:59.409306    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fd0dce6-a2cb-4c83-bb92-35309ea2b40e-credential-socket\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:02:59.409464 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:59.409319    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fd0dce6-a2cb-4c83-bb92-35309ea2b40e-workload-certs\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:02:59.410912 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:59.410887    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fd0dce6-a2cb-4c83-bb92-35309ea2b40e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7fd0dce6-a2cb-4c83-bb92-35309ea2b40e" (UID: "7fd0dce6-a2cb-4c83-bb92-35309ea2b40e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:02:59.410912 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:59.410891    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7fd0dce6-a2cb-4c83-bb92-35309ea2b40e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7fd0dce6-a2cb-4c83-bb92-35309ea2b40e" (UID: "7fd0dce6-a2cb-4c83-bb92-35309ea2b40e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 19:02:59.411404 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:59.411371    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fd0dce6-a2cb-4c83-bb92-35309ea2b40e-kube-api-access-d7wxr" (OuterVolumeSpecName: "kube-api-access-d7wxr") pod "7fd0dce6-a2cb-4c83-bb92-35309ea2b40e" (UID: "7fd0dce6-a2cb-4c83-bb92-35309ea2b40e"). InnerVolumeSpecName "kube-api-access-d7wxr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:02:59.411404 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:59.411384    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fd0dce6-a2cb-4c83-bb92-35309ea2b40e-istio-token" (OuterVolumeSpecName: "istio-token") pod "7fd0dce6-a2cb-4c83-bb92-35309ea2b40e" (UID: "7fd0dce6-a2cb-4c83-bb92-35309ea2b40e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:02:59.509929 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:59.509899    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fd0dce6-a2cb-4c83-bb92-35309ea2b40e-istio-podinfo\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:02:59.509929 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:59.509925    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fd0dce6-a2cb-4c83-bb92-35309ea2b40e-istio-envoy\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:02:59.509929 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:59.509934    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fd0dce6-a2cb-4c83-bb92-35309ea2b40e-istio-token\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:02:59.510157 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:02:59.509942    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7wxr\" (UniqueName: \"kubernetes.io/projected/7fd0dce6-a2cb-4c83-bb92-35309ea2b40e-kube-api-access-d7wxr\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:03:00.131996 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:00.131967    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f0250e49-openshift-default-664bb69d6-p626z"
Sep 11 19:03:00.131996 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:00.131982    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f0250e49-openshift-default-664bb69d6-p626z" event={"ID":"7fd0dce6-a2cb-4c83-bb92-35309ea2b40e","Type":"ContainerDied","Data":"427b4c2e429d142ebb4316c6aadc604c441a999e0e86986c11d647c5cd563d38"}
Sep 11 19:03:00.132498 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:00.132046    2578 scope.go:117] "RemoveContainer" containerID="7735bb19bb142432b065879ca30545bdf9338ece4a734790916bc7966c77445f"
Sep 11 19:03:00.152183 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:00.152159    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f0250e49-openshift-default-664bb69d6-p626z"]
Sep 11 19:03:00.156999 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:00.156977    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-f0250e49-openshift-default-664bb69d6-p626z"]
Sep 11 19:03:00.396713 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:00.396637    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7fd0dce6-a2cb-4c83-bb92-35309ea2b40e" path="/var/lib/kubelet/pods/7fd0dce6-a2cb-4c83-bb92-35309ea2b40e/volumes"
Sep 11 19:03:07.593865 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:07.593827    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-89e163-5cf97d8fcc-7qq94"]
Sep 11 19:03:07.594267 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:07.594094    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-89e163-5cf97d8fcc-7qq94" podUID="2730645f-35cd-486a-b110-e4b2848593d0" containerName="payload-processing" containerID="cri-o://2d027bfb9be4e7e2f5b2391aa7fe23c29b8707f2c08f9641709d71163f8fde49" gracePeriod=30
Sep 11 19:03:07.844385 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:07.844326    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-89e163-5cf97d8fcc-7qq94"
Sep 11 19:03:07.987811 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:07.987760    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9zxx2\" (UniqueName: \"kubernetes.io/projected/2730645f-35cd-486a-b110-e4b2848593d0-kube-api-access-9zxx2\") pod \"2730645f-35cd-486a-b110-e4b2848593d0\" (UID: \"2730645f-35cd-486a-b110-e4b2848593d0\") "
Sep 11 19:03:07.987992 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:07.987870    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2730645f-35cd-486a-b110-e4b2848593d0-plugins-config-volume\") pod \"2730645f-35cd-486a-b110-e4b2848593d0\" (UID: \"2730645f-35cd-486a-b110-e4b2848593d0\") "
Sep 11 19:03:07.988206 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:07.988185    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2730645f-35cd-486a-b110-e4b2848593d0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2730645f-35cd-486a-b110-e4b2848593d0" (UID: "2730645f-35cd-486a-b110-e4b2848593d0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 19:03:07.990190 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:07.990167    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2730645f-35cd-486a-b110-e4b2848593d0-kube-api-access-9zxx2" (OuterVolumeSpecName: "kube-api-access-9zxx2") pod "2730645f-35cd-486a-b110-e4b2848593d0" (UID: "2730645f-35cd-486a-b110-e4b2848593d0"). InnerVolumeSpecName "kube-api-access-9zxx2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:03:08.089168 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:08.089111    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zxx2\" (UniqueName: \"kubernetes.io/projected/2730645f-35cd-486a-b110-e4b2848593d0-kube-api-access-9zxx2\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:03:08.089168 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:08.089160    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2730645f-35cd-486a-b110-e4b2848593d0-plugins-config-volume\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:03:08.172673 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:08.172568    2578 generic.go:356] "Generic (PLEG): container finished" podID="2730645f-35cd-486a-b110-e4b2848593d0" containerID="2d027bfb9be4e7e2f5b2391aa7fe23c29b8707f2c08f9641709d71163f8fde49" exitCode=0
Sep 11 19:03:08.172673 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:08.172631    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-89e163-5cf97d8fcc-7qq94"
Sep 11 19:03:08.172673 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:08.172632    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-89e163-5cf97d8fcc-7qq94" event={"ID":"2730645f-35cd-486a-b110-e4b2848593d0","Type":"ContainerDied","Data":"2d027bfb9be4e7e2f5b2391aa7fe23c29b8707f2c08f9641709d71163f8fde49"}
Sep 11 19:03:08.172922 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:08.172693    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-89e163-5cf97d8fcc-7qq94" event={"ID":"2730645f-35cd-486a-b110-e4b2848593d0","Type":"ContainerDied","Data":"f327d70b2b786b66390d7aca909cebea29d3effeb3068bd9790aba68a1dd35f2"}
Sep 11 19:03:08.172922 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:08.172707    2578 scope.go:117] "RemoveContainer" containerID="2d027bfb9be4e7e2f5b2391aa7fe23c29b8707f2c08f9641709d71163f8fde49"
Sep 11 19:03:08.181643 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:08.181624    2578 scope.go:117] "RemoveContainer" containerID="2d027bfb9be4e7e2f5b2391aa7fe23c29b8707f2c08f9641709d71163f8fde49"
Sep 11 19:03:08.181916 ip-10-0-134-189 kubenswrapper[2578]: E0911 19:03:08.181897    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d027bfb9be4e7e2f5b2391aa7fe23c29b8707f2c08f9641709d71163f8fde49\": container with ID starting with 2d027bfb9be4e7e2f5b2391aa7fe23c29b8707f2c08f9641709d71163f8fde49 not found: ID does not exist" containerID="2d027bfb9be4e7e2f5b2391aa7fe23c29b8707f2c08f9641709d71163f8fde49"
Sep 11 19:03:08.181962 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:08.181926    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d027bfb9be4e7e2f5b2391aa7fe23c29b8707f2c08f9641709d71163f8fde49"} err="failed to get container status \"2d027bfb9be4e7e2f5b2391aa7fe23c29b8707f2c08f9641709d71163f8fde49\": rpc error: code = NotFound desc = could not find container \"2d027bfb9be4e7e2f5b2391aa7fe23c29b8707f2c08f9641709d71163f8fde49\": container with ID starting with 2d027bfb9be4e7e2f5b2391aa7fe23c29b8707f2c08f9641709d71163f8fde49 not found: ID does not exist"
Sep 11 19:03:08.201842 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:08.201816    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-89e163-5cf97d8fcc-7qq94"]
Sep 11 19:03:08.219580 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:08.219553    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-89e163-5cf97d8fcc-7qq94"]
Sep 11 19:03:08.396714 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:08.396685    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2730645f-35cd-486a-b110-e4b2848593d0" path="/var/lib/kubelet/pods/2730645f-35cd-486a-b110-e4b2848593d0/volumes"
Sep 11 19:03:12.284186 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:12.284154    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-89e163/autores-model-89e163-kserve-696c5f94f9-m8lk7"]
Sep 11 19:03:27.220308 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:27.220270    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8092ee69-xbdnd"]
Sep 11 19:03:27.220765 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:27.220657    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2730645f-35cd-486a-b110-e4b2848593d0" containerName="payload-processing"
Sep 11 19:03:27.220765 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:27.220674    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2730645f-35cd-486a-b110-e4b2848593d0" containerName="payload-processing"
Sep 11 19:03:27.220765 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:27.220686    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="575396b2-df84-428b-8274-701b9852c87f" containerName="revoke-keys"
Sep 11 19:03:27.220765 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:27.220695    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="575396b2-df84-428b-8274-701b9852c87f" containerName="revoke-keys"
Sep 11 19:03:27.220765 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:27.220714    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fd0dce6-a2cb-4c83-bb92-35309ea2b40e" containerName="istio-proxy"
Sep 11 19:03:27.220765 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:27.220722    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fd0dce6-a2cb-4c83-bb92-35309ea2b40e" containerName="istio-proxy"
Sep 11 19:03:27.220765 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:27.220735    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fd0dce6-a2cb-4c83-bb92-35309ea2b40e" containerName="istio-proxy"
Sep 11 19:03:27.220765 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:27.220743    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fd0dce6-a2cb-4c83-bb92-35309ea2b40e" containerName="istio-proxy"
Sep 11 19:03:27.221043 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:27.220832    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="575396b2-df84-428b-8274-701b9852c87f" containerName="revoke-keys"
Sep 11 19:03:27.221043 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:27.220843    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2730645f-35cd-486a-b110-e4b2848593d0" containerName="payload-processing"
Sep 11 19:03:27.221043 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:27.220851    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fd0dce6-a2cb-4c83-bb92-35309ea2b40e" containerName="istio-proxy"
Sep 11 19:03:27.221043 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:27.220857    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fd0dce6-a2cb-4c83-bb92-35309ea2b40e" containerName="istio-proxy"
Sep 11 19:03:27.223562 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:27.223545    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8092ee69-xbdnd"
Sep 11 19:03:27.225460 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:27.225439    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ln295\""
Sep 11 19:03:27.225567 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:27.225483    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 19:03:27.231647 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:27.231145    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8092ee69-xbdnd"]
Sep 11 19:03:27.268234 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:27.268160    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53ccfe18-2ffd-4462-a08c-814acee1d87b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-8092ee69-xbdnd\" (UID: \"53ccfe18-2ffd-4462-a08c-814acee1d87b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8092ee69-xbdnd"
Sep 11 19:03:27.303678 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:27.303622    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-89e163/autores-model-89e163-kserve-696c5f94f9-m8lk7" podUID="aa7ca741-597a-4af5-8665-ad042b545512" containerName="main" containerID="cri-o://f2377f837c0bf154b0a166f273cdf4d5233d456a321ec03996f5b0d3a7c1462b" gracePeriod=45
Sep 11 19:03:27.369431 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:27.369407    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53ccfe18-2ffd-4462-a08c-814acee1d87b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-8092ee69-xbdnd\" (UID: \"53ccfe18-2ffd-4462-a08c-814acee1d87b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8092ee69-xbdnd"
Sep 11 19:03:27.369982 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:27.369963    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53ccfe18-2ffd-4462-a08c-814acee1d87b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-8092ee69-xbdnd\" (UID: \"53ccfe18-2ffd-4462-a08c-814acee1d87b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8092ee69-xbdnd"
Sep 11 19:03:27.535193 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:27.535162    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8092ee69-xbdnd"
Sep 11 19:03:27.561572 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:27.561541    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-89e163/autores-model-89e163-kserve-696c5f94f9-m8lk7"
Sep 11 19:03:27.673249 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:27.673219    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa7ca741-597a-4af5-8665-ad042b545512-tmp-dir\") pod \"aa7ca741-597a-4af5-8665-ad042b545512\" (UID: \"aa7ca741-597a-4af5-8665-ad042b545512\") "
Sep 11 19:03:27.673249 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:27.673255    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k56tj\" (UniqueName: \"kubernetes.io/projected/aa7ca741-597a-4af5-8665-ad042b545512-kube-api-access-k56tj\") pod \"aa7ca741-597a-4af5-8665-ad042b545512\" (UID: \"aa7ca741-597a-4af5-8665-ad042b545512\") "
Sep 11 19:03:27.673540 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:27.673299    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa7ca741-597a-4af5-8665-ad042b545512-dshm\") pod \"aa7ca741-597a-4af5-8665-ad042b545512\" (UID: \"aa7ca741-597a-4af5-8665-ad042b545512\") "
Sep 11 19:03:27.673540 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:27.673369    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa7ca741-597a-4af5-8665-ad042b545512-tls-certs\") pod \"aa7ca741-597a-4af5-8665-ad042b545512\" (UID: \"aa7ca741-597a-4af5-8665-ad042b545512\") "
Sep 11 19:03:27.673540 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:27.673403    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa7ca741-597a-4af5-8665-ad042b545512-model-cache\") pod \"aa7ca741-597a-4af5-8665-ad042b545512\" (UID: \"aa7ca741-597a-4af5-8665-ad042b545512\") "
Sep 11 19:03:27.673540 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:27.673452    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa7ca741-597a-4af5-8665-ad042b545512-home\") pod \"aa7ca741-597a-4af5-8665-ad042b545512\" (UID: \"aa7ca741-597a-4af5-8665-ad042b545512\") "
Sep 11 19:03:27.673540 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:27.673480    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa7ca741-597a-4af5-8665-ad042b545512-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "aa7ca741-597a-4af5-8665-ad042b545512" (UID: "aa7ca741-597a-4af5-8665-ad042b545512"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:03:27.673862 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:27.673708    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa7ca741-597a-4af5-8665-ad042b545512-tmp-dir\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:03:27.673862 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:27.673747    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa7ca741-597a-4af5-8665-ad042b545512-model-cache" (OuterVolumeSpecName: "model-cache") pod "aa7ca741-597a-4af5-8665-ad042b545512" (UID: "aa7ca741-597a-4af5-8665-ad042b545512"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:03:27.673862 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:27.673795    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa7ca741-597a-4af5-8665-ad042b545512-home" (OuterVolumeSpecName: "home") pod "aa7ca741-597a-4af5-8665-ad042b545512" (UID: "aa7ca741-597a-4af5-8665-ad042b545512"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:03:27.675689 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:27.675663    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa7ca741-597a-4af5-8665-ad042b545512-dshm" (OuterVolumeSpecName: "dshm") pod "aa7ca741-597a-4af5-8665-ad042b545512" (UID: "aa7ca741-597a-4af5-8665-ad042b545512"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:03:27.675788 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:27.675715    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa7ca741-597a-4af5-8665-ad042b545512-kube-api-access-k56tj" (OuterVolumeSpecName: "kube-api-access-k56tj") pod "aa7ca741-597a-4af5-8665-ad042b545512" (UID: "aa7ca741-597a-4af5-8665-ad042b545512"). InnerVolumeSpecName "kube-api-access-k56tj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:03:27.675788 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:27.675736    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa7ca741-597a-4af5-8665-ad042b545512-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "aa7ca741-597a-4af5-8665-ad042b545512" (UID: "aa7ca741-597a-4af5-8665-ad042b545512"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 19:03:27.774664 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:27.774581    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa7ca741-597a-4af5-8665-ad042b545512-dshm\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:03:27.774664 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:27.774623    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa7ca741-597a-4af5-8665-ad042b545512-tls-certs\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:03:27.774664 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:27.774634    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa7ca741-597a-4af5-8665-ad042b545512-model-cache\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:03:27.774664 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:27.774641    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa7ca741-597a-4af5-8665-ad042b545512-home\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:03:27.774664 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:27.774650    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k56tj\" (UniqueName: \"kubernetes.io/projected/aa7ca741-597a-4af5-8665-ad042b545512-kube-api-access-k56tj\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:03:27.810889 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:27.810864    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8092ee69-xbdnd"]
Sep 11 19:03:27.813217 ip-10-0-134-189 kubenswrapper[2578]: W0911 19:03:27.813190    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53ccfe18_2ffd_4462_a08c_814acee1d87b.slice/crio-0380206b6f75412140a98213aa8c519e986bb91320c61405feaac5d4edb7e87e WatchSource:0}: Error finding container 0380206b6f75412140a98213aa8c519e986bb91320c61405feaac5d4edb7e87e: Status 404 returned error can't find the container with id 0380206b6f75412140a98213aa8c519e986bb91320c61405feaac5d4edb7e87e
Sep 11 19:03:28.277459 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:28.277421    2578 generic.go:356] "Generic (PLEG): container finished" podID="aa7ca741-597a-4af5-8665-ad042b545512" containerID="f2377f837c0bf154b0a166f273cdf4d5233d456a321ec03996f5b0d3a7c1462b" exitCode=0
Sep 11 19:03:28.277843 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:28.277469    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-89e163/autores-model-89e163-kserve-696c5f94f9-m8lk7"
Sep 11 19:03:28.277843 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:28.277509    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-89e163/autores-model-89e163-kserve-696c5f94f9-m8lk7" event={"ID":"aa7ca741-597a-4af5-8665-ad042b545512","Type":"ContainerDied","Data":"f2377f837c0bf154b0a166f273cdf4d5233d456a321ec03996f5b0d3a7c1462b"}
Sep 11 19:03:28.277843 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:28.277555    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-89e163/autores-model-89e163-kserve-696c5f94f9-m8lk7" event={"ID":"aa7ca741-597a-4af5-8665-ad042b545512","Type":"ContainerDied","Data":"f1589fc0bdcb719a04acbcd6047f0be19b704bc1c2444bab717d0c7c59dcb826"}
Sep 11 19:03:28.277843 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:28.277573    2578 scope.go:117] "RemoveContainer" containerID="f2377f837c0bf154b0a166f273cdf4d5233d456a321ec03996f5b0d3a7c1462b"
Sep 11 19:03:28.279377 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:28.279312    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8092ee69-xbdnd" event={"ID":"53ccfe18-2ffd-4462-a08c-814acee1d87b","Type":"ContainerStarted","Data":"de0b866fe122d80118e9994082cdb1fc5c52a6ae94fdba83918053e47ca8d98e"}
Sep 11 19:03:28.279535 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:28.279521    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8092ee69-xbdnd" event={"ID":"53ccfe18-2ffd-4462-a08c-814acee1d87b","Type":"ContainerStarted","Data":"0380206b6f75412140a98213aa8c519e986bb91320c61405feaac5d4edb7e87e"}
Sep 11 19:03:28.286336 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:28.286319    2578 scope.go:117] "RemoveContainer" containerID="f2377f837c0bf154b0a166f273cdf4d5233d456a321ec03996f5b0d3a7c1462b"
Sep 11 19:03:28.286579 ip-10-0-134-189 kubenswrapper[2578]: E0911 19:03:28.286557    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2377f837c0bf154b0a166f273cdf4d5233d456a321ec03996f5b0d3a7c1462b\": container with ID starting with f2377f837c0bf154b0a166f273cdf4d5233d456a321ec03996f5b0d3a7c1462b not found: ID does not exist" containerID="f2377f837c0bf154b0a166f273cdf4d5233d456a321ec03996f5b0d3a7c1462b"
Sep 11 19:03:28.286657 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:28.286587    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2377f837c0bf154b0a166f273cdf4d5233d456a321ec03996f5b0d3a7c1462b"} err="failed to get container status \"f2377f837c0bf154b0a166f273cdf4d5233d456a321ec03996f5b0d3a7c1462b\": rpc error: code = NotFound desc = could not find container \"f2377f837c0bf154b0a166f273cdf4d5233d456a321ec03996f5b0d3a7c1462b\": container with ID starting with f2377f837c0bf154b0a166f273cdf4d5233d456a321ec03996f5b0d3a7c1462b not found: ID does not exist"
Sep 11 19:03:28.296238 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:28.296198    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8092ee69-xbdnd" podStartSLOduration=1.296185381 podStartE2EDuration="1.296185381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:03:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:03:28.294150745 +0000 UTC m=+884.371741300" watchObservedRunningTime="2026-09-11 19:03:28.296185381 +0000 UTC m=+884.373775935"
Sep 11 19:03:28.305855 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:28.305835    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-89e163/autores-model-89e163-kserve-696c5f94f9-m8lk7"]
Sep 11 19:03:28.309766 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:28.309744    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-89e163/autores-model-89e163-kserve-696c5f94f9-m8lk7"]
Sep 11 19:03:28.397832 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:28.397798    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa7ca741-597a-4af5-8665-ad042b545512" path="/var/lib/kubelet/pods/aa7ca741-597a-4af5-8665-ad042b545512/volumes"
Sep 11 19:03:29.285585 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:29.285545    2578 generic.go:356] "Generic (PLEG): container finished" podID="53ccfe18-2ffd-4462-a08c-814acee1d87b" containerID="de0b866fe122d80118e9994082cdb1fc5c52a6ae94fdba83918053e47ca8d98e" exitCode=0
Sep 11 19:03:29.286063 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:29.285644    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8092ee69-xbdnd" event={"ID":"53ccfe18-2ffd-4462-a08c-814acee1d87b","Type":"ContainerDied","Data":"de0b866fe122d80118e9994082cdb1fc5c52a6ae94fdba83918053e47ca8d98e"}
Sep 11 19:03:30.420441 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:30.420415    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8092ee69-xbdnd"
Sep 11 19:03:30.503946 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:30.503917    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53ccfe18-2ffd-4462-a08c-814acee1d87b-maas-api-service-ca\") pod \"53ccfe18-2ffd-4462-a08c-814acee1d87b\" (UID: \"53ccfe18-2ffd-4462-a08c-814acee1d87b\") "
Sep 11 19:03:30.504214 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:30.504191    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53ccfe18-2ffd-4462-a08c-814acee1d87b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "53ccfe18-2ffd-4462-a08c-814acee1d87b" (UID: "53ccfe18-2ffd-4462-a08c-814acee1d87b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 19:03:30.605054 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:30.604971    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53ccfe18-2ffd-4462-a08c-814acee1d87b-maas-api-service-ca\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:03:31.297927 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:31.297899    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8092ee69-xbdnd"
Sep 11 19:03:31.298105 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:31.297944    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8092ee69-xbdnd" event={"ID":"53ccfe18-2ffd-4462-a08c-814acee1d87b","Type":"ContainerDied","Data":"0380206b6f75412140a98213aa8c519e986bb91320c61405feaac5d4edb7e87e"}
Sep 11 19:03:31.298105 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:31.297997    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0380206b6f75412140a98213aa8c519e986bb91320c61405feaac5d4edb7e87e"
Sep 11 19:03:34.115726 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:34.115685    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-c1aebc/premium-w4-c1aebc-kserve-5455b5b4c-5jvlw"]
Sep 11 19:03:34.116206 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:34.116185    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53ccfe18-2ffd-4462-a08c-814acee1d87b" containerName="revoke-keys"
Sep 11 19:03:34.116288 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:34.116210    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="53ccfe18-2ffd-4462-a08c-814acee1d87b" containerName="revoke-keys"
Sep 11 19:03:34.116288 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:34.116234    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa7ca741-597a-4af5-8665-ad042b545512" containerName="main"
Sep 11 19:03:34.116288 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:34.116245    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa7ca741-597a-4af5-8665-ad042b545512" containerName="main"
Sep 11 19:03:34.116432 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:34.116349    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa7ca741-597a-4af5-8665-ad042b545512" containerName="main"
Sep 11 19:03:34.116432 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:34.116363    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="53ccfe18-2ffd-4462-a08c-814acee1d87b" containerName="revoke-keys"
Sep 11 19:03:34.119636 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:34.119587    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c1aebc/premium-w4-c1aebc-kserve-5455b5b4c-5jvlw"
Sep 11 19:03:34.121694 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:34.121671    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-c1aebc\"/\"premium-w4-c1aebc-kserve-self-signed-certs\""
Sep 11 19:03:34.127123 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:34.127102    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c1aebc/premium-w4-c1aebc-kserve-5455b5b4c-5jvlw"]
Sep 11 19:03:34.234274 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:34.234233    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa97d84e-4f85-429c-9368-635a74daa253-home\") pod \"premium-w4-c1aebc-kserve-5455b5b4c-5jvlw\" (UID: \"fa97d84e-4f85-429c-9368-635a74daa253\") " pod="e2e-models-e2e-worker-w4-c1aebc/premium-w4-c1aebc-kserve-5455b5b4c-5jvlw"
Sep 11 19:03:34.234274 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:34.234274    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa97d84e-4f85-429c-9368-635a74daa253-tls-certs\") pod \"premium-w4-c1aebc-kserve-5455b5b4c-5jvlw\" (UID: \"fa97d84e-4f85-429c-9368-635a74daa253\") " pod="e2e-models-e2e-worker-w4-c1aebc/premium-w4-c1aebc-kserve-5455b5b4c-5jvlw"
Sep 11 19:03:34.234508 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:34.234316    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa97d84e-4f85-429c-9368-635a74daa253-model-cache\") pod \"premium-w4-c1aebc-kserve-5455b5b4c-5jvlw\" (UID: \"fa97d84e-4f85-429c-9368-635a74daa253\") " pod="e2e-models-e2e-worker-w4-c1aebc/premium-w4-c1aebc-kserve-5455b5b4c-5jvlw"
Sep 11 19:03:34.234508 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:34.234351    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlfwn\" (UniqueName: \"kubernetes.io/projected/fa97d84e-4f85-429c-9368-635a74daa253-kube-api-access-wlfwn\") pod \"premium-w4-c1aebc-kserve-5455b5b4c-5jvlw\" (UID: \"fa97d84e-4f85-429c-9368-635a74daa253\") " pod="e2e-models-e2e-worker-w4-c1aebc/premium-w4-c1aebc-kserve-5455b5b4c-5jvlw"
Sep 11 19:03:34.234508 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:34.234403    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa97d84e-4f85-429c-9368-635a74daa253-tmp-dir\") pod \"premium-w4-c1aebc-kserve-5455b5b4c-5jvlw\" (UID: \"fa97d84e-4f85-429c-9368-635a74daa253\") " pod="e2e-models-e2e-worker-w4-c1aebc/premium-w4-c1aebc-kserve-5455b5b4c-5jvlw"
Sep 11 19:03:34.234508 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:34.234424    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa97d84e-4f85-429c-9368-635a74daa253-dshm\") pod \"premium-w4-c1aebc-kserve-5455b5b4c-5jvlw\" (UID: \"fa97d84e-4f85-429c-9368-635a74daa253\") " pod="e2e-models-e2e-worker-w4-c1aebc/premium-w4-c1aebc-kserve-5455b5b4c-5jvlw"
Sep 11 19:03:34.335898 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:34.335856    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa97d84e-4f85-429c-9368-635a74daa253-tmp-dir\") pod \"premium-w4-c1aebc-kserve-5455b5b4c-5jvlw\" (UID: \"fa97d84e-4f85-429c-9368-635a74daa253\") " pod="e2e-models-e2e-worker-w4-c1aebc/premium-w4-c1aebc-kserve-5455b5b4c-5jvlw"
Sep 11 19:03:34.335898 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:34.335902    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa97d84e-4f85-429c-9368-635a74daa253-dshm\") pod \"premium-w4-c1aebc-kserve-5455b5b4c-5jvlw\" (UID: \"fa97d84e-4f85-429c-9368-635a74daa253\") " pod="e2e-models-e2e-worker-w4-c1aebc/premium-w4-c1aebc-kserve-5455b5b4c-5jvlw"
Sep 11 19:03:34.336172 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:34.335937    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa97d84e-4f85-429c-9368-635a74daa253-home\") pod \"premium-w4-c1aebc-kserve-5455b5b4c-5jvlw\" (UID: \"fa97d84e-4f85-429c-9368-635a74daa253\") " pod="e2e-models-e2e-worker-w4-c1aebc/premium-w4-c1aebc-kserve-5455b5b4c-5jvlw"
Sep 11 19:03:34.336172 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:34.335955    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa97d84e-4f85-429c-9368-635a74daa253-tls-certs\") pod \"premium-w4-c1aebc-kserve-5455b5b4c-5jvlw\" (UID: \"fa97d84e-4f85-429c-9368-635a74daa253\") " pod="e2e-models-e2e-worker-w4-c1aebc/premium-w4-c1aebc-kserve-5455b5b4c-5jvlw"
Sep 11 19:03:34.336172 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:34.335993    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa97d84e-4f85-429c-9368-635a74daa253-model-cache\") pod \"premium-w4-c1aebc-kserve-5455b5b4c-5jvlw\" (UID: \"fa97d84e-4f85-429c-9368-635a74daa253\") " pod="e2e-models-e2e-worker-w4-c1aebc/premium-w4-c1aebc-kserve-5455b5b4c-5jvlw"
Sep 11 19:03:34.336172 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:34.336013    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlfwn\" (UniqueName: \"kubernetes.io/projected/fa97d84e-4f85-429c-9368-635a74daa253-kube-api-access-wlfwn\") pod \"premium-w4-c1aebc-kserve-5455b5b4c-5jvlw\" (UID: \"fa97d84e-4f85-429c-9368-635a74daa253\") " pod="e2e-models-e2e-worker-w4-c1aebc/premium-w4-c1aebc-kserve-5455b5b4c-5jvlw"
Sep 11 19:03:34.336388 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:34.336321    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa97d84e-4f85-429c-9368-635a74daa253-tmp-dir\") pod \"premium-w4-c1aebc-kserve-5455b5b4c-5jvlw\" (UID: \"fa97d84e-4f85-429c-9368-635a74daa253\") " pod="e2e-models-e2e-worker-w4-c1aebc/premium-w4-c1aebc-kserve-5455b5b4c-5jvlw"
Sep 11 19:03:34.336435 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:34.336382    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa97d84e-4f85-429c-9368-635a74daa253-model-cache\") pod \"premium-w4-c1aebc-kserve-5455b5b4c-5jvlw\" (UID: \"fa97d84e-4f85-429c-9368-635a74daa253\") " pod="e2e-models-e2e-worker-w4-c1aebc/premium-w4-c1aebc-kserve-5455b5b4c-5jvlw"
Sep 11 19:03:34.336435 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:34.336406    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa97d84e-4f85-429c-9368-635a74daa253-home\") pod \"premium-w4-c1aebc-kserve-5455b5b4c-5jvlw\" (UID: \"fa97d84e-4f85-429c-9368-635a74daa253\") " pod="e2e-models-e2e-worker-w4-c1aebc/premium-w4-c1aebc-kserve-5455b5b4c-5jvlw"
Sep 11 19:03:34.338442 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:34.338421    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa97d84e-4f85-429c-9368-635a74daa253-dshm\") pod \"premium-w4-c1aebc-kserve-5455b5b4c-5jvlw\" (UID: \"fa97d84e-4f85-429c-9368-635a74daa253\") " pod="e2e-models-e2e-worker-w4-c1aebc/premium-w4-c1aebc-kserve-5455b5b4c-5jvlw"
Sep 11 19:03:34.338638 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:34.338623    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa97d84e-4f85-429c-9368-635a74daa253-tls-certs\") pod \"premium-w4-c1aebc-kserve-5455b5b4c-5jvlw\" (UID: \"fa97d84e-4f85-429c-9368-635a74daa253\") " pod="e2e-models-e2e-worker-w4-c1aebc/premium-w4-c1aebc-kserve-5455b5b4c-5jvlw"
Sep 11 19:03:34.343570 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:34.343545    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlfwn\" (UniqueName: \"kubernetes.io/projected/fa97d84e-4f85-429c-9368-635a74daa253-kube-api-access-wlfwn\") pod \"premium-w4-c1aebc-kserve-5455b5b4c-5jvlw\" (UID: \"fa97d84e-4f85-429c-9368-635a74daa253\") " pod="e2e-models-e2e-worker-w4-c1aebc/premium-w4-c1aebc-kserve-5455b5b4c-5jvlw"
Sep 11 19:03:34.430270 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:34.430183    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c1aebc/premium-w4-c1aebc-kserve-5455b5b4c-5jvlw"
Sep 11 19:03:34.706027 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:34.705995    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c1aebc/premium-w4-c1aebc-kserve-5455b5b4c-5jvlw"]
Sep 11 19:03:34.708966 ip-10-0-134-189 kubenswrapper[2578]: W0911 19:03:34.708941    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfa97d84e_4f85_429c_9368_635a74daa253.slice/crio-ad35bd19801b2113186fcf64db93d21a67f7f37d896e42cf2e54ae190511fdcd WatchSource:0}: Error finding container ad35bd19801b2113186fcf64db93d21a67f7f37d896e42cf2e54ae190511fdcd: Status 404 returned error can't find the container with id ad35bd19801b2113186fcf64db93d21a67f7f37d896e42cf2e54ae190511fdcd
Sep 11 19:03:35.320394 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:35.320360    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c1aebc/premium-w4-c1aebc-kserve-5455b5b4c-5jvlw" event={"ID":"fa97d84e-4f85-429c-9368-635a74daa253","Type":"ContainerStarted","Data":"581b4d20d24c8272410ac01651828beb1a02f37f6fe35d9050655cf1342267ce"}
Sep 11 19:03:35.320814 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:35.320406    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c1aebc/premium-w4-c1aebc-kserve-5455b5b4c-5jvlw" event={"ID":"fa97d84e-4f85-429c-9368-635a74daa253","Type":"ContainerStarted","Data":"ad35bd19801b2113186fcf64db93d21a67f7f37d896e42cf2e54ae190511fdcd"}
Sep 11 19:03:35.335438 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:35.335388    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-c1aebc/premium-w4-c1aebc-kserve-5455b5b4c-5jvlw" podStartSLOduration=1.335368817 podStartE2EDuration="1.335368817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:03:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:03:35.335223173 +0000 UTC m=+891.412813751" watchObservedRunningTime="2026-09-11 19:03:35.335368817 +0000 UTC m=+891.412959377"
Sep 11 19:03:40.163714 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:40.163678    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8092ee69-xbdnd"]
Sep 11 19:03:40.169021 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:40.168999    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8092ee69-xbdnd"]
Sep 11 19:03:40.396976 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:40.396942    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53ccfe18-2ffd-4462-a08c-814acee1d87b" path="/var/lib/kubelet/pods/53ccfe18-2ffd-4462-a08c-814acee1d87b/volumes"
Sep 11 19:03:44.328493 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:44.328467    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-hsbsf_d0880abe-ec36-48e4-a511-ec663873fffa/console-operator/1.log"
Sep 11 19:03:44.330126 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:44.330101    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-hsbsf_d0880abe-ec36-48e4-a511-ec663873fffa/console-operator/1.log"
Sep 11 19:03:44.431089 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:44.431061    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-c1aebc/premium-w4-c1aebc-kserve-5455b5b4c-5jvlw"
Sep 11 19:03:44.431229 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:44.431106    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c1aebc/premium-w4-c1aebc-kserve-5455b5b4c-5jvlw"
Sep 11 19:03:44.443938 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:44.443912    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-c1aebc/premium-w4-c1aebc-kserve-5455b5b4c-5jvlw"
Sep 11 19:03:45.380303 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:45.380275    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c1aebc/premium-w4-c1aebc-kserve-5455b5b4c-5jvlw"
Sep 11 19:03:50.346284 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:50.346254    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-89e163-openshift-default-7774f9d49d-ddkg6"]
Sep 11 19:03:50.346728 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:50.346484    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-89e163-openshift-default-7774f9d49d-ddkg6" podUID="0f145e25-8489-4f68-b919-c981f05e730b" containerName="istio-proxy" containerID="cri-o://5698c9420e53c244bcdf4d0268198b3033760d9196eba5d8dede32307b7de22c" gracePeriod=30
Sep 11 19:03:52.177199 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:52.177151    2578 patch_prober.go:28] interesting pod/e2e-autores-89e163-openshift-default-7774f9d49d-ddkg6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 11 19:03:52.177579 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:52.177219    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-89e163-openshift-default-7774f9d49d-ddkg6" podUID="0f145e25-8489-4f68-b919-c981f05e730b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 19:03:55.238650 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:55.238617    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-62499f/premium-w0-62499f-kserve-5f94979576-n2zc8"]
Sep 11 19:03:55.243239 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:55.243222    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-62499f/premium-w0-62499f-kserve-5f94979576-n2zc8"
Sep 11 19:03:55.245267 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:55.245249    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-62499f\"/\"premium-w0-62499f-kserve-self-signed-certs\""
Sep 11 19:03:55.250504 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:55.250482    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-62499f/premium-w0-62499f-kserve-5f94979576-n2zc8"]
Sep 11 19:03:55.346294 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:55.346263    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8034fa2-7642-4160-894e-c7ebbe3f7937-home\") pod \"premium-w0-62499f-kserve-5f94979576-n2zc8\" (UID: \"e8034fa2-7642-4160-894e-c7ebbe3f7937\") " pod="e2e-models-e2e-worker-w0-62499f/premium-w0-62499f-kserve-5f94979576-n2zc8"
Sep 11 19:03:55.346447 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:55.346328    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8034fa2-7642-4160-894e-c7ebbe3f7937-tls-certs\") pod \"premium-w0-62499f-kserve-5f94979576-n2zc8\" (UID: \"e8034fa2-7642-4160-894e-c7ebbe3f7937\") " pod="e2e-models-e2e-worker-w0-62499f/premium-w0-62499f-kserve-5f94979576-n2zc8"
Sep 11 19:03:55.346447 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:55.346371    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8034fa2-7642-4160-894e-c7ebbe3f7937-tmp-dir\") pod \"premium-w0-62499f-kserve-5f94979576-n2zc8\" (UID: \"e8034fa2-7642-4160-894e-c7ebbe3f7937\") " pod="e2e-models-e2e-worker-w0-62499f/premium-w0-62499f-kserve-5f94979576-n2zc8"
Sep 11 19:03:55.346447 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:55.346395    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftxpw\" (UniqueName: \"kubernetes.io/projected/e8034fa2-7642-4160-894e-c7ebbe3f7937-kube-api-access-ftxpw\") pod \"premium-w0-62499f-kserve-5f94979576-n2zc8\" (UID: \"e8034fa2-7642-4160-894e-c7ebbe3f7937\") " pod="e2e-models-e2e-worker-w0-62499f/premium-w0-62499f-kserve-5f94979576-n2zc8"
Sep 11 19:03:55.346548 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:55.346442    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8034fa2-7642-4160-894e-c7ebbe3f7937-dshm\") pod \"premium-w0-62499f-kserve-5f94979576-n2zc8\" (UID: \"e8034fa2-7642-4160-894e-c7ebbe3f7937\") " pod="e2e-models-e2e-worker-w0-62499f/premium-w0-62499f-kserve-5f94979576-n2zc8"
Sep 11 19:03:55.346548 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:55.346480    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8034fa2-7642-4160-894e-c7ebbe3f7937-model-cache\") pod \"premium-w0-62499f-kserve-5f94979576-n2zc8\" (UID: \"e8034fa2-7642-4160-894e-c7ebbe3f7937\") " pod="e2e-models-e2e-worker-w0-62499f/premium-w0-62499f-kserve-5f94979576-n2zc8"
Sep 11 19:03:55.447778 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:55.447748    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8034fa2-7642-4160-894e-c7ebbe3f7937-tls-certs\") pod \"premium-w0-62499f-kserve-5f94979576-n2zc8\" (UID: \"e8034fa2-7642-4160-894e-c7ebbe3f7937\") " pod="e2e-models-e2e-worker-w0-62499f/premium-w0-62499f-kserve-5f94979576-n2zc8"
Sep 11 19:03:55.447948 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:55.447916    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8034fa2-7642-4160-894e-c7ebbe3f7937-tmp-dir\") pod \"premium-w0-62499f-kserve-5f94979576-n2zc8\" (UID: \"e8034fa2-7642-4160-894e-c7ebbe3f7937\") " pod="e2e-models-e2e-worker-w0-62499f/premium-w0-62499f-kserve-5f94979576-n2zc8"
Sep 11 19:03:55.448018 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:55.447972    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftxpw\" (UniqueName: \"kubernetes.io/projected/e8034fa2-7642-4160-894e-c7ebbe3f7937-kube-api-access-ftxpw\") pod \"premium-w0-62499f-kserve-5f94979576-n2zc8\" (UID: \"e8034fa2-7642-4160-894e-c7ebbe3f7937\") " pod="e2e-models-e2e-worker-w0-62499f/premium-w0-62499f-kserve-5f94979576-n2zc8"
Sep 11 19:03:55.448018 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:55.447999    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8034fa2-7642-4160-894e-c7ebbe3f7937-dshm\") pod \"premium-w0-62499f-kserve-5f94979576-n2zc8\" (UID: \"e8034fa2-7642-4160-894e-c7ebbe3f7937\") " pod="e2e-models-e2e-worker-w0-62499f/premium-w0-62499f-kserve-5f94979576-n2zc8"
Sep 11 19:03:55.448125 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:55.448036    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8034fa2-7642-4160-894e-c7ebbe3f7937-model-cache\") pod \"premium-w0-62499f-kserve-5f94979576-n2zc8\" (UID: \"e8034fa2-7642-4160-894e-c7ebbe3f7937\") " pod="e2e-models-e2e-worker-w0-62499f/premium-w0-62499f-kserve-5f94979576-n2zc8"
Sep 11 19:03:55.448125 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:55.448076    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8034fa2-7642-4160-894e-c7ebbe3f7937-home\") pod \"premium-w0-62499f-kserve-5f94979576-n2zc8\" (UID: \"e8034fa2-7642-4160-894e-c7ebbe3f7937\") " pod="e2e-models-e2e-worker-w0-62499f/premium-w0-62499f-kserve-5f94979576-n2zc8"
Sep 11 19:03:55.448358 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:55.448336    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8034fa2-7642-4160-894e-c7ebbe3f7937-tmp-dir\") pod \"premium-w0-62499f-kserve-5f94979576-n2zc8\" (UID: \"e8034fa2-7642-4160-894e-c7ebbe3f7937\") " pod="e2e-models-e2e-worker-w0-62499f/premium-w0-62499f-kserve-5f94979576-n2zc8"
Sep 11 19:03:55.448479 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:55.448456    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8034fa2-7642-4160-894e-c7ebbe3f7937-model-cache\") pod \"premium-w0-62499f-kserve-5f94979576-n2zc8\" (UID: \"e8034fa2-7642-4160-894e-c7ebbe3f7937\") " pod="e2e-models-e2e-worker-w0-62499f/premium-w0-62499f-kserve-5f94979576-n2zc8"
Sep 11 19:03:55.448537 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:55.448489    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8034fa2-7642-4160-894e-c7ebbe3f7937-home\") pod \"premium-w0-62499f-kserve-5f94979576-n2zc8\" (UID: \"e8034fa2-7642-4160-894e-c7ebbe3f7937\") " pod="e2e-models-e2e-worker-w0-62499f/premium-w0-62499f-kserve-5f94979576-n2zc8"
Sep 11 19:03:55.450360 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:55.450337    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8034fa2-7642-4160-894e-c7ebbe3f7937-dshm\") pod \"premium-w0-62499f-kserve-5f94979576-n2zc8\" (UID: \"e8034fa2-7642-4160-894e-c7ebbe3f7937\") " pod="e2e-models-e2e-worker-w0-62499f/premium-w0-62499f-kserve-5f94979576-n2zc8"
Sep 11 19:03:55.450685 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:55.450661    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8034fa2-7642-4160-894e-c7ebbe3f7937-tls-certs\") pod \"premium-w0-62499f-kserve-5f94979576-n2zc8\" (UID: \"e8034fa2-7642-4160-894e-c7ebbe3f7937\") " pod="e2e-models-e2e-worker-w0-62499f/premium-w0-62499f-kserve-5f94979576-n2zc8"
Sep 11 19:03:55.456801 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:55.456781    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftxpw\" (UniqueName: \"kubernetes.io/projected/e8034fa2-7642-4160-894e-c7ebbe3f7937-kube-api-access-ftxpw\") pod \"premium-w0-62499f-kserve-5f94979576-n2zc8\" (UID: \"e8034fa2-7642-4160-894e-c7ebbe3f7937\") " pod="e2e-models-e2e-worker-w0-62499f/premium-w0-62499f-kserve-5f94979576-n2zc8"
Sep 11 19:03:55.553351 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:55.553318    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-62499f/premium-w0-62499f-kserve-5f94979576-n2zc8"
Sep 11 19:03:55.589076 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:55.589051    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-89e163-openshift-default-7774f9d49d-ddkg6"
Sep 11 19:03:55.649922 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:55.649895    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f145e25-8489-4f68-b919-c981f05e730b-istio-podinfo\") pod \"0f145e25-8489-4f68-b919-c981f05e730b\" (UID: \"0f145e25-8489-4f68-b919-c981f05e730b\") "
Sep 11 19:03:55.650075 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:55.649993    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f145e25-8489-4f68-b919-c981f05e730b-istiod-ca-cert\") pod \"0f145e25-8489-4f68-b919-c981f05e730b\" (UID: \"0f145e25-8489-4f68-b919-c981f05e730b\") "
Sep 11 19:03:55.650075 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:55.650023    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f145e25-8489-4f68-b919-c981f05e730b-istio-envoy\") pod \"0f145e25-8489-4f68-b919-c981f05e730b\" (UID: \"0f145e25-8489-4f68-b919-c981f05e730b\") "
Sep 11 19:03:55.650075 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:55.650046    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f145e25-8489-4f68-b919-c981f05e730b-credential-socket\") pod \"0f145e25-8489-4f68-b919-c981f05e730b\" (UID: \"0f145e25-8489-4f68-b919-c981f05e730b\") "
Sep 11 19:03:55.650208 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:55.650073    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f145e25-8489-4f68-b919-c981f05e730b-workload-socket\") pod \"0f145e25-8489-4f68-b919-c981f05e730b\" (UID: \"0f145e25-8489-4f68-b919-c981f05e730b\") "
Sep 11 19:03:55.650208 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:55.650116    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f145e25-8489-4f68-b919-c981f05e730b-istio-token\") pod \"0f145e25-8489-4f68-b919-c981f05e730b\" (UID: \"0f145e25-8489-4f68-b919-c981f05e730b\") "
Sep 11 19:03:55.650208 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:55.650149    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wkzk7\" (UniqueName: \"kubernetes.io/projected/0f145e25-8489-4f68-b919-c981f05e730b-kube-api-access-wkzk7\") pod \"0f145e25-8489-4f68-b919-c981f05e730b\" (UID: \"0f145e25-8489-4f68-b919-c981f05e730b\") "
Sep 11 19:03:55.650208 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:55.650178    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f145e25-8489-4f68-b919-c981f05e730b-workload-certs\") pod \"0f145e25-8489-4f68-b919-c981f05e730b\" (UID: \"0f145e25-8489-4f68-b919-c981f05e730b\") "
Sep 11 19:03:55.650208 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:55.650200    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f145e25-8489-4f68-b919-c981f05e730b-istio-data\") pod \"0f145e25-8489-4f68-b919-c981f05e730b\" (UID: \"0f145e25-8489-4f68-b919-c981f05e730b\") "
Sep 11 19:03:55.650450 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:55.650391    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f145e25-8489-4f68-b919-c981f05e730b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0f145e25-8489-4f68-b919-c981f05e730b" (UID: "0f145e25-8489-4f68-b919-c981f05e730b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:03:55.650450 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:55.650406    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f145e25-8489-4f68-b919-c981f05e730b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0f145e25-8489-4f68-b919-c981f05e730b" (UID: "0f145e25-8489-4f68-b919-c981f05e730b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:03:55.650450 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:55.650425    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f145e25-8489-4f68-b919-c981f05e730b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0f145e25-8489-4f68-b919-c981f05e730b" (UID: "0f145e25-8489-4f68-b919-c981f05e730b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 19:03:55.650925 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:55.650557    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f145e25-8489-4f68-b919-c981f05e730b-istiod-ca-cert\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:03:55.650925 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:55.650576    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f145e25-8489-4f68-b919-c981f05e730b-credential-socket\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:03:55.650925 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:55.650704    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f145e25-8489-4f68-b919-c981f05e730b-workload-socket\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:03:55.650925 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:55.650789    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f145e25-8489-4f68-b919-c981f05e730b-istio-data" (OuterVolumeSpecName: "istio-data") pod "0f145e25-8489-4f68-b919-c981f05e730b" (UID: "0f145e25-8489-4f68-b919-c981f05e730b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:03:55.650925 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:55.650829    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f145e25-8489-4f68-b919-c981f05e730b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0f145e25-8489-4f68-b919-c981f05e730b" (UID: "0f145e25-8489-4f68-b919-c981f05e730b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:03:55.652844 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:55.652823    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f145e25-8489-4f68-b919-c981f05e730b-kube-api-access-wkzk7" (OuterVolumeSpecName: "kube-api-access-wkzk7") pod "0f145e25-8489-4f68-b919-c981f05e730b" (UID: "0f145e25-8489-4f68-b919-c981f05e730b"). InnerVolumeSpecName "kube-api-access-wkzk7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:03:55.653030 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:55.653009    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f145e25-8489-4f68-b919-c981f05e730b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0f145e25-8489-4f68-b919-c981f05e730b" (UID: "0f145e25-8489-4f68-b919-c981f05e730b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:03:55.653079 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:55.653009    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0f145e25-8489-4f68-b919-c981f05e730b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0f145e25-8489-4f68-b919-c981f05e730b" (UID: "0f145e25-8489-4f68-b919-c981f05e730b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 19:03:55.653079 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:55.653038    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f145e25-8489-4f68-b919-c981f05e730b-istio-token" (OuterVolumeSpecName: "istio-token") pod "0f145e25-8489-4f68-b919-c981f05e730b" (UID: "0f145e25-8489-4f68-b919-c981f05e730b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:03:55.751880 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:55.751852    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f145e25-8489-4f68-b919-c981f05e730b-istio-token\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:03:55.751880 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:55.751875    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkzk7\" (UniqueName: \"kubernetes.io/projected/0f145e25-8489-4f68-b919-c981f05e730b-kube-api-access-wkzk7\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:03:55.751880 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:55.751887    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f145e25-8489-4f68-b919-c981f05e730b-workload-certs\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:03:55.752102 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:55.751897    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f145e25-8489-4f68-b919-c981f05e730b-istio-data\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:03:55.752102 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:55.751904    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f145e25-8489-4f68-b919-c981f05e730b-istio-podinfo\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:03:55.752102 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:55.751911    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f145e25-8489-4f68-b919-c981f05e730b-istio-envoy\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:03:55.827472 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:55.827448    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-62499f/premium-w0-62499f-kserve-5f94979576-n2zc8"]
Sep 11 19:03:55.829647 ip-10-0-134-189 kubenswrapper[2578]: W0911 19:03:55.829619    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode8034fa2_7642_4160_894e_c7ebbe3f7937.slice/crio-f7e48a7fd7e48fd28f8233a154b48c9c6f02539bba96e0947d516b08fb38e025 WatchSource:0}: Error finding container f7e48a7fd7e48fd28f8233a154b48c9c6f02539bba96e0947d516b08fb38e025: Status 404 returned error can't find the container with id f7e48a7fd7e48fd28f8233a154b48c9c6f02539bba96e0947d516b08fb38e025
Sep 11 19:03:56.427770 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:56.427740    2578 generic.go:356] "Generic (PLEG): container finished" podID="0f145e25-8489-4f68-b919-c981f05e730b" containerID="5698c9420e53c244bcdf4d0268198b3033760d9196eba5d8dede32307b7de22c" exitCode=0
Sep 11 19:03:56.428114 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:56.427800    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-89e163-openshift-default-7774f9d49d-ddkg6"
Sep 11 19:03:56.428114 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:56.427793    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-89e163-openshift-default-7774f9d49d-ddkg6" event={"ID":"0f145e25-8489-4f68-b919-c981f05e730b","Type":"ContainerDied","Data":"5698c9420e53c244bcdf4d0268198b3033760d9196eba5d8dede32307b7de22c"}
Sep 11 19:03:56.428114 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:56.427854    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-89e163-openshift-default-7774f9d49d-ddkg6" event={"ID":"0f145e25-8489-4f68-b919-c981f05e730b","Type":"ContainerDied","Data":"3d72b560947250254de23d018e71ebde4d407d8fea4cabf6701869228c36ff33"}
Sep 11 19:03:56.428114 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:56.427884    2578 scope.go:117] "RemoveContainer" containerID="5698c9420e53c244bcdf4d0268198b3033760d9196eba5d8dede32307b7de22c"
Sep 11 19:03:56.429768 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:56.429726    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-62499f/premium-w0-62499f-kserve-5f94979576-n2zc8" event={"ID":"e8034fa2-7642-4160-894e-c7ebbe3f7937","Type":"ContainerStarted","Data":"2ff6d9dd3a45e134b32d1d2932f4bc2e6e6aae1a9cd0c6abb5235d67fc496cfe"}
Sep 11 19:03:56.429768 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:56.429764    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-62499f/premium-w0-62499f-kserve-5f94979576-n2zc8" event={"ID":"e8034fa2-7642-4160-894e-c7ebbe3f7937","Type":"ContainerStarted","Data":"f7e48a7fd7e48fd28f8233a154b48c9c6f02539bba96e0947d516b08fb38e025"}
Sep 11 19:03:56.436652 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:56.436637    2578 scope.go:117] "RemoveContainer" containerID="76501c1fa721cf292975c7481206b43b935df0872994538afd6a02b8a80de777"
Sep 11 19:03:56.443223 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:56.443204    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-89e163-openshift-default-7774f9d49d-ddkg6"]
Sep 11 19:03:56.445007 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:56.444987    2578 scope.go:117] "RemoveContainer" containerID="5698c9420e53c244bcdf4d0268198b3033760d9196eba5d8dede32307b7de22c"
Sep 11 19:03:56.445283 ip-10-0-134-189 kubenswrapper[2578]: E0911 19:03:56.445263    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5698c9420e53c244bcdf4d0268198b3033760d9196eba5d8dede32307b7de22c\": container with ID starting with 5698c9420e53c244bcdf4d0268198b3033760d9196eba5d8dede32307b7de22c not found: ID does not exist" containerID="5698c9420e53c244bcdf4d0268198b3033760d9196eba5d8dede32307b7de22c"
Sep 11 19:03:56.445345 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:56.445289    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5698c9420e53c244bcdf4d0268198b3033760d9196eba5d8dede32307b7de22c"} err="failed to get container status \"5698c9420e53c244bcdf4d0268198b3033760d9196eba5d8dede32307b7de22c\": rpc error: code = NotFound desc = could not find container \"5698c9420e53c244bcdf4d0268198b3033760d9196eba5d8dede32307b7de22c\": container with ID starting with 5698c9420e53c244bcdf4d0268198b3033760d9196eba5d8dede32307b7de22c not found: ID does not exist"
Sep 11 19:03:56.445345 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:56.445304    2578 scope.go:117] "RemoveContainer" containerID="76501c1fa721cf292975c7481206b43b935df0872994538afd6a02b8a80de777"
Sep 11 19:03:56.445536 ip-10-0-134-189 kubenswrapper[2578]: E0911 19:03:56.445522    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76501c1fa721cf292975c7481206b43b935df0872994538afd6a02b8a80de777\": container with ID starting with 76501c1fa721cf292975c7481206b43b935df0872994538afd6a02b8a80de777 not found: ID does not exist" containerID="76501c1fa721cf292975c7481206b43b935df0872994538afd6a02b8a80de777"
Sep 11 19:03:56.445576 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:56.445539    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76501c1fa721cf292975c7481206b43b935df0872994538afd6a02b8a80de777"} err="failed to get container status \"76501c1fa721cf292975c7481206b43b935df0872994538afd6a02b8a80de777\": rpc error: code = NotFound desc = could not find container \"76501c1fa721cf292975c7481206b43b935df0872994538afd6a02b8a80de777\": container with ID starting with 76501c1fa721cf292975c7481206b43b935df0872994538afd6a02b8a80de777 not found: ID does not exist"
Sep 11 19:03:56.448385 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:56.448368    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-89e163-openshift-default-7774f9d49d-ddkg6"]
Sep 11 19:03:56.460554 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:56.460520    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-62499f/premium-w0-62499f-kserve-5f94979576-n2zc8" podStartSLOduration=1.4604856210000001 podStartE2EDuration="1.460485621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:03:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:03:56.459917408 +0000 UTC m=+912.537507976" watchObservedRunningTime="2026-09-11 19:03:56.460485621 +0000 UTC m=+912.538076200"
Sep 11 19:03:58.396740 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:03:58.396706    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f145e25-8489-4f68-b919-c981f05e730b" path="/var/lib/kubelet/pods/0f145e25-8489-4f68-b919-c981f05e730b/volumes"
Sep 11 19:04:05.554352 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:04:05.554310    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-62499f/premium-w0-62499f-kserve-5f94979576-n2zc8"
Sep 11 19:04:05.554853 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:04:05.554371    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-62499f/premium-w0-62499f-kserve-5f94979576-n2zc8"
Sep 11 19:04:05.566859 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:04:05.566830    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-62499f/premium-w0-62499f-kserve-5f94979576-n2zc8"
Sep 11 19:04:06.487723 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:04:06.487694    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-62499f/premium-w0-62499f-kserve-5f94979576-n2zc8"
Sep 11 19:04:19.555281 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:04:19.555251    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-093f6c/test-model-093f6c-kserve-544f68f7bc-8wj5z"]
Sep 11 19:04:19.555710 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:04:19.555619    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f145e25-8489-4f68-b919-c981f05e730b" containerName="istio-proxy"
Sep 11 19:04:19.555710 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:04:19.555631    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f145e25-8489-4f68-b919-c981f05e730b" containerName="istio-proxy"
Sep 11 19:04:19.555710 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:04:19.555693    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f145e25-8489-4f68-b919-c981f05e730b" containerName="istio-proxy"
Sep 11 19:04:19.555710 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:04:19.555700    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f145e25-8489-4f68-b919-c981f05e730b" containerName="istio-proxy"
Sep 11 19:04:19.558675 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:04:19.558660    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-093f6c/test-model-093f6c-kserve-544f68f7bc-8wj5z"
Sep 11 19:04:19.560683 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:04:19.560653    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-093f6c\"/\"kube-root-ca.crt\""
Sep 11 19:04:19.561188 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:04:19.561166    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-093f6c\"/\"openshift-service-ca.crt\""
Sep 11 19:04:19.561287 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:04:19.561185    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-093f6c\"/\"default-dockercfg-8kds8\""
Sep 11 19:04:19.561287 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:04:19.561214    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-093f6c\"/\"test-model-093f6c-kserve-self-signed-certs\""
Sep 11 19:04:19.566295 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:04:19.566273    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-093f6c/test-model-093f6c-kserve-544f68f7bc-8wj5z"]
Sep 11 19:04:19.687788 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:04:19.687753    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21fc85b7-8175-4764-85ce-e026bccccc08-tls-certs\") pod \"test-model-093f6c-kserve-544f68f7bc-8wj5z\" (UID: \"21fc85b7-8175-4764-85ce-e026bccccc08\") " pod="ai-tenant-e2e-inf-b-093f6c/test-model-093f6c-kserve-544f68f7bc-8wj5z"
Sep 11 19:04:19.687788 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:04:19.687792    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21fc85b7-8175-4764-85ce-e026bccccc08-tmp-dir\") pod \"test-model-093f6c-kserve-544f68f7bc-8wj5z\" (UID: \"21fc85b7-8175-4764-85ce-e026bccccc08\") " pod="ai-tenant-e2e-inf-b-093f6c/test-model-093f6c-kserve-544f68f7bc-8wj5z"
Sep 11 19:04:19.687989 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:04:19.687828    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21fc85b7-8175-4764-85ce-e026bccccc08-home\") pod \"test-model-093f6c-kserve-544f68f7bc-8wj5z\" (UID: \"21fc85b7-8175-4764-85ce-e026bccccc08\") " pod="ai-tenant-e2e-inf-b-093f6c/test-model-093f6c-kserve-544f68f7bc-8wj5z"
Sep 11 19:04:19.687989 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:04:19.687921    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21fc85b7-8175-4764-85ce-e026bccccc08-dshm\") pod \"test-model-093f6c-kserve-544f68f7bc-8wj5z\" (UID: \"21fc85b7-8175-4764-85ce-e026bccccc08\") " pod="ai-tenant-e2e-inf-b-093f6c/test-model-093f6c-kserve-544f68f7bc-8wj5z"
Sep 11 19:04:19.687989 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:04:19.687941    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlpg2\" (UniqueName: \"kubernetes.io/projected/21fc85b7-8175-4764-85ce-e026bccccc08-kube-api-access-qlpg2\") pod \"test-model-093f6c-kserve-544f68f7bc-8wj5z\" (UID: \"21fc85b7-8175-4764-85ce-e026bccccc08\") " pod="ai-tenant-e2e-inf-b-093f6c/test-model-093f6c-kserve-544f68f7bc-8wj5z"
Sep 11 19:04:19.687989 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:04:19.687965    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21fc85b7-8175-4764-85ce-e026bccccc08-model-cache\") pod \"test-model-093f6c-kserve-544f68f7bc-8wj5z\" (UID: \"21fc85b7-8175-4764-85ce-e026bccccc08\") " pod="ai-tenant-e2e-inf-b-093f6c/test-model-093f6c-kserve-544f68f7bc-8wj5z"
Sep 11 19:04:19.789481 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:04:19.789449    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21fc85b7-8175-4764-85ce-e026bccccc08-tls-certs\") pod \"test-model-093f6c-kserve-544f68f7bc-8wj5z\" (UID: \"21fc85b7-8175-4764-85ce-e026bccccc08\") " pod="ai-tenant-e2e-inf-b-093f6c/test-model-093f6c-kserve-544f68f7bc-8wj5z"
Sep 11 19:04:19.789481 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:04:19.789488    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21fc85b7-8175-4764-85ce-e026bccccc08-tmp-dir\") pod \"test-model-093f6c-kserve-544f68f7bc-8wj5z\" (UID: \"21fc85b7-8175-4764-85ce-e026bccccc08\") " pod="ai-tenant-e2e-inf-b-093f6c/test-model-093f6c-kserve-544f68f7bc-8wj5z"
Sep 11 19:04:19.789754 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:04:19.789515    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21fc85b7-8175-4764-85ce-e026bccccc08-home\") pod \"test-model-093f6c-kserve-544f68f7bc-8wj5z\" (UID: \"21fc85b7-8175-4764-85ce-e026bccccc08\") " pod="ai-tenant-e2e-inf-b-093f6c/test-model-093f6c-kserve-544f68f7bc-8wj5z"
Sep 11 19:04:19.789754 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:04:19.789573    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21fc85b7-8175-4764-85ce-e026bccccc08-dshm\") pod \"test-model-093f6c-kserve-544f68f7bc-8wj5z\" (UID: \"21fc85b7-8175-4764-85ce-e026bccccc08\") " pod="ai-tenant-e2e-inf-b-093f6c/test-model-093f6c-kserve-544f68f7bc-8wj5z"
Sep 11 19:04:19.789754 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:04:19.789611    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlpg2\" (UniqueName: \"kubernetes.io/projected/21fc85b7-8175-4764-85ce-e026bccccc08-kube-api-access-qlpg2\") pod \"test-model-093f6c-kserve-544f68f7bc-8wj5z\" (UID: \"21fc85b7-8175-4764-85ce-e026bccccc08\") " pod="ai-tenant-e2e-inf-b-093f6c/test-model-093f6c-kserve-544f68f7bc-8wj5z"
Sep 11 19:04:19.789754 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:04:19.789646    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21fc85b7-8175-4764-85ce-e026bccccc08-model-cache\") pod \"test-model-093f6c-kserve-544f68f7bc-8wj5z\" (UID: \"21fc85b7-8175-4764-85ce-e026bccccc08\") " pod="ai-tenant-e2e-inf-b-093f6c/test-model-093f6c-kserve-544f68f7bc-8wj5z"
Sep 11 19:04:19.789970 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:04:19.789933    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21fc85b7-8175-4764-85ce-e026bccccc08-tmp-dir\") pod \"test-model-093f6c-kserve-544f68f7bc-8wj5z\" (UID: \"21fc85b7-8175-4764-85ce-e026bccccc08\") " pod="ai-tenant-e2e-inf-b-093f6c/test-model-093f6c-kserve-544f68f7bc-8wj5z"
Sep 11 19:04:19.790017 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:04:19.789968    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21fc85b7-8175-4764-85ce-e026bccccc08-model-cache\") pod \"test-model-093f6c-kserve-544f68f7bc-8wj5z\" (UID: \"21fc85b7-8175-4764-85ce-e026bccccc08\") " pod="ai-tenant-e2e-inf-b-093f6c/test-model-093f6c-kserve-544f68f7bc-8wj5z"
Sep 11 19:04:19.790017 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:04:19.790005    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21fc85b7-8175-4764-85ce-e026bccccc08-home\") pod \"test-model-093f6c-kserve-544f68f7bc-8wj5z\" (UID: \"21fc85b7-8175-4764-85ce-e026bccccc08\") " pod="ai-tenant-e2e-inf-b-093f6c/test-model-093f6c-kserve-544f68f7bc-8wj5z"
Sep 11 19:04:19.791944 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:04:19.791924    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21fc85b7-8175-4764-85ce-e026bccccc08-dshm\") pod \"test-model-093f6c-kserve-544f68f7bc-8wj5z\" (UID: \"21fc85b7-8175-4764-85ce-e026bccccc08\") " pod="ai-tenant-e2e-inf-b-093f6c/test-model-093f6c-kserve-544f68f7bc-8wj5z"
Sep 11 19:04:19.792103 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:04:19.792086    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21fc85b7-8175-4764-85ce-e026bccccc08-tls-certs\") pod \"test-model-093f6c-kserve-544f68f7bc-8wj5z\" (UID: \"21fc85b7-8175-4764-85ce-e026bccccc08\") " pod="ai-tenant-e2e-inf-b-093f6c/test-model-093f6c-kserve-544f68f7bc-8wj5z"
Sep 11 19:04:19.796808 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:04:19.796782    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlpg2\" (UniqueName: \"kubernetes.io/projected/21fc85b7-8175-4764-85ce-e026bccccc08-kube-api-access-qlpg2\") pod \"test-model-093f6c-kserve-544f68f7bc-8wj5z\" (UID: \"21fc85b7-8175-4764-85ce-e026bccccc08\") " pod="ai-tenant-e2e-inf-b-093f6c/test-model-093f6c-kserve-544f68f7bc-8wj5z"
Sep 11 19:04:19.869154 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:04:19.869072    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-093f6c/test-model-093f6c-kserve-544f68f7bc-8wj5z"
Sep 11 19:04:20.143056 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:04:20.143025    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-093f6c/test-model-093f6c-kserve-544f68f7bc-8wj5z"]
Sep 11 19:04:20.144879 ip-10-0-134-189 kubenswrapper[2578]: W0911 19:04:20.144851    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod21fc85b7_8175_4764_85ce_e026bccccc08.slice/crio-3487067cf0c6094708e1bd4928c687a140d8e0ac6ca5d4c90912efa5756e2a0e WatchSource:0}: Error finding container 3487067cf0c6094708e1bd4928c687a140d8e0ac6ca5d4c90912efa5756e2a0e: Status 404 returned error can't find the container with id 3487067cf0c6094708e1bd4928c687a140d8e0ac6ca5d4c90912efa5756e2a0e
Sep 11 19:04:20.552755 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:04:20.552714    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-093f6c/test-model-093f6c-kserve-544f68f7bc-8wj5z" event={"ID":"21fc85b7-8175-4764-85ce-e026bccccc08","Type":"ContainerStarted","Data":"594ec5ff346ab2450d165b91832c96c537c7cf6373cbc83c849b2a2c25f59634"}
Sep 11 19:04:20.552755 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:04:20.552763    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-093f6c/test-model-093f6c-kserve-544f68f7bc-8wj5z" event={"ID":"21fc85b7-8175-4764-85ce-e026bccccc08","Type":"ContainerStarted","Data":"3487067cf0c6094708e1bd4928c687a140d8e0ac6ca5d4c90912efa5756e2a0e"}
Sep 11 19:04:20.568563 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:04:20.568500    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-093f6c/test-model-093f6c-kserve-544f68f7bc-8wj5z" podStartSLOduration=1.568479408 podStartE2EDuration="1.568479408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:04:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:04:20.56807884 +0000 UTC m=+936.645669395" watchObservedRunningTime="2026-09-11 19:04:20.568479408 +0000 UTC m=+936.646069964"
Sep 11 19:04:29.869792 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:04:29.869741    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-093f6c/test-model-093f6c-kserve-544f68f7bc-8wj5z"
Sep 11 19:04:29.869792 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:04:29.869809    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-093f6c/test-model-093f6c-kserve-544f68f7bc-8wj5z"
Sep 11 19:04:29.882368 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:04:29.882333    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-093f6c/test-model-093f6c-kserve-544f68f7bc-8wj5z"
Sep 11 19:04:30.614699 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:04:30.614668    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-093f6c/test-model-093f6c-kserve-544f68f7bc-8wj5z"
Sep 11 19:05:21.830820 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:21.830726    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-f60615/explicit-model-f60615-kserve-75797f546f-m2cxk"]
Sep 11 19:05:21.831264 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:21.831111    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f145e25-8489-4f68-b919-c981f05e730b" containerName="istio-proxy"
Sep 11 19:05:21.831264 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:21.831125    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f145e25-8489-4f68-b919-c981f05e730b" containerName="istio-proxy"
Sep 11 19:05:21.834161 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:21.834145    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-f60615/explicit-model-f60615-kserve-75797f546f-m2cxk"
Sep 11 19:05:21.836696 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:21.836674    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-f60615\"/\"kube-root-ca.crt\""
Sep 11 19:05:21.836825 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:21.836679    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-f60615\"/\"openshift-service-ca.crt\""
Sep 11 19:05:21.837382 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:21.837362    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-f60615\"/\"explicit-model-f60615-kserve-self-signed-certs\""
Sep 11 19:05:21.837470 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:21.837432    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-f60615\"/\"default-dockercfg-jmrxw\""
Sep 11 19:05:21.844089 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:21.844067    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-f60615/explicit-model-f60615-kserve-75797f546f-m2cxk"]
Sep 11 19:05:21.897304 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:21.897265    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37e265c6-5c6e-428e-ae71-516da6a503e5-tls-certs\") pod \"explicit-model-f60615-kserve-75797f546f-m2cxk\" (UID: \"37e265c6-5c6e-428e-ae71-516da6a503e5\") " pod="ai-tenant-e2e-explicit-f60615/explicit-model-f60615-kserve-75797f546f-m2cxk"
Sep 11 19:05:21.897468 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:21.897362    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37e265c6-5c6e-428e-ae71-516da6a503e5-model-cache\") pod \"explicit-model-f60615-kserve-75797f546f-m2cxk\" (UID: \"37e265c6-5c6e-428e-ae71-516da6a503e5\") " pod="ai-tenant-e2e-explicit-f60615/explicit-model-f60615-kserve-75797f546f-m2cxk"
Sep 11 19:05:21.897468 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:21.897389    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37e265c6-5c6e-428e-ae71-516da6a503e5-tmp-dir\") pod \"explicit-model-f60615-kserve-75797f546f-m2cxk\" (UID: \"37e265c6-5c6e-428e-ae71-516da6a503e5\") " pod="ai-tenant-e2e-explicit-f60615/explicit-model-f60615-kserve-75797f546f-m2cxk"
Sep 11 19:05:21.897556 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:21.897471    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xs6v4\" (UniqueName: \"kubernetes.io/projected/37e265c6-5c6e-428e-ae71-516da6a503e5-kube-api-access-xs6v4\") pod \"explicit-model-f60615-kserve-75797f546f-m2cxk\" (UID: \"37e265c6-5c6e-428e-ae71-516da6a503e5\") " pod="ai-tenant-e2e-explicit-f60615/explicit-model-f60615-kserve-75797f546f-m2cxk"
Sep 11 19:05:21.897618 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:21.897546    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37e265c6-5c6e-428e-ae71-516da6a503e5-home\") pod \"explicit-model-f60615-kserve-75797f546f-m2cxk\" (UID: \"37e265c6-5c6e-428e-ae71-516da6a503e5\") " pod="ai-tenant-e2e-explicit-f60615/explicit-model-f60615-kserve-75797f546f-m2cxk"
Sep 11 19:05:21.897618 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:21.897580    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37e265c6-5c6e-428e-ae71-516da6a503e5-dshm\") pod \"explicit-model-f60615-kserve-75797f546f-m2cxk\" (UID: \"37e265c6-5c6e-428e-ae71-516da6a503e5\") " pod="ai-tenant-e2e-explicit-f60615/explicit-model-f60615-kserve-75797f546f-m2cxk"
Sep 11 19:05:21.998905 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:21.998870    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37e265c6-5c6e-428e-ae71-516da6a503e5-tls-certs\") pod \"explicit-model-f60615-kserve-75797f546f-m2cxk\" (UID: \"37e265c6-5c6e-428e-ae71-516da6a503e5\") " pod="ai-tenant-e2e-explicit-f60615/explicit-model-f60615-kserve-75797f546f-m2cxk"
Sep 11 19:05:21.999088 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:21.998925    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37e265c6-5c6e-428e-ae71-516da6a503e5-model-cache\") pod \"explicit-model-f60615-kserve-75797f546f-m2cxk\" (UID: \"37e265c6-5c6e-428e-ae71-516da6a503e5\") " pod="ai-tenant-e2e-explicit-f60615/explicit-model-f60615-kserve-75797f546f-m2cxk"
Sep 11 19:05:21.999088 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:21.998943    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37e265c6-5c6e-428e-ae71-516da6a503e5-tmp-dir\") pod \"explicit-model-f60615-kserve-75797f546f-m2cxk\" (UID: \"37e265c6-5c6e-428e-ae71-516da6a503e5\") " pod="ai-tenant-e2e-explicit-f60615/explicit-model-f60615-kserve-75797f546f-m2cxk"
Sep 11 19:05:21.999088 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:21.998974    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs6v4\" (UniqueName: \"kubernetes.io/projected/37e265c6-5c6e-428e-ae71-516da6a503e5-kube-api-access-xs6v4\") pod \"explicit-model-f60615-kserve-75797f546f-m2cxk\" (UID: \"37e265c6-5c6e-428e-ae71-516da6a503e5\") " pod="ai-tenant-e2e-explicit-f60615/explicit-model-f60615-kserve-75797f546f-m2cxk"
Sep 11 19:05:21.999088 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:21.998997    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37e265c6-5c6e-428e-ae71-516da6a503e5-home\") pod \"explicit-model-f60615-kserve-75797f546f-m2cxk\" (UID: \"37e265c6-5c6e-428e-ae71-516da6a503e5\") " pod="ai-tenant-e2e-explicit-f60615/explicit-model-f60615-kserve-75797f546f-m2cxk"
Sep 11 19:05:21.999088 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:21.999011    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37e265c6-5c6e-428e-ae71-516da6a503e5-dshm\") pod \"explicit-model-f60615-kserve-75797f546f-m2cxk\" (UID: \"37e265c6-5c6e-428e-ae71-516da6a503e5\") " pod="ai-tenant-e2e-explicit-f60615/explicit-model-f60615-kserve-75797f546f-m2cxk"
Sep 11 19:05:21.999430 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:21.999406    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37e265c6-5c6e-428e-ae71-516da6a503e5-tmp-dir\") pod \"explicit-model-f60615-kserve-75797f546f-m2cxk\" (UID: \"37e265c6-5c6e-428e-ae71-516da6a503e5\") " pod="ai-tenant-e2e-explicit-f60615/explicit-model-f60615-kserve-75797f546f-m2cxk"
Sep 11 19:05:21.999499 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:21.999437    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37e265c6-5c6e-428e-ae71-516da6a503e5-model-cache\") pod \"explicit-model-f60615-kserve-75797f546f-m2cxk\" (UID: \"37e265c6-5c6e-428e-ae71-516da6a503e5\") " pod="ai-tenant-e2e-explicit-f60615/explicit-model-f60615-kserve-75797f546f-m2cxk"
Sep 11 19:05:21.999545 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:21.999497    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37e265c6-5c6e-428e-ae71-516da6a503e5-home\") pod \"explicit-model-f60615-kserve-75797f546f-m2cxk\" (UID: \"37e265c6-5c6e-428e-ae71-516da6a503e5\") " pod="ai-tenant-e2e-explicit-f60615/explicit-model-f60615-kserve-75797f546f-m2cxk"
Sep 11 19:05:22.001379 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:22.001352    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37e265c6-5c6e-428e-ae71-516da6a503e5-dshm\") pod \"explicit-model-f60615-kserve-75797f546f-m2cxk\" (UID: \"37e265c6-5c6e-428e-ae71-516da6a503e5\") " pod="ai-tenant-e2e-explicit-f60615/explicit-model-f60615-kserve-75797f546f-m2cxk"
Sep 11 19:05:22.001661 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:22.001639    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37e265c6-5c6e-428e-ae71-516da6a503e5-tls-certs\") pod \"explicit-model-f60615-kserve-75797f546f-m2cxk\" (UID: \"37e265c6-5c6e-428e-ae71-516da6a503e5\") " pod="ai-tenant-e2e-explicit-f60615/explicit-model-f60615-kserve-75797f546f-m2cxk"
Sep 11 19:05:22.034543 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:22.034512    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs6v4\" (UniqueName: \"kubernetes.io/projected/37e265c6-5c6e-428e-ae71-516da6a503e5-kube-api-access-xs6v4\") pod \"explicit-model-f60615-kserve-75797f546f-m2cxk\" (UID: \"37e265c6-5c6e-428e-ae71-516da6a503e5\") " pod="ai-tenant-e2e-explicit-f60615/explicit-model-f60615-kserve-75797f546f-m2cxk"
Sep 11 19:05:22.144516 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:22.144410    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-f60615/explicit-model-f60615-kserve-75797f546f-m2cxk"
Sep 11 19:05:22.417301 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:22.417279    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-f60615/explicit-model-f60615-kserve-75797f546f-m2cxk"]
Sep 11 19:05:22.419304 ip-10-0-134-189 kubenswrapper[2578]: W0911 19:05:22.419274    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod37e265c6_5c6e_428e_ae71_516da6a503e5.slice/crio-a6a55e849acc7bf6f55ebb1845855e2e93f6e689f5f325a316b2af0829b3a4b4 WatchSource:0}: Error finding container a6a55e849acc7bf6f55ebb1845855e2e93f6e689f5f325a316b2af0829b3a4b4: Status 404 returned error can't find the container with id a6a55e849acc7bf6f55ebb1845855e2e93f6e689f5f325a316b2af0829b3a4b4
Sep 11 19:05:22.866817 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:22.866780    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-f60615/explicit-model-f60615-kserve-75797f546f-m2cxk" event={"ID":"37e265c6-5c6e-428e-ae71-516da6a503e5","Type":"ContainerStarted","Data":"5d399cf15b84f4257c75930171873b4e5fd000c64ffdba4dc84c13ccc2caba44"}
Sep 11 19:05:22.867225 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:22.866826    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-f60615/explicit-model-f60615-kserve-75797f546f-m2cxk" event={"ID":"37e265c6-5c6e-428e-ae71-516da6a503e5","Type":"ContainerStarted","Data":"a6a55e849acc7bf6f55ebb1845855e2e93f6e689f5f325a316b2af0829b3a4b4"}
Sep 11 19:05:22.885005 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:22.884958    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-f60615/explicit-model-f60615-kserve-75797f546f-m2cxk" podStartSLOduration=1.884941896 podStartE2EDuration="1.884941896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:05:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:05:22.883725156 +0000 UTC m=+998.961315724" watchObservedRunningTime="2026-09-11 19:05:22.884941896 +0000 UTC m=+998.962532454"
Sep 11 19:05:32.144772 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:32.144733    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-f60615/explicit-model-f60615-kserve-75797f546f-m2cxk"
Sep 11 19:05:32.144772 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:32.144782    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-f60615/explicit-model-f60615-kserve-75797f546f-m2cxk"
Sep 11 19:05:32.157307 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:32.157279    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-f60615/explicit-model-f60615-kserve-75797f546f-m2cxk"
Sep 11 19:05:32.927966 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:32.927932    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-f60615/explicit-model-f60615-kserve-75797f546f-m2cxk"
Sep 11 19:05:46.431360 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:46.431325    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-c1aebc/e2e-conflict-a-bb3c1a9d-kserve-6b8d5675c8-wzrlw"]
Sep 11 19:05:46.434806 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:46.434790    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c1aebc/e2e-conflict-a-bb3c1a9d-kserve-6b8d5675c8-wzrlw"
Sep 11 19:05:46.436886 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:46.436860    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-c1aebc\"/\"e2e-conflict-a-bb3c1a9d-kserve-self-signed-certs\""
Sep 11 19:05:46.442346 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:46.442328    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c1aebc/e2e-conflict-a-bb3c1a9d-kserve-6b8d5675c8-wzrlw"]
Sep 11 19:05:46.541902 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:46.541857    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b6675d9-8462-41df-aa72-45a9d176137c-home\") pod \"e2e-conflict-a-bb3c1a9d-kserve-6b8d5675c8-wzrlw\" (UID: \"6b6675d9-8462-41df-aa72-45a9d176137c\") " pod="e2e-models-e2e-worker-w4-c1aebc/e2e-conflict-a-bb3c1a9d-kserve-6b8d5675c8-wzrlw"
Sep 11 19:05:46.542104 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:46.541984    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b6675d9-8462-41df-aa72-45a9d176137c-model-cache\") pod \"e2e-conflict-a-bb3c1a9d-kserve-6b8d5675c8-wzrlw\" (UID: \"6b6675d9-8462-41df-aa72-45a9d176137c\") " pod="e2e-models-e2e-worker-w4-c1aebc/e2e-conflict-a-bb3c1a9d-kserve-6b8d5675c8-wzrlw"
Sep 11 19:05:46.542104 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:46.542039    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b6675d9-8462-41df-aa72-45a9d176137c-dshm\") pod \"e2e-conflict-a-bb3c1a9d-kserve-6b8d5675c8-wzrlw\" (UID: \"6b6675d9-8462-41df-aa72-45a9d176137c\") " pod="e2e-models-e2e-worker-w4-c1aebc/e2e-conflict-a-bb3c1a9d-kserve-6b8d5675c8-wzrlw"
Sep 11 19:05:46.542104 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:46.542079    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b6675d9-8462-41df-aa72-45a9d176137c-tmp-dir\") pod \"e2e-conflict-a-bb3c1a9d-kserve-6b8d5675c8-wzrlw\" (UID: \"6b6675d9-8462-41df-aa72-45a9d176137c\") " pod="e2e-models-e2e-worker-w4-c1aebc/e2e-conflict-a-bb3c1a9d-kserve-6b8d5675c8-wzrlw"
Sep 11 19:05:46.542104 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:46.542098    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5nd4\" (UniqueName: \"kubernetes.io/projected/6b6675d9-8462-41df-aa72-45a9d176137c-kube-api-access-w5nd4\") pod \"e2e-conflict-a-bb3c1a9d-kserve-6b8d5675c8-wzrlw\" (UID: \"6b6675d9-8462-41df-aa72-45a9d176137c\") " pod="e2e-models-e2e-worker-w4-c1aebc/e2e-conflict-a-bb3c1a9d-kserve-6b8d5675c8-wzrlw"
Sep 11 19:05:46.542321 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:46.542172    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b6675d9-8462-41df-aa72-45a9d176137c-tls-certs\") pod \"e2e-conflict-a-bb3c1a9d-kserve-6b8d5675c8-wzrlw\" (UID: \"6b6675d9-8462-41df-aa72-45a9d176137c\") " pod="e2e-models-e2e-worker-w4-c1aebc/e2e-conflict-a-bb3c1a9d-kserve-6b8d5675c8-wzrlw"
Sep 11 19:05:46.643393 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:46.643353    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b6675d9-8462-41df-aa72-45a9d176137c-model-cache\") pod \"e2e-conflict-a-bb3c1a9d-kserve-6b8d5675c8-wzrlw\" (UID: \"6b6675d9-8462-41df-aa72-45a9d176137c\") " pod="e2e-models-e2e-worker-w4-c1aebc/e2e-conflict-a-bb3c1a9d-kserve-6b8d5675c8-wzrlw"
Sep 11 19:05:46.643629 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:46.643403    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b6675d9-8462-41df-aa72-45a9d176137c-dshm\") pod \"e2e-conflict-a-bb3c1a9d-kserve-6b8d5675c8-wzrlw\" (UID: \"6b6675d9-8462-41df-aa72-45a9d176137c\") " pod="e2e-models-e2e-worker-w4-c1aebc/e2e-conflict-a-bb3c1a9d-kserve-6b8d5675c8-wzrlw"
Sep 11 19:05:46.643629 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:46.643434    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b6675d9-8462-41df-aa72-45a9d176137c-tmp-dir\") pod \"e2e-conflict-a-bb3c1a9d-kserve-6b8d5675c8-wzrlw\" (UID: \"6b6675d9-8462-41df-aa72-45a9d176137c\") " pod="e2e-models-e2e-worker-w4-c1aebc/e2e-conflict-a-bb3c1a9d-kserve-6b8d5675c8-wzrlw"
Sep 11 19:05:46.643629 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:46.643451    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5nd4\" (UniqueName: \"kubernetes.io/projected/6b6675d9-8462-41df-aa72-45a9d176137c-kube-api-access-w5nd4\") pod \"e2e-conflict-a-bb3c1a9d-kserve-6b8d5675c8-wzrlw\" (UID: \"6b6675d9-8462-41df-aa72-45a9d176137c\") " pod="e2e-models-e2e-worker-w4-c1aebc/e2e-conflict-a-bb3c1a9d-kserve-6b8d5675c8-wzrlw"
Sep 11 19:05:46.643629 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:46.643476    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b6675d9-8462-41df-aa72-45a9d176137c-tls-certs\") pod \"e2e-conflict-a-bb3c1a9d-kserve-6b8d5675c8-wzrlw\" (UID: \"6b6675d9-8462-41df-aa72-45a9d176137c\") " pod="e2e-models-e2e-worker-w4-c1aebc/e2e-conflict-a-bb3c1a9d-kserve-6b8d5675c8-wzrlw"
Sep 11 19:05:46.643629 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:46.643506    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b6675d9-8462-41df-aa72-45a9d176137c-home\") pod \"e2e-conflict-a-bb3c1a9d-kserve-6b8d5675c8-wzrlw\" (UID: \"6b6675d9-8462-41df-aa72-45a9d176137c\") " pod="e2e-models-e2e-worker-w4-c1aebc/e2e-conflict-a-bb3c1a9d-kserve-6b8d5675c8-wzrlw"
Sep 11 19:05:46.643904 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:46.643808    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b6675d9-8462-41df-aa72-45a9d176137c-model-cache\") pod \"e2e-conflict-a-bb3c1a9d-kserve-6b8d5675c8-wzrlw\" (UID: \"6b6675d9-8462-41df-aa72-45a9d176137c\") " pod="e2e-models-e2e-worker-w4-c1aebc/e2e-conflict-a-bb3c1a9d-kserve-6b8d5675c8-wzrlw"
Sep 11 19:05:46.643904 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:46.643863    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b6675d9-8462-41df-aa72-45a9d176137c-tmp-dir\") pod \"e2e-conflict-a-bb3c1a9d-kserve-6b8d5675c8-wzrlw\" (UID: \"6b6675d9-8462-41df-aa72-45a9d176137c\") " pod="e2e-models-e2e-worker-w4-c1aebc/e2e-conflict-a-bb3c1a9d-kserve-6b8d5675c8-wzrlw"
Sep 11 19:05:46.643978 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:46.643952    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b6675d9-8462-41df-aa72-45a9d176137c-home\") pod \"e2e-conflict-a-bb3c1a9d-kserve-6b8d5675c8-wzrlw\" (UID: \"6b6675d9-8462-41df-aa72-45a9d176137c\") " pod="e2e-models-e2e-worker-w4-c1aebc/e2e-conflict-a-bb3c1a9d-kserve-6b8d5675c8-wzrlw"
Sep 11 19:05:46.645911 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:46.645881    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b6675d9-8462-41df-aa72-45a9d176137c-dshm\") pod \"e2e-conflict-a-bb3c1a9d-kserve-6b8d5675c8-wzrlw\" (UID: \"6b6675d9-8462-41df-aa72-45a9d176137c\") " pod="e2e-models-e2e-worker-w4-c1aebc/e2e-conflict-a-bb3c1a9d-kserve-6b8d5675c8-wzrlw"
Sep 11 19:05:46.646136 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:46.646119    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b6675d9-8462-41df-aa72-45a9d176137c-tls-certs\") pod \"e2e-conflict-a-bb3c1a9d-kserve-6b8d5675c8-wzrlw\" (UID: \"6b6675d9-8462-41df-aa72-45a9d176137c\") " pod="e2e-models-e2e-worker-w4-c1aebc/e2e-conflict-a-bb3c1a9d-kserve-6b8d5675c8-wzrlw"
Sep 11 19:05:46.651397 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:46.651371    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5nd4\" (UniqueName: \"kubernetes.io/projected/6b6675d9-8462-41df-aa72-45a9d176137c-kube-api-access-w5nd4\") pod \"e2e-conflict-a-bb3c1a9d-kserve-6b8d5675c8-wzrlw\" (UID: \"6b6675d9-8462-41df-aa72-45a9d176137c\") " pod="e2e-models-e2e-worker-w4-c1aebc/e2e-conflict-a-bb3c1a9d-kserve-6b8d5675c8-wzrlw"
Sep 11 19:05:46.744667 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:46.744589    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c1aebc/e2e-conflict-a-bb3c1a9d-kserve-6b8d5675c8-wzrlw"
Sep 11 19:05:47.018635 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:47.018608    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c1aebc/e2e-conflict-a-bb3c1a9d-kserve-6b8d5675c8-wzrlw"]
Sep 11 19:05:47.023481 ip-10-0-134-189 kubenswrapper[2578]: W0911 19:05:47.023429    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6b6675d9_8462_41df_aa72_45a9d176137c.slice/crio-64667937c4554c6c90fd28231596b5dee84a5bb4426483675bd6ecb1273c32cd WatchSource:0}: Error finding container 64667937c4554c6c90fd28231596b5dee84a5bb4426483675bd6ecb1273c32cd: Status 404 returned error can't find the container with id 64667937c4554c6c90fd28231596b5dee84a5bb4426483675bd6ecb1273c32cd
Sep 11 19:05:47.994425 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:47.994383    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c1aebc/e2e-conflict-a-bb3c1a9d-kserve-6b8d5675c8-wzrlw" event={"ID":"6b6675d9-8462-41df-aa72-45a9d176137c","Type":"ContainerStarted","Data":"fbf74543c12723b8596ca48406bef2784161a37e719d1c1c26ba636c93778019"}
Sep 11 19:05:47.994895 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:47.994438    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c1aebc/e2e-conflict-a-bb3c1a9d-kserve-6b8d5675c8-wzrlw" event={"ID":"6b6675d9-8462-41df-aa72-45a9d176137c","Type":"ContainerStarted","Data":"64667937c4554c6c90fd28231596b5dee84a5bb4426483675bd6ecb1273c32cd"}
Sep 11 19:05:48.009646 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:48.009572    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-c1aebc/e2e-conflict-a-bb3c1a9d-kserve-6b8d5675c8-wzrlw" podStartSLOduration=2.009556949 podStartE2EDuration="2.009556949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:05:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:05:48.008375792 +0000 UTC m=+1024.085966347" watchObservedRunningTime="2026-09-11 19:05:48.009556949 +0000 UTC m=+1024.087147504"
Sep 11 19:05:50.158848 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:50.158812    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-f60615/explicit-model-f60615-kserve-75797f546f-m2cxk"]
Sep 11 19:05:50.311329 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:50.311295    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f60615-d6hc8"]
Sep 11 19:05:50.314791 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:50.314770    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f60615-d6hc8"
Sep 11 19:05:50.316681 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:50.316657    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 19:05:50.316896 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:50.316882    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ln295\""
Sep 11 19:05:50.322365 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:50.322319    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f60615-d6hc8"]
Sep 11 19:05:50.380386 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:50.380346    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31bd92dc-8b63-4151-9dbf-4fb21971e70e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-f60615-d6hc8\" (UID: \"31bd92dc-8b63-4151-9dbf-4fb21971e70e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f60615-d6hc8"
Sep 11 19:05:50.480972 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:50.480932    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31bd92dc-8b63-4151-9dbf-4fb21971e70e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-f60615-d6hc8\" (UID: \"31bd92dc-8b63-4151-9dbf-4fb21971e70e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f60615-d6hc8"
Sep 11 19:05:50.481576 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:50.481555    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31bd92dc-8b63-4151-9dbf-4fb21971e70e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-f60615-d6hc8\" (UID: \"31bd92dc-8b63-4151-9dbf-4fb21971e70e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f60615-d6hc8"
Sep 11 19:05:50.626720 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:50.626684    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f60615-d6hc8"
Sep 11 19:05:50.901195 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:50.901168    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f60615-d6hc8"]
Sep 11 19:05:50.903310 ip-10-0-134-189 kubenswrapper[2578]: W0911 19:05:50.903285    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31bd92dc_8b63_4151_9dbf_4fb21971e70e.slice/crio-8122dde88f8baf093c6b75c34ec63e17b5d6f887b27f7ae13b02d6fd77809dc0 WatchSource:0}: Error finding container 8122dde88f8baf093c6b75c34ec63e17b5d6f887b27f7ae13b02d6fd77809dc0: Status 404 returned error can't find the container with id 8122dde88f8baf093c6b75c34ec63e17b5d6f887b27f7ae13b02d6fd77809dc0
Sep 11 19:05:51.012474 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:51.012435    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f60615-d6hc8" event={"ID":"31bd92dc-8b63-4151-9dbf-4fb21971e70e","Type":"ContainerStarted","Data":"8122dde88f8baf093c6b75c34ec63e17b5d6f887b27f7ae13b02d6fd77809dc0"}
Sep 11 19:05:52.019458 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:52.019421    2578 generic.go:356] "Generic (PLEG): container finished" podID="31bd92dc-8b63-4151-9dbf-4fb21971e70e" containerID="cc6f7f131a258e4757f8018fae984ac3ee2c7c2fb3db06f5873508d557d06c1f" exitCode=0
Sep 11 19:05:52.019923 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:52.019497    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f60615-d6hc8" event={"ID":"31bd92dc-8b63-4151-9dbf-4fb21971e70e","Type":"ContainerDied","Data":"cc6f7f131a258e4757f8018fae984ac3ee2c7c2fb3db06f5873508d557d06c1f"}
Sep 11 19:05:53.167752 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:53.167722    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f60615-d6hc8"
Sep 11 19:05:53.207513 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:53.207474    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31bd92dc-8b63-4151-9dbf-4fb21971e70e-maas-api-service-ca\") pod \"31bd92dc-8b63-4151-9dbf-4fb21971e70e\" (UID: \"31bd92dc-8b63-4151-9dbf-4fb21971e70e\") "
Sep 11 19:05:53.207866 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:53.207841    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31bd92dc-8b63-4151-9dbf-4fb21971e70e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "31bd92dc-8b63-4151-9dbf-4fb21971e70e" (UID: "31bd92dc-8b63-4151-9dbf-4fb21971e70e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 19:05:53.308810 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:53.308714    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31bd92dc-8b63-4151-9dbf-4fb21971e70e-maas-api-service-ca\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:05:54.032166 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:54.032134    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f60615-d6hc8"
Sep 11 19:05:54.032166 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:54.032161    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f60615-d6hc8" event={"ID":"31bd92dc-8b63-4151-9dbf-4fb21971e70e","Type":"ContainerDied","Data":"8122dde88f8baf093c6b75c34ec63e17b5d6f887b27f7ae13b02d6fd77809dc0"}
Sep 11 19:05:54.032400 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:54.032210    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8122dde88f8baf093c6b75c34ec63e17b5d6f887b27f7ae13b02d6fd77809dc0"
Sep 11 19:05:56.745154 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:56.745118    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-c1aebc/e2e-conflict-a-bb3c1a9d-kserve-6b8d5675c8-wzrlw"
Sep 11 19:05:56.745569 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:56.745176    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c1aebc/e2e-conflict-a-bb3c1a9d-kserve-6b8d5675c8-wzrlw"
Sep 11 19:05:56.757985 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:56.757957    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-c1aebc/e2e-conflict-a-bb3c1a9d-kserve-6b8d5675c8-wzrlw"
Sep 11 19:05:57.058497 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:57.058414    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c1aebc/e2e-conflict-a-bb3c1a9d-kserve-6b8d5675c8-wzrlw"
Sep 11 19:05:57.908269 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:57.908238    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-c6c610/test-model-c6c610-kserve-5bfd568d7f-5mm5x"]
Sep 11 19:05:58.977008 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:58.976969    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9a60dde0-fjptc"]
Sep 11 19:05:58.977395 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:58.977370    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31bd92dc-8b63-4151-9dbf-4fb21971e70e" containerName="revoke-keys"
Sep 11 19:05:58.977395 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:58.977381    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="31bd92dc-8b63-4151-9dbf-4fb21971e70e" containerName="revoke-keys"
Sep 11 19:05:58.977466 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:58.977440    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="31bd92dc-8b63-4151-9dbf-4fb21971e70e" containerName="revoke-keys"
Sep 11 19:05:58.980492 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:58.980475    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9a60dde0-fjptc"
Sep 11 19:05:58.982862 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:58.982840    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 19:05:58.983008 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:58.982848    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ln295\""
Sep 11 19:05:58.987705 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:58.987681    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9a60dde0-fjptc"]
Sep 11 19:05:59.066384 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:59.066331    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75e3e873-f895-48a8-801f-58d62d5b96e9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-9a60dde0-fjptc\" (UID: \"75e3e873-f895-48a8-801f-58d62d5b96e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9a60dde0-fjptc"
Sep 11 19:05:59.167577 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:59.167539    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75e3e873-f895-48a8-801f-58d62d5b96e9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-9a60dde0-fjptc\" (UID: \"75e3e873-f895-48a8-801f-58d62d5b96e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9a60dde0-fjptc"
Sep 11 19:05:59.168191 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:59.168165    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75e3e873-f895-48a8-801f-58d62d5b96e9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-9a60dde0-fjptc\" (UID: \"75e3e873-f895-48a8-801f-58d62d5b96e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9a60dde0-fjptc"
Sep 11 19:05:59.292711 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:59.292624    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9a60dde0-fjptc"
Sep 11 19:05:59.568435 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:05:59.568408    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9a60dde0-fjptc"]
Sep 11 19:05:59.570492 ip-10-0-134-189 kubenswrapper[2578]: W0911 19:05:59.570463    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75e3e873_f895_48a8_801f_58d62d5b96e9.slice/crio-97132e1b289e0ad573f85f88ee73dd406268d7b5eb654640b2298b18ec6e8280 WatchSource:0}: Error finding container 97132e1b289e0ad573f85f88ee73dd406268d7b5eb654640b2298b18ec6e8280: Status 404 returned error can't find the container with id 97132e1b289e0ad573f85f88ee73dd406268d7b5eb654640b2298b18ec6e8280
Sep 11 19:06:00.066167 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:00.066132    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9a60dde0-fjptc" event={"ID":"75e3e873-f895-48a8-801f-58d62d5b96e9","Type":"ContainerStarted","Data":"66b4b7551633890d0d99bfa1c8660c219eb8e83f0ebc7fb041c4210c2cda6b2d"}
Sep 11 19:06:00.066586 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:00.066187    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9a60dde0-fjptc" event={"ID":"75e3e873-f895-48a8-801f-58d62d5b96e9","Type":"ContainerStarted","Data":"97132e1b289e0ad573f85f88ee73dd406268d7b5eb654640b2298b18ec6e8280"}
Sep 11 19:06:00.083179 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:00.083116    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9a60dde0-fjptc" podStartSLOduration=2.083095204 podStartE2EDuration="2.083095204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:05:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:06:00.082218034 +0000 UTC m=+1036.159808626" watchObservedRunningTime="2026-09-11 19:06:00.083095204 +0000 UTC m=+1036.160685763"
Sep 11 19:06:01.073408 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:01.073366    2578 generic.go:356] "Generic (PLEG): container finished" podID="75e3e873-f895-48a8-801f-58d62d5b96e9" containerID="66b4b7551633890d0d99bfa1c8660c219eb8e83f0ebc7fb041c4210c2cda6b2d" exitCode=0
Sep 11 19:06:01.073922 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:01.073464    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9a60dde0-fjptc" event={"ID":"75e3e873-f895-48a8-801f-58d62d5b96e9","Type":"ContainerDied","Data":"66b4b7551633890d0d99bfa1c8660c219eb8e83f0ebc7fb041c4210c2cda6b2d"}
Sep 11 19:06:01.713815 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:01.713781    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f60615-d6hc8"]
Sep 11 19:06:01.719076 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:01.719046    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f60615-d6hc8"]
Sep 11 19:06:02.210661 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:02.210635    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9a60dde0-fjptc"
Sep 11 19:06:02.304766 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:02.304717    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75e3e873-f895-48a8-801f-58d62d5b96e9-maas-api-service-ca\") pod \"75e3e873-f895-48a8-801f-58d62d5b96e9\" (UID: \"75e3e873-f895-48a8-801f-58d62d5b96e9\") "
Sep 11 19:06:02.305230 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:02.305197    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75e3e873-f895-48a8-801f-58d62d5b96e9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "75e3e873-f895-48a8-801f-58d62d5b96e9" (UID: "75e3e873-f895-48a8-801f-58d62d5b96e9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 19:06:02.397744 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:02.397661    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31bd92dc-8b63-4151-9dbf-4fb21971e70e" path="/var/lib/kubelet/pods/31bd92dc-8b63-4151-9dbf-4fb21971e70e/volumes"
Sep 11 19:06:02.405539 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:02.405517    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75e3e873-f895-48a8-801f-58d62d5b96e9-maas-api-service-ca\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:06:03.085665 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:03.085638    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9a60dde0-fjptc"
Sep 11 19:06:03.085846 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:03.085630    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9a60dde0-fjptc" event={"ID":"75e3e873-f895-48a8-801f-58d62d5b96e9","Type":"ContainerDied","Data":"97132e1b289e0ad573f85f88ee73dd406268d7b5eb654640b2298b18ec6e8280"}
Sep 11 19:06:03.085846 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:03.085757    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="97132e1b289e0ad573f85f88ee73dd406268d7b5eb654640b2298b18ec6e8280"
Sep 11 19:06:05.181083 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:05.181045    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-f60615/explicit-model-f60615-kserve-75797f546f-m2cxk" podUID="37e265c6-5c6e-428e-ae71-516da6a503e5" containerName="main" containerID="cri-o://5d399cf15b84f4257c75930171873b4e5fd000c64ffdba4dc84c13ccc2caba44" gracePeriod=45
Sep 11 19:06:05.429252 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:05.429220    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-f60615/explicit-model-f60615-kserve-75797f546f-m2cxk"
Sep 11 19:06:05.536845 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:05.536813    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37e265c6-5c6e-428e-ae71-516da6a503e5-tmp-dir\") pod \"37e265c6-5c6e-428e-ae71-516da6a503e5\" (UID: \"37e265c6-5c6e-428e-ae71-516da6a503e5\") "
Sep 11 19:06:05.537059 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:05.536882    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37e265c6-5c6e-428e-ae71-516da6a503e5-tls-certs\") pod \"37e265c6-5c6e-428e-ae71-516da6a503e5\" (UID: \"37e265c6-5c6e-428e-ae71-516da6a503e5\") "
Sep 11 19:06:05.537059 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:05.536923    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xs6v4\" (UniqueName: \"kubernetes.io/projected/37e265c6-5c6e-428e-ae71-516da6a503e5-kube-api-access-xs6v4\") pod \"37e265c6-5c6e-428e-ae71-516da6a503e5\" (UID: \"37e265c6-5c6e-428e-ae71-516da6a503e5\") "
Sep 11 19:06:05.537059 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:05.536941    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37e265c6-5c6e-428e-ae71-516da6a503e5-dshm\") pod \"37e265c6-5c6e-428e-ae71-516da6a503e5\" (UID: \"37e265c6-5c6e-428e-ae71-516da6a503e5\") "
Sep 11 19:06:05.537059 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:05.536970    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37e265c6-5c6e-428e-ae71-516da6a503e5-home\") pod \"37e265c6-5c6e-428e-ae71-516da6a503e5\" (UID: \"37e265c6-5c6e-428e-ae71-516da6a503e5\") "
Sep 11 19:06:05.537059 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:05.537027    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37e265c6-5c6e-428e-ae71-516da6a503e5-model-cache\") pod \"37e265c6-5c6e-428e-ae71-516da6a503e5\" (UID: \"37e265c6-5c6e-428e-ae71-516da6a503e5\") "
Sep 11 19:06:05.537346 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:05.537098    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37e265c6-5c6e-428e-ae71-516da6a503e5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "37e265c6-5c6e-428e-ae71-516da6a503e5" (UID: "37e265c6-5c6e-428e-ae71-516da6a503e5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:06:05.537346 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:05.537264    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37e265c6-5c6e-428e-ae71-516da6a503e5-home" (OuterVolumeSpecName: "home") pod "37e265c6-5c6e-428e-ae71-516da6a503e5" (UID: "37e265c6-5c6e-428e-ae71-516da6a503e5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:06:05.537471 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:05.537350    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37e265c6-5c6e-428e-ae71-516da6a503e5-tmp-dir\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:06:05.537471 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:05.537351    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37e265c6-5c6e-428e-ae71-516da6a503e5-model-cache" (OuterVolumeSpecName: "model-cache") pod "37e265c6-5c6e-428e-ae71-516da6a503e5" (UID: "37e265c6-5c6e-428e-ae71-516da6a503e5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:06:05.537471 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:05.537372    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37e265c6-5c6e-428e-ae71-516da6a503e5-home\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:06:05.539388 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:05.539364    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37e265c6-5c6e-428e-ae71-516da6a503e5-dshm" (OuterVolumeSpecName: "dshm") pod "37e265c6-5c6e-428e-ae71-516da6a503e5" (UID: "37e265c6-5c6e-428e-ae71-516da6a503e5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:06:05.539496 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:05.539404    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37e265c6-5c6e-428e-ae71-516da6a503e5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "37e265c6-5c6e-428e-ae71-516da6a503e5" (UID: "37e265c6-5c6e-428e-ae71-516da6a503e5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 19:06:05.539496 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:05.539411    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37e265c6-5c6e-428e-ae71-516da6a503e5-kube-api-access-xs6v4" (OuterVolumeSpecName: "kube-api-access-xs6v4") pod "37e265c6-5c6e-428e-ae71-516da6a503e5" (UID: "37e265c6-5c6e-428e-ae71-516da6a503e5"). InnerVolumeSpecName "kube-api-access-xs6v4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:06:05.638197 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:05.638162    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37e265c6-5c6e-428e-ae71-516da6a503e5-model-cache\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:06:05.638197 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:05.638191    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37e265c6-5c6e-428e-ae71-516da6a503e5-tls-certs\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:06:05.638197 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:05.638201    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xs6v4\" (UniqueName: \"kubernetes.io/projected/37e265c6-5c6e-428e-ae71-516da6a503e5-kube-api-access-xs6v4\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:06:05.638431 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:05.638210    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37e265c6-5c6e-428e-ae71-516da6a503e5-dshm\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:06:06.106639 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:06.106580    2578 generic.go:356] "Generic (PLEG): container finished" podID="37e265c6-5c6e-428e-ae71-516da6a503e5" containerID="5d399cf15b84f4257c75930171873b4e5fd000c64ffdba4dc84c13ccc2caba44" exitCode=0
Sep 11 19:06:06.106855 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:06.106649    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-f60615/explicit-model-f60615-kserve-75797f546f-m2cxk"
Sep 11 19:06:06.106855 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:06.106665    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-f60615/explicit-model-f60615-kserve-75797f546f-m2cxk" event={"ID":"37e265c6-5c6e-428e-ae71-516da6a503e5","Type":"ContainerDied","Data":"5d399cf15b84f4257c75930171873b4e5fd000c64ffdba4dc84c13ccc2caba44"}
Sep 11 19:06:06.106855 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:06.106713    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-f60615/explicit-model-f60615-kserve-75797f546f-m2cxk" event={"ID":"37e265c6-5c6e-428e-ae71-516da6a503e5","Type":"ContainerDied","Data":"a6a55e849acc7bf6f55ebb1845855e2e93f6e689f5f325a316b2af0829b3a4b4"}
Sep 11 19:06:06.106855 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:06.106731    2578 scope.go:117] "RemoveContainer" containerID="5d399cf15b84f4257c75930171873b4e5fd000c64ffdba4dc84c13ccc2caba44"
Sep 11 19:06:06.116640 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:06.116585    2578 scope.go:117] "RemoveContainer" containerID="5d399cf15b84f4257c75930171873b4e5fd000c64ffdba4dc84c13ccc2caba44"
Sep 11 19:06:06.116969 ip-10-0-134-189 kubenswrapper[2578]: E0911 19:06:06.116946    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d399cf15b84f4257c75930171873b4e5fd000c64ffdba4dc84c13ccc2caba44\": container with ID starting with 5d399cf15b84f4257c75930171873b4e5fd000c64ffdba4dc84c13ccc2caba44 not found: ID does not exist" containerID="5d399cf15b84f4257c75930171873b4e5fd000c64ffdba4dc84c13ccc2caba44"
Sep 11 19:06:06.117043 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:06.116983    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d399cf15b84f4257c75930171873b4e5fd000c64ffdba4dc84c13ccc2caba44"} err="failed to get container status \"5d399cf15b84f4257c75930171873b4e5fd000c64ffdba4dc84c13ccc2caba44\": rpc error: code = NotFound desc = could not find container \"5d399cf15b84f4257c75930171873b4e5fd000c64ffdba4dc84c13ccc2caba44\": container with ID starting with 5d399cf15b84f4257c75930171873b4e5fd000c64ffdba4dc84c13ccc2caba44 not found: ID does not exist"
Sep 11 19:06:06.128317 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:06.128290    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-f60615/explicit-model-f60615-kserve-75797f546f-m2cxk"]
Sep 11 19:06:06.135027 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:06.134981    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-f60615/explicit-model-f60615-kserve-75797f546f-m2cxk"]
Sep 11 19:06:06.397037 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:06.396948    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37e265c6-5c6e-428e-ae71-516da6a503e5" path="/var/lib/kubelet/pods/37e265c6-5c6e-428e-ae71-516da6a503e5/volumes"
Sep 11 19:06:09.524818 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:09.524774    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9a60dde0-fjptc"]
Sep 11 19:06:09.528823 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:09.528788    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9a60dde0-fjptc"]
Sep 11 19:06:10.397886 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:10.397848    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75e3e873-f895-48a8-801f-58d62d5b96e9" path="/var/lib/kubelet/pods/75e3e873-f895-48a8-801f-58d62d5b96e9/volumes"
Sep 11 19:06:12.928742 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:12.928680    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-c6c610/test-model-c6c610-kserve-5bfd568d7f-5mm5x" podUID="da1f5b99-5812-438f-a59b-dbd45c9ca454" containerName="main" containerID="cri-o://9b766c5f8f3f7f4b0275ea21fa2ccea56b698d3e35e3becf868053d9e2827f8a" gracePeriod=45
Sep 11 19:06:13.146025 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:13.145988    2578 generic.go:356] "Generic (PLEG): container finished" podID="da1f5b99-5812-438f-a59b-dbd45c9ca454" containerID="9b766c5f8f3f7f4b0275ea21fa2ccea56b698d3e35e3becf868053d9e2827f8a" exitCode=0
Sep 11 19:06:13.146222 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:13.146029    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c6c610/test-model-c6c610-kserve-5bfd568d7f-5mm5x" event={"ID":"da1f5b99-5812-438f-a59b-dbd45c9ca454","Type":"ContainerDied","Data":"9b766c5f8f3f7f4b0275ea21fa2ccea56b698d3e35e3becf868053d9e2827f8a"}
Sep 11 19:06:13.177749 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:13.177721    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c6c610/test-model-c6c610-kserve-5bfd568d7f-5mm5x"
Sep 11 19:06:13.315907 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:13.315876    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nmt26\" (UniqueName: \"kubernetes.io/projected/da1f5b99-5812-438f-a59b-dbd45c9ca454-kube-api-access-nmt26\") pod \"da1f5b99-5812-438f-a59b-dbd45c9ca454\" (UID: \"da1f5b99-5812-438f-a59b-dbd45c9ca454\") "
Sep 11 19:06:13.315907 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:13.315915    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da1f5b99-5812-438f-a59b-dbd45c9ca454-model-cache\") pod \"da1f5b99-5812-438f-a59b-dbd45c9ca454\" (UID: \"da1f5b99-5812-438f-a59b-dbd45c9ca454\") "
Sep 11 19:06:13.316161 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:13.315950    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da1f5b99-5812-438f-a59b-dbd45c9ca454-tls-certs\") pod \"da1f5b99-5812-438f-a59b-dbd45c9ca454\" (UID: \"da1f5b99-5812-438f-a59b-dbd45c9ca454\") "
Sep 11 19:06:13.316161 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:13.315979    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da1f5b99-5812-438f-a59b-dbd45c9ca454-dshm\") pod \"da1f5b99-5812-438f-a59b-dbd45c9ca454\" (UID: \"da1f5b99-5812-438f-a59b-dbd45c9ca454\") "
Sep 11 19:06:13.316161 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:13.316039    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da1f5b99-5812-438f-a59b-dbd45c9ca454-home\") pod \"da1f5b99-5812-438f-a59b-dbd45c9ca454\" (UID: \"da1f5b99-5812-438f-a59b-dbd45c9ca454\") "
Sep 11 19:06:13.316328 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:13.316175    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da1f5b99-5812-438f-a59b-dbd45c9ca454-tmp-dir\") pod \"da1f5b99-5812-438f-a59b-dbd45c9ca454\" (UID: \"da1f5b99-5812-438f-a59b-dbd45c9ca454\") "
Sep 11 19:06:13.316328 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:13.316212    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da1f5b99-5812-438f-a59b-dbd45c9ca454-model-cache" (OuterVolumeSpecName: "model-cache") pod "da1f5b99-5812-438f-a59b-dbd45c9ca454" (UID: "da1f5b99-5812-438f-a59b-dbd45c9ca454"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:06:13.316439 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:13.316334    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da1f5b99-5812-438f-a59b-dbd45c9ca454-home" (OuterVolumeSpecName: "home") pod "da1f5b99-5812-438f-a59b-dbd45c9ca454" (UID: "da1f5b99-5812-438f-a59b-dbd45c9ca454"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:06:13.316496 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:13.316449    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da1f5b99-5812-438f-a59b-dbd45c9ca454-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "da1f5b99-5812-438f-a59b-dbd45c9ca454" (UID: "da1f5b99-5812-438f-a59b-dbd45c9ca454"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:06:13.316645 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:13.316590    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da1f5b99-5812-438f-a59b-dbd45c9ca454-home\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:06:13.316645 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:13.316633    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da1f5b99-5812-438f-a59b-dbd45c9ca454-tmp-dir\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:06:13.316759 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:13.316648    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da1f5b99-5812-438f-a59b-dbd45c9ca454-model-cache\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:06:13.318477 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:13.318444    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da1f5b99-5812-438f-a59b-dbd45c9ca454-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "da1f5b99-5812-438f-a59b-dbd45c9ca454" (UID: "da1f5b99-5812-438f-a59b-dbd45c9ca454"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 19:06:13.318477 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:13.318458    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da1f5b99-5812-438f-a59b-dbd45c9ca454-dshm" (OuterVolumeSpecName: "dshm") pod "da1f5b99-5812-438f-a59b-dbd45c9ca454" (UID: "da1f5b99-5812-438f-a59b-dbd45c9ca454"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:06:13.318657 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:13.318512    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da1f5b99-5812-438f-a59b-dbd45c9ca454-kube-api-access-nmt26" (OuterVolumeSpecName: "kube-api-access-nmt26") pod "da1f5b99-5812-438f-a59b-dbd45c9ca454" (UID: "da1f5b99-5812-438f-a59b-dbd45c9ca454"). InnerVolumeSpecName "kube-api-access-nmt26". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:06:13.417330 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:13.417288    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmt26\" (UniqueName: \"kubernetes.io/projected/da1f5b99-5812-438f-a59b-dbd45c9ca454-kube-api-access-nmt26\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:06:13.417330 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:13.417321    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da1f5b99-5812-438f-a59b-dbd45c9ca454-tls-certs\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:06:13.417330 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:13.417331    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da1f5b99-5812-438f-a59b-dbd45c9ca454-dshm\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:06:14.152510 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:14.152475    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c6c610/test-model-c6c610-kserve-5bfd568d7f-5mm5x"
Sep 11 19:06:14.153004 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:14.152529    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c6c610/test-model-c6c610-kserve-5bfd568d7f-5mm5x" event={"ID":"da1f5b99-5812-438f-a59b-dbd45c9ca454","Type":"ContainerDied","Data":"9304ea7da662078de5446b3ea552ae6db7b6d06cac0f1cd0bad22779dfff9c23"}
Sep 11 19:06:14.153004 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:14.152579    2578 scope.go:117] "RemoveContainer" containerID="9b766c5f8f3f7f4b0275ea21fa2ccea56b698d3e35e3becf868053d9e2827f8a"
Sep 11 19:06:14.173112 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:14.173085    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-c6c610/test-model-c6c610-kserve-5bfd568d7f-5mm5x"]
Sep 11 19:06:14.178352 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:14.178329    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-c6c610/test-model-c6c610-kserve-5bfd568d7f-5mm5x"]
Sep 11 19:06:14.397535 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:14.397497    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da1f5b99-5812-438f-a59b-dbd45c9ca454" path="/var/lib/kubelet/pods/da1f5b99-5812-438f-a59b-dbd45c9ca454/volumes"
Sep 11 19:06:37.258213 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:37.258129    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-093f6c/test-model-093f6c-kserve-544f68f7bc-8wj5z"]
Sep 11 19:06:39.160009 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:39.159975    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c6c610-8cf45f4d7-lq42w"]
Sep 11 19:06:39.160538 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:39.160213    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-c6c610-8cf45f4d7-lq42w" podUID="fe0fddea-982f-4f3f-81bd-7e1a8c268157" containerName="payload-processing" containerID="cri-o://b83cd9fd8704d8ea0bbd8b88a9620afb04c6d18ca708e41220b37a37a4d6c25f" gracePeriod=30
Sep 11 19:06:39.287658 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:39.287627    2578 generic.go:356] "Generic (PLEG): container finished" podID="fe0fddea-982f-4f3f-81bd-7e1a8c268157" containerID="b83cd9fd8704d8ea0bbd8b88a9620afb04c6d18ca708e41220b37a37a4d6c25f" exitCode=0
Sep 11 19:06:39.287813 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:39.287636    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c6c610-8cf45f4d7-lq42w" event={"ID":"fe0fddea-982f-4f3f-81bd-7e1a8c268157","Type":"ContainerDied","Data":"b83cd9fd8704d8ea0bbd8b88a9620afb04c6d18ca708e41220b37a37a4d6c25f"}
Sep 11 19:06:39.423985 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:39.423918    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c6c610-8cf45f4d7-lq42w"
Sep 11 19:06:39.584147 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:39.584112    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe0fddea-982f-4f3f-81bd-7e1a8c268157-plugins-config-volume\") pod \"fe0fddea-982f-4f3f-81bd-7e1a8c268157\" (UID: \"fe0fddea-982f-4f3f-81bd-7e1a8c268157\") "
Sep 11 19:06:39.584326 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:39.584183    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gc8j4\" (UniqueName: \"kubernetes.io/projected/fe0fddea-982f-4f3f-81bd-7e1a8c268157-kube-api-access-gc8j4\") pod \"fe0fddea-982f-4f3f-81bd-7e1a8c268157\" (UID: \"fe0fddea-982f-4f3f-81bd-7e1a8c268157\") "
Sep 11 19:06:39.584504 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:39.584481    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe0fddea-982f-4f3f-81bd-7e1a8c268157-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fe0fddea-982f-4f3f-81bd-7e1a8c268157" (UID: "fe0fddea-982f-4f3f-81bd-7e1a8c268157"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 19:06:39.586565 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:39.586540    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe0fddea-982f-4f3f-81bd-7e1a8c268157-kube-api-access-gc8j4" (OuterVolumeSpecName: "kube-api-access-gc8j4") pod "fe0fddea-982f-4f3f-81bd-7e1a8c268157" (UID: "fe0fddea-982f-4f3f-81bd-7e1a8c268157"). InnerVolumeSpecName "kube-api-access-gc8j4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:06:39.685440 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:39.685347    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe0fddea-982f-4f3f-81bd-7e1a8c268157-plugins-config-volume\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:06:39.685440 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:39.685379    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gc8j4\" (UniqueName: \"kubernetes.io/projected/fe0fddea-982f-4f3f-81bd-7e1a8c268157-kube-api-access-gc8j4\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:06:40.294464 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:40.294433    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c6c610-8cf45f4d7-lq42w"
Sep 11 19:06:40.294931 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:40.294431    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c6c610-8cf45f4d7-lq42w" event={"ID":"fe0fddea-982f-4f3f-81bd-7e1a8c268157","Type":"ContainerDied","Data":"6bdb42afab39d8b076ae1836f0a7457b2f58cd26b2d0087be85b140d77fc328b"}
Sep 11 19:06:40.294931 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:40.294580    2578 scope.go:117] "RemoveContainer" containerID="b83cd9fd8704d8ea0bbd8b88a9620afb04c6d18ca708e41220b37a37a4d6c25f"
Sep 11 19:06:40.319770 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:40.319733    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c6c610-8cf45f4d7-lq42w"]
Sep 11 19:06:40.341177 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:40.341141    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c6c610-8cf45f4d7-lq42w"]
Sep 11 19:06:40.397379 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:40.397342    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe0fddea-982f-4f3f-81bd-7e1a8c268157" path="/var/lib/kubelet/pods/fe0fddea-982f-4f3f-81bd-7e1a8c268157/volumes"
Sep 11 19:06:46.813888 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:46.813845    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-093f6c-d94rc"]
Sep 11 19:06:46.814381 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:46.814332    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da1f5b99-5812-438f-a59b-dbd45c9ca454" containerName="main"
Sep 11 19:06:46.814381 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:46.814351    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="da1f5b99-5812-438f-a59b-dbd45c9ca454" containerName="main"
Sep 11 19:06:46.814381 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:46.814380    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37e265c6-5c6e-428e-ae71-516da6a503e5" containerName="main"
Sep 11 19:06:46.814556 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:46.814388    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="37e265c6-5c6e-428e-ae71-516da6a503e5" containerName="main"
Sep 11 19:06:46.814556 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:46.814403    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75e3e873-f895-48a8-801f-58d62d5b96e9" containerName="revoke-keys"
Sep 11 19:06:46.814556 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:46.814412    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="75e3e873-f895-48a8-801f-58d62d5b96e9" containerName="revoke-keys"
Sep 11 19:06:46.814556 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:46.814434    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe0fddea-982f-4f3f-81bd-7e1a8c268157" containerName="payload-processing"
Sep 11 19:06:46.814556 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:46.814443    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe0fddea-982f-4f3f-81bd-7e1a8c268157" containerName="payload-processing"
Sep 11 19:06:46.814556 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:46.814530    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="75e3e873-f895-48a8-801f-58d62d5b96e9" containerName="revoke-keys"
Sep 11 19:06:46.814556 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:46.814544    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="37e265c6-5c6e-428e-ae71-516da6a503e5" containerName="main"
Sep 11 19:06:46.814556 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:46.814554    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe0fddea-982f-4f3f-81bd-7e1a8c268157" containerName="payload-processing"
Sep 11 19:06:46.815011 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:46.814563    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="da1f5b99-5812-438f-a59b-dbd45c9ca454" containerName="main"
Sep 11 19:06:46.818846 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:46.818822    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-093f6c-d94rc"
Sep 11 19:06:46.821447 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:46.821424    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 19:06:46.821713 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:46.821682    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ln295\""
Sep 11 19:06:46.830036 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:46.830004    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-093f6c-d94rc"]
Sep 11 19:06:46.954662 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:46.954618    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/accb7b81-6ab9-4aaa-aae3-52f4ebb062be-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-093f6c-d94rc\" (UID: \"accb7b81-6ab9-4aaa-aae3-52f4ebb062be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-093f6c-d94rc"
Sep 11 19:06:47.056182 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:47.056143    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/accb7b81-6ab9-4aaa-aae3-52f4ebb062be-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-093f6c-d94rc\" (UID: \"accb7b81-6ab9-4aaa-aae3-52f4ebb062be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-093f6c-d94rc"
Sep 11 19:06:47.056833 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:47.056813    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/accb7b81-6ab9-4aaa-aae3-52f4ebb062be-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-093f6c-d94rc\" (UID: \"accb7b81-6ab9-4aaa-aae3-52f4ebb062be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-093f6c-d94rc"
Sep 11 19:06:47.130913 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:47.130827    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-093f6c-d94rc"
Sep 11 19:06:47.407335 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:47.407287    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-093f6c-d94rc"]
Sep 11 19:06:47.409783 ip-10-0-134-189 kubenswrapper[2578]: W0911 19:06:47.409753    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaccb7b81_6ab9_4aaa_aae3_52f4ebb062be.slice/crio-ddffca173c626040f7023e6b08f3cd3acd979d3c8f6c94e9aa48d3d0c80904d7 WatchSource:0}: Error finding container ddffca173c626040f7023e6b08f3cd3acd979d3c8f6c94e9aa48d3d0c80904d7: Status 404 returned error can't find the container with id ddffca173c626040f7023e6b08f3cd3acd979d3c8f6c94e9aa48d3d0c80904d7
Sep 11 19:06:48.337748 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:48.337648    2578 generic.go:356] "Generic (PLEG): container finished" podID="accb7b81-6ab9-4aaa-aae3-52f4ebb062be" containerID="42bf6979d1825340b0266da137bf14b110f83dbe57ed6d9916771046f09d40b5" exitCode=0
Sep 11 19:06:48.338132 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:48.337736    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-093f6c-d94rc" event={"ID":"accb7b81-6ab9-4aaa-aae3-52f4ebb062be","Type":"ContainerDied","Data":"42bf6979d1825340b0266da137bf14b110f83dbe57ed6d9916771046f09d40b5"}
Sep 11 19:06:48.338132 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:48.337800    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-093f6c-d94rc" event={"ID":"accb7b81-6ab9-4aaa-aae3-52f4ebb062be","Type":"ContainerStarted","Data":"ddffca173c626040f7023e6b08f3cd3acd979d3c8f6c94e9aa48d3d0c80904d7"}
Sep 11 19:06:49.479265 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:49.479232    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-093f6c-d94rc"
Sep 11 19:06:49.574291 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:49.574254    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/accb7b81-6ab9-4aaa-aae3-52f4ebb062be-maas-api-service-ca\") pod \"accb7b81-6ab9-4aaa-aae3-52f4ebb062be\" (UID: \"accb7b81-6ab9-4aaa-aae3-52f4ebb062be\") "
Sep 11 19:06:49.574656 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:49.574629    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/accb7b81-6ab9-4aaa-aae3-52f4ebb062be-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "accb7b81-6ab9-4aaa-aae3-52f4ebb062be" (UID: "accb7b81-6ab9-4aaa-aae3-52f4ebb062be"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 19:06:49.675790 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:49.675697    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/accb7b81-6ab9-4aaa-aae3-52f4ebb062be-maas-api-service-ca\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:06:50.349039 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:50.349008    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-093f6c-d94rc"
Sep 11 19:06:50.349251 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:50.349054    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-093f6c-d94rc" event={"ID":"accb7b81-6ab9-4aaa-aae3-52f4ebb062be","Type":"ContainerDied","Data":"ddffca173c626040f7023e6b08f3cd3acd979d3c8f6c94e9aa48d3d0c80904d7"}
Sep 11 19:06:50.349251 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:50.349104    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ddffca173c626040f7023e6b08f3cd3acd979d3c8f6c94e9aa48d3d0c80904d7"
Sep 11 19:06:52.278684 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:52.278644    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-093f6c/test-model-093f6c-kserve-544f68f7bc-8wj5z" podUID="21fc85b7-8175-4764-85ce-e026bccccc08" containerName="main" containerID="cri-o://594ec5ff346ab2450d165b91832c96c537c7cf6373cbc83c849b2a2c25f59634" gracePeriod=45
Sep 11 19:06:52.528092 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:52.528064    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-093f6c/test-model-093f6c-kserve-544f68f7bc-8wj5z"
Sep 11 19:06:52.599758 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:52.599670    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21fc85b7-8175-4764-85ce-e026bccccc08-tmp-dir\") pod \"21fc85b7-8175-4764-85ce-e026bccccc08\" (UID: \"21fc85b7-8175-4764-85ce-e026bccccc08\") "
Sep 11 19:06:52.599922 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:52.599772    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21fc85b7-8175-4764-85ce-e026bccccc08-tls-certs\") pod \"21fc85b7-8175-4764-85ce-e026bccccc08\" (UID: \"21fc85b7-8175-4764-85ce-e026bccccc08\") "
Sep 11 19:06:52.599922 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:52.599801    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21fc85b7-8175-4764-85ce-e026bccccc08-home\") pod \"21fc85b7-8175-4764-85ce-e026bccccc08\" (UID: \"21fc85b7-8175-4764-85ce-e026bccccc08\") "
Sep 11 19:06:52.599922 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:52.599823    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21fc85b7-8175-4764-85ce-e026bccccc08-dshm\") pod \"21fc85b7-8175-4764-85ce-e026bccccc08\" (UID: \"21fc85b7-8175-4764-85ce-e026bccccc08\") "
Sep 11 19:06:52.599922 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:52.599850    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21fc85b7-8175-4764-85ce-e026bccccc08-model-cache\") pod \"21fc85b7-8175-4764-85ce-e026bccccc08\" (UID: \"21fc85b7-8175-4764-85ce-e026bccccc08\") "
Sep 11 19:06:52.600169 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:52.599934    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qlpg2\" (UniqueName: \"kubernetes.io/projected/21fc85b7-8175-4764-85ce-e026bccccc08-kube-api-access-qlpg2\") pod \"21fc85b7-8175-4764-85ce-e026bccccc08\" (UID: \"21fc85b7-8175-4764-85ce-e026bccccc08\") "
Sep 11 19:06:52.600169 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:52.599982    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21fc85b7-8175-4764-85ce-e026bccccc08-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "21fc85b7-8175-4764-85ce-e026bccccc08" (UID: "21fc85b7-8175-4764-85ce-e026bccccc08"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:06:52.600169 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:52.600078    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21fc85b7-8175-4764-85ce-e026bccccc08-home" (OuterVolumeSpecName: "home") pod "21fc85b7-8175-4764-85ce-e026bccccc08" (UID: "21fc85b7-8175-4764-85ce-e026bccccc08"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:06:52.600169 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:52.600088    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21fc85b7-8175-4764-85ce-e026bccccc08-model-cache" (OuterVolumeSpecName: "model-cache") pod "21fc85b7-8175-4764-85ce-e026bccccc08" (UID: "21fc85b7-8175-4764-85ce-e026bccccc08"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:06:52.600314 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:52.600238    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21fc85b7-8175-4764-85ce-e026bccccc08-tmp-dir\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:06:52.600314 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:52.600250    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21fc85b7-8175-4764-85ce-e026bccccc08-home\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:06:52.600314 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:52.600258    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21fc85b7-8175-4764-85ce-e026bccccc08-model-cache\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:06:52.602222 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:52.602197    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21fc85b7-8175-4764-85ce-e026bccccc08-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "21fc85b7-8175-4764-85ce-e026bccccc08" (UID: "21fc85b7-8175-4764-85ce-e026bccccc08"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 19:06:52.602338 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:52.602247    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21fc85b7-8175-4764-85ce-e026bccccc08-dshm" (OuterVolumeSpecName: "dshm") pod "21fc85b7-8175-4764-85ce-e026bccccc08" (UID: "21fc85b7-8175-4764-85ce-e026bccccc08"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:06:52.602338 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:52.602251    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21fc85b7-8175-4764-85ce-e026bccccc08-kube-api-access-qlpg2" (OuterVolumeSpecName: "kube-api-access-qlpg2") pod "21fc85b7-8175-4764-85ce-e026bccccc08" (UID: "21fc85b7-8175-4764-85ce-e026bccccc08"). InnerVolumeSpecName "kube-api-access-qlpg2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:06:52.700896 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:52.700854    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21fc85b7-8175-4764-85ce-e026bccccc08-tls-certs\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:06:52.700896 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:52.700884    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21fc85b7-8175-4764-85ce-e026bccccc08-dshm\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:06:52.700896 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:52.700895    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlpg2\" (UniqueName: \"kubernetes.io/projected/21fc85b7-8175-4764-85ce-e026bccccc08-kube-api-access-qlpg2\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:06:53.365824 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:53.365788    2578 generic.go:356] "Generic (PLEG): container finished" podID="21fc85b7-8175-4764-85ce-e026bccccc08" containerID="594ec5ff346ab2450d165b91832c96c537c7cf6373cbc83c849b2a2c25f59634" exitCode=0
Sep 11 19:06:53.365824 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:53.365826    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-093f6c/test-model-093f6c-kserve-544f68f7bc-8wj5z"
Sep 11 19:06:53.366323 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:53.365876    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-093f6c/test-model-093f6c-kserve-544f68f7bc-8wj5z" event={"ID":"21fc85b7-8175-4764-85ce-e026bccccc08","Type":"ContainerDied","Data":"594ec5ff346ab2450d165b91832c96c537c7cf6373cbc83c849b2a2c25f59634"}
Sep 11 19:06:53.366323 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:53.365914    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-093f6c/test-model-093f6c-kserve-544f68f7bc-8wj5z" event={"ID":"21fc85b7-8175-4764-85ce-e026bccccc08","Type":"ContainerDied","Data":"3487067cf0c6094708e1bd4928c687a140d8e0ac6ca5d4c90912efa5756e2a0e"}
Sep 11 19:06:53.366323 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:53.365934    2578 scope.go:117] "RemoveContainer" containerID="594ec5ff346ab2450d165b91832c96c537c7cf6373cbc83c849b2a2c25f59634"
Sep 11 19:06:53.375322 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:53.375302    2578 scope.go:117] "RemoveContainer" containerID="594ec5ff346ab2450d165b91832c96c537c7cf6373cbc83c849b2a2c25f59634"
Sep 11 19:06:53.375638 ip-10-0-134-189 kubenswrapper[2578]: E0911 19:06:53.375618    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"594ec5ff346ab2450d165b91832c96c537c7cf6373cbc83c849b2a2c25f59634\": container with ID starting with 594ec5ff346ab2450d165b91832c96c537c7cf6373cbc83c849b2a2c25f59634 not found: ID does not exist" containerID="594ec5ff346ab2450d165b91832c96c537c7cf6373cbc83c849b2a2c25f59634"
Sep 11 19:06:53.375713 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:53.375653    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"594ec5ff346ab2450d165b91832c96c537c7cf6373cbc83c849b2a2c25f59634"} err="failed to get container status \"594ec5ff346ab2450d165b91832c96c537c7cf6373cbc83c849b2a2c25f59634\": rpc error: code = NotFound desc = could not find container \"594ec5ff346ab2450d165b91832c96c537c7cf6373cbc83c849b2a2c25f59634\": container with ID starting with 594ec5ff346ab2450d165b91832c96c537c7cf6373cbc83c849b2a2c25f59634 not found: ID does not exist"
Sep 11 19:06:53.387214 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:53.387185    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-093f6c/test-model-093f6c-kserve-544f68f7bc-8wj5z"]
Sep 11 19:06:53.393356 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:53.393334    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-093f6c/test-model-093f6c-kserve-544f68f7bc-8wj5z"]
Sep 11 19:06:54.399796 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:06:54.398266    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21fc85b7-8175-4764-85ce-e026bccccc08" path="/var/lib/kubelet/pods/21fc85b7-8175-4764-85ce-e026bccccc08/volumes"
Sep 11 19:07:01.432435 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:01.432401    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-093f6c-d94rc"]
Sep 11 19:07:01.439968 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:01.439943    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-093f6c-d94rc"]
Sep 11 19:07:02.397416 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:02.397375    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="accb7b81-6ab9-4aaa-aae3-52f4ebb062be" path="/var/lib/kubelet/pods/accb7b81-6ab9-4aaa-aae3-52f4ebb062be/volumes"
Sep 11 19:07:02.485226 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:02.485193    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-37ce96-85fb758d8f-wqfhk"]
Sep 11 19:07:02.485745 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:02.485726    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21fc85b7-8175-4764-85ce-e026bccccc08" containerName="main"
Sep 11 19:07:02.485838 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:02.485748    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="21fc85b7-8175-4764-85ce-e026bccccc08" containerName="main"
Sep 11 19:07:02.485838 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:02.485767    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="accb7b81-6ab9-4aaa-aae3-52f4ebb062be" containerName="revoke-keys"
Sep 11 19:07:02.485838 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:02.485775    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="accb7b81-6ab9-4aaa-aae3-52f4ebb062be" containerName="revoke-keys"
Sep 11 19:07:02.485987 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:02.485880    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="21fc85b7-8175-4764-85ce-e026bccccc08" containerName="main"
Sep 11 19:07:02.485987 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:02.485897    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="accb7b81-6ab9-4aaa-aae3-52f4ebb062be" containerName="revoke-keys"
Sep 11 19:07:02.490495 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:02.490473    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-37ce96-85fb758d8f-wqfhk"
Sep 11 19:07:02.493164 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:02.493140    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-37ce96\""
Sep 11 19:07:02.493279 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:02.493226    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-37ce96-dockercfg-k6n5n\""
Sep 11 19:07:02.502210 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:02.502181    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-37ce96-85fb758d8f-wqfhk"]
Sep 11 19:07:02.601582 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:02.601544    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb79a5c3-ada2-4aa7-bd25-fc07436a10a6-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-37ce96-85fb758d8f-wqfhk\" (UID: \"fb79a5c3-ada2-4aa7-bd25-fc07436a10a6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-37ce96-85fb758d8f-wqfhk"
Sep 11 19:07:02.601798 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:02.601645    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmj6f\" (UniqueName: \"kubernetes.io/projected/fb79a5c3-ada2-4aa7-bd25-fc07436a10a6-kube-api-access-tmj6f\") pod \"payload-processing-e2e-shared-b-w2-37ce96-85fb758d8f-wqfhk\" (UID: \"fb79a5c3-ada2-4aa7-bd25-fc07436a10a6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-37ce96-85fb758d8f-wqfhk"
Sep 11 19:07:02.702906 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:02.702872    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb79a5c3-ada2-4aa7-bd25-fc07436a10a6-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-37ce96-85fb758d8f-wqfhk\" (UID: \"fb79a5c3-ada2-4aa7-bd25-fc07436a10a6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-37ce96-85fb758d8f-wqfhk"
Sep 11 19:07:02.703062 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:02.702929    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmj6f\" (UniqueName: \"kubernetes.io/projected/fb79a5c3-ada2-4aa7-bd25-fc07436a10a6-kube-api-access-tmj6f\") pod \"payload-processing-e2e-shared-b-w2-37ce96-85fb758d8f-wqfhk\" (UID: \"fb79a5c3-ada2-4aa7-bd25-fc07436a10a6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-37ce96-85fb758d8f-wqfhk"
Sep 11 19:07:02.703709 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:02.703690    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb79a5c3-ada2-4aa7-bd25-fc07436a10a6-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-37ce96-85fb758d8f-wqfhk\" (UID: \"fb79a5c3-ada2-4aa7-bd25-fc07436a10a6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-37ce96-85fb758d8f-wqfhk"
Sep 11 19:07:02.714129 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:02.714095    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmj6f\" (UniqueName: \"kubernetes.io/projected/fb79a5c3-ada2-4aa7-bd25-fc07436a10a6-kube-api-access-tmj6f\") pod \"payload-processing-e2e-shared-b-w2-37ce96-85fb758d8f-wqfhk\" (UID: \"fb79a5c3-ada2-4aa7-bd25-fc07436a10a6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-37ce96-85fb758d8f-wqfhk"
Sep 11 19:07:02.801001 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:02.800962    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-37ce96-85fb758d8f-wqfhk"
Sep 11 19:07:03.096576 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:03.096550    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-37ce96-85fb758d8f-wqfhk"]
Sep 11 19:07:03.099102 ip-10-0-134-189 kubenswrapper[2578]: W0911 19:07:03.099076    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb79a5c3_ada2_4aa7_bd25_fc07436a10a6.slice/crio-468b48fb47f4de882937f0195a10604f10737712abbdd954657f4b565c282850 WatchSource:0}: Error finding container 468b48fb47f4de882937f0195a10604f10737712abbdd954657f4b565c282850: Status 404 returned error can't find the container with id 468b48fb47f4de882937f0195a10604f10737712abbdd954657f4b565c282850
Sep 11 19:07:03.422200 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:03.422092    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-37ce96-85fb758d8f-wqfhk" event={"ID":"fb79a5c3-ada2-4aa7-bd25-fc07436a10a6","Type":"ContainerStarted","Data":"c1c0a49d9db6eae59342d1595986bd82c29c377a095b0abb18e0214e6022aba8"}
Sep 11 19:07:03.422200 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:03.422164    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-37ce96-85fb758d8f-wqfhk"
Sep 11 19:07:03.422200 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:03.422187    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-37ce96-85fb758d8f-wqfhk" event={"ID":"fb79a5c3-ada2-4aa7-bd25-fc07436a10a6","Type":"ContainerStarted","Data":"468b48fb47f4de882937f0195a10604f10737712abbdd954657f4b565c282850"}
Sep 11 19:07:03.443357 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:03.443295    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-37ce96-85fb758d8f-wqfhk" podStartSLOduration=1.4432767499999999 podStartE2EDuration="1.44327675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:07:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:07:03.441040912 +0000 UTC m=+1099.518631477" watchObservedRunningTime="2026-09-11 19:07:03.44327675 +0000 UTC m=+1099.520867304"
Sep 11 19:07:04.922039 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:04.922002    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-c1aebc/e2e-conflict-b-bb3c1a9d-kserve-6d468cf4c6-tmthf"]
Sep 11 19:07:04.925286 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:04.925264    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c1aebc/e2e-conflict-b-bb3c1a9d-kserve-6d468cf4c6-tmthf"
Sep 11 19:07:04.927590 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:04.927572    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-c1aebc\"/\"e2e-conflict-b-bb3c1a9d-kserve-self-signed-certs\""
Sep 11 19:07:04.934716 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:04.934691    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c1aebc/e2e-conflict-b-bb3c1a9d-kserve-6d468cf4c6-tmthf"]
Sep 11 19:07:05.026667 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:05.026583    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpqx7\" (UniqueName: \"kubernetes.io/projected/f025a72f-bdab-4de7-b958-ddda7786c4fa-kube-api-access-jpqx7\") pod \"e2e-conflict-b-bb3c1a9d-kserve-6d468cf4c6-tmthf\" (UID: \"f025a72f-bdab-4de7-b958-ddda7786c4fa\") " pod="e2e-models-e2e-worker-w4-c1aebc/e2e-conflict-b-bb3c1a9d-kserve-6d468cf4c6-tmthf"
Sep 11 19:07:05.026873 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:05.026782    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f025a72f-bdab-4de7-b958-ddda7786c4fa-dshm\") pod \"e2e-conflict-b-bb3c1a9d-kserve-6d468cf4c6-tmthf\" (UID: \"f025a72f-bdab-4de7-b958-ddda7786c4fa\") " pod="e2e-models-e2e-worker-w4-c1aebc/e2e-conflict-b-bb3c1a9d-kserve-6d468cf4c6-tmthf"
Sep 11 19:07:05.026873 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:05.026815    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f025a72f-bdab-4de7-b958-ddda7786c4fa-model-cache\") pod \"e2e-conflict-b-bb3c1a9d-kserve-6d468cf4c6-tmthf\" (UID: \"f025a72f-bdab-4de7-b958-ddda7786c4fa\") " pod="e2e-models-e2e-worker-w4-c1aebc/e2e-conflict-b-bb3c1a9d-kserve-6d468cf4c6-tmthf"
Sep 11 19:07:05.026873 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:05.026837    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f025a72f-bdab-4de7-b958-ddda7786c4fa-tmp-dir\") pod \"e2e-conflict-b-bb3c1a9d-kserve-6d468cf4c6-tmthf\" (UID: \"f025a72f-bdab-4de7-b958-ddda7786c4fa\") " pod="e2e-models-e2e-worker-w4-c1aebc/e2e-conflict-b-bb3c1a9d-kserve-6d468cf4c6-tmthf"
Sep 11 19:07:05.026987 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:05.026900    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f025a72f-bdab-4de7-b958-ddda7786c4fa-tls-certs\") pod \"e2e-conflict-b-bb3c1a9d-kserve-6d468cf4c6-tmthf\" (UID: \"f025a72f-bdab-4de7-b958-ddda7786c4fa\") " pod="e2e-models-e2e-worker-w4-c1aebc/e2e-conflict-b-bb3c1a9d-kserve-6d468cf4c6-tmthf"
Sep 11 19:07:05.026987 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:05.026938    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f025a72f-bdab-4de7-b958-ddda7786c4fa-home\") pod \"e2e-conflict-b-bb3c1a9d-kserve-6d468cf4c6-tmthf\" (UID: \"f025a72f-bdab-4de7-b958-ddda7786c4fa\") " pod="e2e-models-e2e-worker-w4-c1aebc/e2e-conflict-b-bb3c1a9d-kserve-6d468cf4c6-tmthf"
Sep 11 19:07:05.128298 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:05.128259    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f025a72f-bdab-4de7-b958-ddda7786c4fa-dshm\") pod \"e2e-conflict-b-bb3c1a9d-kserve-6d468cf4c6-tmthf\" (UID: \"f025a72f-bdab-4de7-b958-ddda7786c4fa\") " pod="e2e-models-e2e-worker-w4-c1aebc/e2e-conflict-b-bb3c1a9d-kserve-6d468cf4c6-tmthf"
Sep 11 19:07:05.128298 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:05.128299    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f025a72f-bdab-4de7-b958-ddda7786c4fa-model-cache\") pod \"e2e-conflict-b-bb3c1a9d-kserve-6d468cf4c6-tmthf\" (UID: \"f025a72f-bdab-4de7-b958-ddda7786c4fa\") " pod="e2e-models-e2e-worker-w4-c1aebc/e2e-conflict-b-bb3c1a9d-kserve-6d468cf4c6-tmthf"
Sep 11 19:07:05.128528 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:05.128321    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f025a72f-bdab-4de7-b958-ddda7786c4fa-tmp-dir\") pod \"e2e-conflict-b-bb3c1a9d-kserve-6d468cf4c6-tmthf\" (UID: \"f025a72f-bdab-4de7-b958-ddda7786c4fa\") " pod="e2e-models-e2e-worker-w4-c1aebc/e2e-conflict-b-bb3c1a9d-kserve-6d468cf4c6-tmthf"
Sep 11 19:07:05.128528 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:05.128363    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f025a72f-bdab-4de7-b958-ddda7786c4fa-tls-certs\") pod \"e2e-conflict-b-bb3c1a9d-kserve-6d468cf4c6-tmthf\" (UID: \"f025a72f-bdab-4de7-b958-ddda7786c4fa\") " pod="e2e-models-e2e-worker-w4-c1aebc/e2e-conflict-b-bb3c1a9d-kserve-6d468cf4c6-tmthf"
Sep 11 19:07:05.128528 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:05.128408    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f025a72f-bdab-4de7-b958-ddda7786c4fa-home\") pod \"e2e-conflict-b-bb3c1a9d-kserve-6d468cf4c6-tmthf\" (UID: \"f025a72f-bdab-4de7-b958-ddda7786c4fa\") " pod="e2e-models-e2e-worker-w4-c1aebc/e2e-conflict-b-bb3c1a9d-kserve-6d468cf4c6-tmthf"
Sep 11 19:07:05.128528 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:05.128461    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpqx7\" (UniqueName: \"kubernetes.io/projected/f025a72f-bdab-4de7-b958-ddda7786c4fa-kube-api-access-jpqx7\") pod \"e2e-conflict-b-bb3c1a9d-kserve-6d468cf4c6-tmthf\" (UID: \"f025a72f-bdab-4de7-b958-ddda7786c4fa\") " pod="e2e-models-e2e-worker-w4-c1aebc/e2e-conflict-b-bb3c1a9d-kserve-6d468cf4c6-tmthf"
Sep 11 19:07:05.128871 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:05.128824    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f025a72f-bdab-4de7-b958-ddda7786c4fa-tmp-dir\") pod \"e2e-conflict-b-bb3c1a9d-kserve-6d468cf4c6-tmthf\" (UID: \"f025a72f-bdab-4de7-b958-ddda7786c4fa\") " pod="e2e-models-e2e-worker-w4-c1aebc/e2e-conflict-b-bb3c1a9d-kserve-6d468cf4c6-tmthf"
Sep 11 19:07:05.128871 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:05.128844    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f025a72f-bdab-4de7-b958-ddda7786c4fa-model-cache\") pod \"e2e-conflict-b-bb3c1a9d-kserve-6d468cf4c6-tmthf\" (UID: \"f025a72f-bdab-4de7-b958-ddda7786c4fa\") " pod="e2e-models-e2e-worker-w4-c1aebc/e2e-conflict-b-bb3c1a9d-kserve-6d468cf4c6-tmthf"
Sep 11 19:07:05.129042 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:05.128882    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f025a72f-bdab-4de7-b958-ddda7786c4fa-home\") pod \"e2e-conflict-b-bb3c1a9d-kserve-6d468cf4c6-tmthf\" (UID: \"f025a72f-bdab-4de7-b958-ddda7786c4fa\") " pod="e2e-models-e2e-worker-w4-c1aebc/e2e-conflict-b-bb3c1a9d-kserve-6d468cf4c6-tmthf"
Sep 11 19:07:05.130851 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:05.130825    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f025a72f-bdab-4de7-b958-ddda7786c4fa-dshm\") pod \"e2e-conflict-b-bb3c1a9d-kserve-6d468cf4c6-tmthf\" (UID: \"f025a72f-bdab-4de7-b958-ddda7786c4fa\") " pod="e2e-models-e2e-worker-w4-c1aebc/e2e-conflict-b-bb3c1a9d-kserve-6d468cf4c6-tmthf"
Sep 11 19:07:05.131269 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:05.131248    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f025a72f-bdab-4de7-b958-ddda7786c4fa-tls-certs\") pod \"e2e-conflict-b-bb3c1a9d-kserve-6d468cf4c6-tmthf\" (UID: \"f025a72f-bdab-4de7-b958-ddda7786c4fa\") " pod="e2e-models-e2e-worker-w4-c1aebc/e2e-conflict-b-bb3c1a9d-kserve-6d468cf4c6-tmthf"
Sep 11 19:07:05.136822 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:05.136801    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpqx7\" (UniqueName: \"kubernetes.io/projected/f025a72f-bdab-4de7-b958-ddda7786c4fa-kube-api-access-jpqx7\") pod \"e2e-conflict-b-bb3c1a9d-kserve-6d468cf4c6-tmthf\" (UID: \"f025a72f-bdab-4de7-b958-ddda7786c4fa\") " pod="e2e-models-e2e-worker-w4-c1aebc/e2e-conflict-b-bb3c1a9d-kserve-6d468cf4c6-tmthf"
Sep 11 19:07:05.235094 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:05.235054    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c1aebc/e2e-conflict-b-bb3c1a9d-kserve-6d468cf4c6-tmthf"
Sep 11 19:07:05.512209 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:05.512093    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c1aebc/e2e-conflict-b-bb3c1a9d-kserve-6d468cf4c6-tmthf"]
Sep 11 19:07:05.514830 ip-10-0-134-189 kubenswrapper[2578]: W0911 19:07:05.514801    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf025a72f_bdab_4de7_b958_ddda7786c4fa.slice/crio-825ba7426ee4d1a5d8371ab21fe165d30fa4fd9ee4de1a18182348a712ec5f87 WatchSource:0}: Error finding container 825ba7426ee4d1a5d8371ab21fe165d30fa4fd9ee4de1a18182348a712ec5f87: Status 404 returned error can't find the container with id 825ba7426ee4d1a5d8371ab21fe165d30fa4fd9ee4de1a18182348a712ec5f87
Sep 11 19:07:06.438104 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:06.438058    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c1aebc/e2e-conflict-b-bb3c1a9d-kserve-6d468cf4c6-tmthf" event={"ID":"f025a72f-bdab-4de7-b958-ddda7786c4fa","Type":"ContainerStarted","Data":"c4b2100328493c2106c3d34d490a52ddd64b36060438af794cf61aaefd0de9f1"}
Sep 11 19:07:06.438507 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:06.438114    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c1aebc/e2e-conflict-b-bb3c1a9d-kserve-6d468cf4c6-tmthf" event={"ID":"f025a72f-bdab-4de7-b958-ddda7786c4fa","Type":"ContainerStarted","Data":"825ba7426ee4d1a5d8371ab21fe165d30fa4fd9ee4de1a18182348a712ec5f87"}
Sep 11 19:07:06.456368 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:06.456317    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-c1aebc/e2e-conflict-b-bb3c1a9d-kserve-6d468cf4c6-tmthf" podStartSLOduration=2.456300742 podStartE2EDuration="2.456300742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:07:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:07:06.455482246 +0000 UTC m=+1102.533072821" watchObservedRunningTime="2026-09-11 19:07:06.456300742 +0000 UTC m=+1102.533891294"
Sep 11 19:07:08.381891 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:08.381856    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-093f6c-openshift-default-b5fc96998-8cgj5"]
Sep 11 19:07:08.382427 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:08.382100    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-093f6c-openshift-default-b5fc96998-8cgj5" podUID="2745e9fb-5682-4fdf-8394-d98e50ebd7a7" containerName="istio-proxy" containerID="cri-o://858f77a5fadd28954d7c37746eed0d18af56c4ee0aed9e4b4b73220c5f0f7056" gracePeriod=30
Sep 11 19:07:13.638399 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:13.638368    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-093f6c-openshift-default-b5fc96998-8cgj5"
Sep 11 19:07:13.714549 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:13.714514    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2745e9fb-5682-4fdf-8394-d98e50ebd7a7-istio-podinfo\") pod \"2745e9fb-5682-4fdf-8394-d98e50ebd7a7\" (UID: \"2745e9fb-5682-4fdf-8394-d98e50ebd7a7\") "
Sep 11 19:07:13.714771 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:13.714569    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2745e9fb-5682-4fdf-8394-d98e50ebd7a7-credential-socket\") pod \"2745e9fb-5682-4fdf-8394-d98e50ebd7a7\" (UID: \"2745e9fb-5682-4fdf-8394-d98e50ebd7a7\") "
Sep 11 19:07:13.714771 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:13.714655    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2745e9fb-5682-4fdf-8394-d98e50ebd7a7-workload-socket\") pod \"2745e9fb-5682-4fdf-8394-d98e50ebd7a7\" (UID: \"2745e9fb-5682-4fdf-8394-d98e50ebd7a7\") "
Sep 11 19:07:13.714771 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:13.714683    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2745e9fb-5682-4fdf-8394-d98e50ebd7a7-istio-token\") pod \"2745e9fb-5682-4fdf-8394-d98e50ebd7a7\" (UID: \"2745e9fb-5682-4fdf-8394-d98e50ebd7a7\") "
Sep 11 19:07:13.714937 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:13.714784    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2745e9fb-5682-4fdf-8394-d98e50ebd7a7-workload-certs\") pod \"2745e9fb-5682-4fdf-8394-d98e50ebd7a7\" (UID: \"2745e9fb-5682-4fdf-8394-d98e50ebd7a7\") "
Sep 11 19:07:13.714937 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:13.714834    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c6g2w\" (UniqueName: \"kubernetes.io/projected/2745e9fb-5682-4fdf-8394-d98e50ebd7a7-kube-api-access-c6g2w\") pod \"2745e9fb-5682-4fdf-8394-d98e50ebd7a7\" (UID: \"2745e9fb-5682-4fdf-8394-d98e50ebd7a7\") "
Sep 11 19:07:13.714937 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:13.714919    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2745e9fb-5682-4fdf-8394-d98e50ebd7a7-istio-data\") pod \"2745e9fb-5682-4fdf-8394-d98e50ebd7a7\" (UID: \"2745e9fb-5682-4fdf-8394-d98e50ebd7a7\") "
Sep 11 19:07:13.715087 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:13.714932    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2745e9fb-5682-4fdf-8394-d98e50ebd7a7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2745e9fb-5682-4fdf-8394-d98e50ebd7a7" (UID: "2745e9fb-5682-4fdf-8394-d98e50ebd7a7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:07:13.715087 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:13.714945    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2745e9fb-5682-4fdf-8394-d98e50ebd7a7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2745e9fb-5682-4fdf-8394-d98e50ebd7a7" (UID: "2745e9fb-5682-4fdf-8394-d98e50ebd7a7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:07:13.715087 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:13.715017    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2745e9fb-5682-4fdf-8394-d98e50ebd7a7-istio-envoy\") pod \"2745e9fb-5682-4fdf-8394-d98e50ebd7a7\" (UID: \"2745e9fb-5682-4fdf-8394-d98e50ebd7a7\") "
Sep 11 19:07:13.715087 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:13.715031    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2745e9fb-5682-4fdf-8394-d98e50ebd7a7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2745e9fb-5682-4fdf-8394-d98e50ebd7a7" (UID: "2745e9fb-5682-4fdf-8394-d98e50ebd7a7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:07:13.715087 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:13.715051    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2745e9fb-5682-4fdf-8394-d98e50ebd7a7-istiod-ca-cert\") pod \"2745e9fb-5682-4fdf-8394-d98e50ebd7a7\" (UID: \"2745e9fb-5682-4fdf-8394-d98e50ebd7a7\") "
Sep 11 19:07:13.715323 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:13.715214    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2745e9fb-5682-4fdf-8394-d98e50ebd7a7-istio-data" (OuterVolumeSpecName: "istio-data") pod "2745e9fb-5682-4fdf-8394-d98e50ebd7a7" (UID: "2745e9fb-5682-4fdf-8394-d98e50ebd7a7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:07:13.715377 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:13.715318    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2745e9fb-5682-4fdf-8394-d98e50ebd7a7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2745e9fb-5682-4fdf-8394-d98e50ebd7a7" (UID: "2745e9fb-5682-4fdf-8394-d98e50ebd7a7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 19:07:13.715436 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:13.715417    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2745e9fb-5682-4fdf-8394-d98e50ebd7a7-istiod-ca-cert\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:07:13.715488 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:13.715444    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2745e9fb-5682-4fdf-8394-d98e50ebd7a7-credential-socket\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:07:13.715488 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:13.715460    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2745e9fb-5682-4fdf-8394-d98e50ebd7a7-workload-socket\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:07:13.715565 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:13.715474    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2745e9fb-5682-4fdf-8394-d98e50ebd7a7-workload-certs\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:07:13.715646 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:13.715554    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2745e9fb-5682-4fdf-8394-d98e50ebd7a7-istio-data\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:07:13.717176 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:13.717151    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2745e9fb-5682-4fdf-8394-d98e50ebd7a7-kube-api-access-c6g2w" (OuterVolumeSpecName: "kube-api-access-c6g2w") pod "2745e9fb-5682-4fdf-8394-d98e50ebd7a7" (UID: "2745e9fb-5682-4fdf-8394-d98e50ebd7a7"). InnerVolumeSpecName "kube-api-access-c6g2w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:07:13.717176 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:13.717160    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2745e9fb-5682-4fdf-8394-d98e50ebd7a7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2745e9fb-5682-4fdf-8394-d98e50ebd7a7" (UID: "2745e9fb-5682-4fdf-8394-d98e50ebd7a7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:07:13.717409 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:13.717392    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2745e9fb-5682-4fdf-8394-d98e50ebd7a7-istio-token" (OuterVolumeSpecName: "istio-token") pod "2745e9fb-5682-4fdf-8394-d98e50ebd7a7" (UID: "2745e9fb-5682-4fdf-8394-d98e50ebd7a7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:07:13.717468 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:13.717442    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2745e9fb-5682-4fdf-8394-d98e50ebd7a7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2745e9fb-5682-4fdf-8394-d98e50ebd7a7" (UID: "2745e9fb-5682-4fdf-8394-d98e50ebd7a7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 19:07:13.817164 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:13.817126    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2745e9fb-5682-4fdf-8394-d98e50ebd7a7-istio-envoy\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:07:13.817164 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:13.817157    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2745e9fb-5682-4fdf-8394-d98e50ebd7a7-istio-podinfo\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:07:13.817164 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:13.817166    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2745e9fb-5682-4fdf-8394-d98e50ebd7a7-istio-token\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:07:13.817164 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:13.817177    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6g2w\" (UniqueName: \"kubernetes.io/projected/2745e9fb-5682-4fdf-8394-d98e50ebd7a7-kube-api-access-c6g2w\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:07:14.429134 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:14.429099    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-37ce96-85fb758d8f-wqfhk"
Sep 11 19:07:14.481016 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:14.480981    2578 generic.go:356] "Generic (PLEG): container finished" podID="2745e9fb-5682-4fdf-8394-d98e50ebd7a7" containerID="858f77a5fadd28954d7c37746eed0d18af56c4ee0aed9e4b4b73220c5f0f7056" exitCode=0
Sep 11 19:07:14.481212 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:14.481034    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-093f6c-openshift-default-b5fc96998-8cgj5" event={"ID":"2745e9fb-5682-4fdf-8394-d98e50ebd7a7","Type":"ContainerDied","Data":"858f77a5fadd28954d7c37746eed0d18af56c4ee0aed9e4b4b73220c5f0f7056"}
Sep 11 19:07:14.481212 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:14.481046    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-093f6c-openshift-default-b5fc96998-8cgj5"
Sep 11 19:07:14.481212 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:14.481090    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-093f6c-openshift-default-b5fc96998-8cgj5" event={"ID":"2745e9fb-5682-4fdf-8394-d98e50ebd7a7","Type":"ContainerDied","Data":"64e99b9afcfa9770fa7d95f35597f14ca13ec2f4a70a7d487421bf8bbfe0e907"}
Sep 11 19:07:14.481212 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:14.481120    2578 scope.go:117] "RemoveContainer" containerID="858f77a5fadd28954d7c37746eed0d18af56c4ee0aed9e4b4b73220c5f0f7056"
Sep 11 19:07:14.491888 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:14.491859    2578 scope.go:117] "RemoveContainer" containerID="3bc562a4e47e6d79a7524562f2b7cf0021605ea74d71141462f17be5665093ab"
Sep 11 19:07:14.500546 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:14.500512    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-093f6c-openshift-default-b5fc96998-8cgj5"]
Sep 11 19:07:14.502750 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:14.502481    2578 scope.go:117] "RemoveContainer" containerID="858f77a5fadd28954d7c37746eed0d18af56c4ee0aed9e4b4b73220c5f0f7056"
Sep 11 19:07:14.502984 ip-10-0-134-189 kubenswrapper[2578]: E0911 19:07:14.502936    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"858f77a5fadd28954d7c37746eed0d18af56c4ee0aed9e4b4b73220c5f0f7056\": container with ID starting with 858f77a5fadd28954d7c37746eed0d18af56c4ee0aed9e4b4b73220c5f0f7056 not found: ID does not exist" containerID="858f77a5fadd28954d7c37746eed0d18af56c4ee0aed9e4b4b73220c5f0f7056"
Sep 11 19:07:14.503077 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:14.502981    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"858f77a5fadd28954d7c37746eed0d18af56c4ee0aed9e4b4b73220c5f0f7056"} err="failed to get container status \"858f77a5fadd28954d7c37746eed0d18af56c4ee0aed9e4b4b73220c5f0f7056\": rpc error: code = NotFound desc = could not find container \"858f77a5fadd28954d7c37746eed0d18af56c4ee0aed9e4b4b73220c5f0f7056\": container with ID starting with 858f77a5fadd28954d7c37746eed0d18af56c4ee0aed9e4b4b73220c5f0f7056 not found: ID does not exist"
Sep 11 19:07:14.503077 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:14.503007    2578 scope.go:117] "RemoveContainer" containerID="3bc562a4e47e6d79a7524562f2b7cf0021605ea74d71141462f17be5665093ab"
Sep 11 19:07:14.503341 ip-10-0-134-189 kubenswrapper[2578]: E0911 19:07:14.503316    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bc562a4e47e6d79a7524562f2b7cf0021605ea74d71141462f17be5665093ab\": container with ID starting with 3bc562a4e47e6d79a7524562f2b7cf0021605ea74d71141462f17be5665093ab not found: ID does not exist" containerID="3bc562a4e47e6d79a7524562f2b7cf0021605ea74d71141462f17be5665093ab"
Sep 11 19:07:14.503418 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:14.503352    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bc562a4e47e6d79a7524562f2b7cf0021605ea74d71141462f17be5665093ab"} err="failed to get container status \"3bc562a4e47e6d79a7524562f2b7cf0021605ea74d71141462f17be5665093ab\": rpc error: code = NotFound desc = could not find container \"3bc562a4e47e6d79a7524562f2b7cf0021605ea74d71141462f17be5665093ab\": container with ID starting with 3bc562a4e47e6d79a7524562f2b7cf0021605ea74d71141462f17be5665093ab not found: ID does not exist"
Sep 11 19:07:14.504641 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:14.504625    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-093f6c-openshift-default-b5fc96998-8cgj5"]
Sep 11 19:07:15.236184 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:15.236145    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c1aebc/e2e-conflict-b-bb3c1a9d-kserve-6d468cf4c6-tmthf"
Sep 11 19:07:15.236626 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:15.236202    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-c1aebc/e2e-conflict-b-bb3c1a9d-kserve-6d468cf4c6-tmthf"
Sep 11 19:07:15.248967 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:15.248940    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-c1aebc/e2e-conflict-b-bb3c1a9d-kserve-6d468cf4c6-tmthf"
Sep 11 19:07:15.498508 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:15.498416    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c1aebc/e2e-conflict-b-bb3c1a9d-kserve-6d468cf4c6-tmthf"
Sep 11 19:07:16.397418 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:16.397382    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2745e9fb-5682-4fdf-8394-d98e50ebd7a7" path="/var/lib/kubelet/pods/2745e9fb-5682-4fdf-8394-d98e50ebd7a7/volumes"
Sep 11 19:07:17.897830 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:17.897791    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8280eb-5c759788c8-kgjp2"]
Sep 11 19:07:17.898220 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:17.898153    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2745e9fb-5682-4fdf-8394-d98e50ebd7a7" containerName="istio-proxy"
Sep 11 19:07:17.898220 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:17.898169    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2745e9fb-5682-4fdf-8394-d98e50ebd7a7" containerName="istio-proxy"
Sep 11 19:07:17.898294 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:17.898256    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2745e9fb-5682-4fdf-8394-d98e50ebd7a7" containerName="istio-proxy"
Sep 11 19:07:17.898294 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:17.898265    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2745e9fb-5682-4fdf-8394-d98e50ebd7a7" containerName="istio-proxy"
Sep 11 19:07:17.902967 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:17.902947    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8280eb-5c759788c8-kgjp2"
Sep 11 19:07:17.905804 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:17.905780    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-8280eb\""
Sep 11 19:07:17.905939 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:17.905785    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-8280eb-dockercfg-mskwn\""
Sep 11 19:07:17.914435 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:17.914412    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8280eb-5c759788c8-kgjp2"]
Sep 11 19:07:18.060144 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:18.060105    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdxrk\" (UniqueName: \"kubernetes.io/projected/4b3672a7-8dfe-47e5-b418-48bb90d73622-kube-api-access-vdxrk\") pod \"payload-pre-processing-e2e-shared-a-w3-8280eb-5c759788c8-kgjp2\" (UID: \"4b3672a7-8dfe-47e5-b418-48bb90d73622\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8280eb-5c759788c8-kgjp2"
Sep 11 19:07:18.060144 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:18.060151    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b3672a7-8dfe-47e5-b418-48bb90d73622-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-8280eb-5c759788c8-kgjp2\" (UID: \"4b3672a7-8dfe-47e5-b418-48bb90d73622\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8280eb-5c759788c8-kgjp2"
Sep 11 19:07:18.161514 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:18.161407    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdxrk\" (UniqueName: \"kubernetes.io/projected/4b3672a7-8dfe-47e5-b418-48bb90d73622-kube-api-access-vdxrk\") pod \"payload-pre-processing-e2e-shared-a-w3-8280eb-5c759788c8-kgjp2\" (UID: \"4b3672a7-8dfe-47e5-b418-48bb90d73622\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8280eb-5c759788c8-kgjp2"
Sep 11 19:07:18.161514 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:18.161460    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b3672a7-8dfe-47e5-b418-48bb90d73622-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-8280eb-5c759788c8-kgjp2\" (UID: \"4b3672a7-8dfe-47e5-b418-48bb90d73622\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8280eb-5c759788c8-kgjp2"
Sep 11 19:07:18.162263 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:18.162241    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b3672a7-8dfe-47e5-b418-48bb90d73622-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-8280eb-5c759788c8-kgjp2\" (UID: \"4b3672a7-8dfe-47e5-b418-48bb90d73622\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8280eb-5c759788c8-kgjp2"
Sep 11 19:07:18.170119 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:18.170095    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdxrk\" (UniqueName: \"kubernetes.io/projected/4b3672a7-8dfe-47e5-b418-48bb90d73622-kube-api-access-vdxrk\") pod \"payload-pre-processing-e2e-shared-a-w3-8280eb-5c759788c8-kgjp2\" (UID: \"4b3672a7-8dfe-47e5-b418-48bb90d73622\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8280eb-5c759788c8-kgjp2"
Sep 11 19:07:18.213888 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:18.213829    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8280eb-5c759788c8-kgjp2"
Sep 11 19:07:18.493220 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:18.493193    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8280eb-5c759788c8-kgjp2"]
Sep 11 19:07:18.495140 ip-10-0-134-189 kubenswrapper[2578]: W0911 19:07:18.495104    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b3672a7_8dfe_47e5_b418_48bb90d73622.slice/crio-234ede84ea3a62edf00973e2046d2c88afed502577d380da04651b41485b3da4 WatchSource:0}: Error finding container 234ede84ea3a62edf00973e2046d2c88afed502577d380da04651b41485b3da4: Status 404 returned error can't find the container with id 234ede84ea3a62edf00973e2046d2c88afed502577d380da04651b41485b3da4
Sep 11 19:07:18.502702 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:18.502664    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8280eb-5c759788c8-kgjp2" event={"ID":"4b3672a7-8dfe-47e5-b418-48bb90d73622","Type":"ContainerStarted","Data":"234ede84ea3a62edf00973e2046d2c88afed502577d380da04651b41485b3da4"}
Sep 11 19:07:19.509831 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:19.509797    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8280eb-5c759788c8-kgjp2" event={"ID":"4b3672a7-8dfe-47e5-b418-48bb90d73622","Type":"ContainerStarted","Data":"ea97ec975c4959012702854ffc6555679a9552896dc36e8f2d804e46fb79bce1"}
Sep 11 19:07:19.510352 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:19.509930    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8280eb-5c759788c8-kgjp2"
Sep 11 19:07:19.531233 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:19.531163    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8280eb-5c759788c8-kgjp2" podStartSLOduration=2.531139361 podStartE2EDuration="2.531139361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:07:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:07:19.529232777 +0000 UTC m=+1115.606823357" watchObservedRunningTime="2026-09-11 19:07:19.531139361 +0000 UTC m=+1115.608729918"
Sep 11 19:07:30.518012 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:30.517979    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8280eb-5c759788c8-kgjp2"
Sep 11 19:07:38.242684 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:38.242649    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-3b7391-openshift-default-7cd68cb6bd-kwtt7"]
Sep 11 19:07:38.243085 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:38.243041    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2745e9fb-5682-4fdf-8394-d98e50ebd7a7" containerName="istio-proxy"
Sep 11 19:07:38.243085 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:38.243054    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2745e9fb-5682-4fdf-8394-d98e50ebd7a7" containerName="istio-proxy"
Sep 11 19:07:38.246354 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:38.246337    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-3b7391-openshift-default-7cd68cb6bd-kwtt7"
Sep 11 19:07:38.249881 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:38.249854    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-3b7391-openshift-default-dockercfg-wx4xt\""
Sep 11 19:07:38.267614 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:38.267565    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-3b7391-openshift-default-7cd68cb6bd-kwtt7"]
Sep 11 19:07:38.341042 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:38.341002    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59wr7\" (UniqueName: \"kubernetes.io/projected/0eff5cf7-789d-4dbf-9894-568df3b2ba47-kube-api-access-59wr7\") pod \"e2e-shared-b-w3-3b7391-openshift-default-7cd68cb6bd-kwtt7\" (UID: \"0eff5cf7-789d-4dbf-9894-568df3b2ba47\") " pod="openshift-ingress/e2e-shared-b-w3-3b7391-openshift-default-7cd68cb6bd-kwtt7"
Sep 11 19:07:38.341042 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:38.341068    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0eff5cf7-789d-4dbf-9894-568df3b2ba47-istiod-ca-cert\") pod \"e2e-shared-b-w3-3b7391-openshift-default-7cd68cb6bd-kwtt7\" (UID: \"0eff5cf7-789d-4dbf-9894-568df3b2ba47\") " pod="openshift-ingress/e2e-shared-b-w3-3b7391-openshift-default-7cd68cb6bd-kwtt7"
Sep 11 19:07:38.341426 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:38.341154    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0eff5cf7-789d-4dbf-9894-568df3b2ba47-workload-socket\") pod \"e2e-shared-b-w3-3b7391-openshift-default-7cd68cb6bd-kwtt7\" (UID: \"0eff5cf7-789d-4dbf-9894-568df3b2ba47\") " pod="openshift-ingress/e2e-shared-b-w3-3b7391-openshift-default-7cd68cb6bd-kwtt7"
Sep 11 19:07:38.341426 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:38.341196    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0eff5cf7-789d-4dbf-9894-568df3b2ba47-workload-certs\") pod \"e2e-shared-b-w3-3b7391-openshift-default-7cd68cb6bd-kwtt7\" (UID: \"0eff5cf7-789d-4dbf-9894-568df3b2ba47\") " pod="openshift-ingress/e2e-shared-b-w3-3b7391-openshift-default-7cd68cb6bd-kwtt7"
Sep 11 19:07:38.341426 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:38.341259    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0eff5cf7-789d-4dbf-9894-568df3b2ba47-istio-envoy\") pod \"e2e-shared-b-w3-3b7391-openshift-default-7cd68cb6bd-kwtt7\" (UID: \"0eff5cf7-789d-4dbf-9894-568df3b2ba47\") " pod="openshift-ingress/e2e-shared-b-w3-3b7391-openshift-default-7cd68cb6bd-kwtt7"
Sep 11 19:07:38.341426 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:38.341299    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0eff5cf7-789d-4dbf-9894-568df3b2ba47-istio-token\") pod \"e2e-shared-b-w3-3b7391-openshift-default-7cd68cb6bd-kwtt7\" (UID: \"0eff5cf7-789d-4dbf-9894-568df3b2ba47\") " pod="openshift-ingress/e2e-shared-b-w3-3b7391-openshift-default-7cd68cb6bd-kwtt7"
Sep 11 19:07:38.341426 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:38.341400    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0eff5cf7-789d-4dbf-9894-568df3b2ba47-istio-data\") pod \"e2e-shared-b-w3-3b7391-openshift-default-7cd68cb6bd-kwtt7\" (UID: \"0eff5cf7-789d-4dbf-9894-568df3b2ba47\") " pod="openshift-ingress/e2e-shared-b-w3-3b7391-openshift-default-7cd68cb6bd-kwtt7"
Sep 11 19:07:38.341710 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:38.341492    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0eff5cf7-789d-4dbf-9894-568df3b2ba47-istio-podinfo\") pod \"e2e-shared-b-w3-3b7391-openshift-default-7cd68cb6bd-kwtt7\" (UID: \"0eff5cf7-789d-4dbf-9894-568df3b2ba47\") " pod="openshift-ingress/e2e-shared-b-w3-3b7391-openshift-default-7cd68cb6bd-kwtt7"
Sep 11 19:07:38.341710 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:38.341531    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0eff5cf7-789d-4dbf-9894-568df3b2ba47-credential-socket\") pod \"e2e-shared-b-w3-3b7391-openshift-default-7cd68cb6bd-kwtt7\" (UID: \"0eff5cf7-789d-4dbf-9894-568df3b2ba47\") " pod="openshift-ingress/e2e-shared-b-w3-3b7391-openshift-default-7cd68cb6bd-kwtt7"
Sep 11 19:07:38.443010 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:38.442970    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0eff5cf7-789d-4dbf-9894-568df3b2ba47-istio-data\") pod \"e2e-shared-b-w3-3b7391-openshift-default-7cd68cb6bd-kwtt7\" (UID: \"0eff5cf7-789d-4dbf-9894-568df3b2ba47\") " pod="openshift-ingress/e2e-shared-b-w3-3b7391-openshift-default-7cd68cb6bd-kwtt7"
Sep 11 19:07:38.443010 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:38.443016    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0eff5cf7-789d-4dbf-9894-568df3b2ba47-istio-podinfo\") pod \"e2e-shared-b-w3-3b7391-openshift-default-7cd68cb6bd-kwtt7\" (UID: \"0eff5cf7-789d-4dbf-9894-568df3b2ba47\") " pod="openshift-ingress/e2e-shared-b-w3-3b7391-openshift-default-7cd68cb6bd-kwtt7"
Sep 11 19:07:38.443281 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:38.443043    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0eff5cf7-789d-4dbf-9894-568df3b2ba47-credential-socket\") pod \"e2e-shared-b-w3-3b7391-openshift-default-7cd68cb6bd-kwtt7\" (UID: \"0eff5cf7-789d-4dbf-9894-568df3b2ba47\") " pod="openshift-ingress/e2e-shared-b-w3-3b7391-openshift-default-7cd68cb6bd-kwtt7"
Sep 11 19:07:38.443281 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:38.443079    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59wr7\" (UniqueName: \"kubernetes.io/projected/0eff5cf7-789d-4dbf-9894-568df3b2ba47-kube-api-access-59wr7\") pod \"e2e-shared-b-w3-3b7391-openshift-default-7cd68cb6bd-kwtt7\" (UID: \"0eff5cf7-789d-4dbf-9894-568df3b2ba47\") " pod="openshift-ingress/e2e-shared-b-w3-3b7391-openshift-default-7cd68cb6bd-kwtt7"
Sep 11 19:07:38.443281 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:38.443110    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0eff5cf7-789d-4dbf-9894-568df3b2ba47-istiod-ca-cert\") pod \"e2e-shared-b-w3-3b7391-openshift-default-7cd68cb6bd-kwtt7\" (UID: \"0eff5cf7-789d-4dbf-9894-568df3b2ba47\") " pod="openshift-ingress/e2e-shared-b-w3-3b7391-openshift-default-7cd68cb6bd-kwtt7"
Sep 11 19:07:38.443281 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:38.443142    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0eff5cf7-789d-4dbf-9894-568df3b2ba47-workload-socket\") pod \"e2e-shared-b-w3-3b7391-openshift-default-7cd68cb6bd-kwtt7\" (UID: \"0eff5cf7-789d-4dbf-9894-568df3b2ba47\") " pod="openshift-ingress/e2e-shared-b-w3-3b7391-openshift-default-7cd68cb6bd-kwtt7"
Sep 11 19:07:38.443488 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:38.443273    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0eff5cf7-789d-4dbf-9894-568df3b2ba47-workload-certs\") pod \"e2e-shared-b-w3-3b7391-openshift-default-7cd68cb6bd-kwtt7\" (UID: \"0eff5cf7-789d-4dbf-9894-568df3b2ba47\") " pod="openshift-ingress/e2e-shared-b-w3-3b7391-openshift-default-7cd68cb6bd-kwtt7"
Sep 11 19:07:38.443488 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:38.443330    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0eff5cf7-789d-4dbf-9894-568df3b2ba47-istio-envoy\") pod \"e2e-shared-b-w3-3b7391-openshift-default-7cd68cb6bd-kwtt7\" (UID: \"0eff5cf7-789d-4dbf-9894-568df3b2ba47\") " pod="openshift-ingress/e2e-shared-b-w3-3b7391-openshift-default-7cd68cb6bd-kwtt7"
Sep 11 19:07:38.443488 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:38.443364    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0eff5cf7-789d-4dbf-9894-568df3b2ba47-istio-token\") pod \"e2e-shared-b-w3-3b7391-openshift-default-7cd68cb6bd-kwtt7\" (UID: \"0eff5cf7-789d-4dbf-9894-568df3b2ba47\") " pod="openshift-ingress/e2e-shared-b-w3-3b7391-openshift-default-7cd68cb6bd-kwtt7"
Sep 11 19:07:38.443488 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:38.443431    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0eff5cf7-789d-4dbf-9894-568df3b2ba47-istio-data\") pod \"e2e-shared-b-w3-3b7391-openshift-default-7cd68cb6bd-kwtt7\" (UID: \"0eff5cf7-789d-4dbf-9894-568df3b2ba47\") " pod="openshift-ingress/e2e-shared-b-w3-3b7391-openshift-default-7cd68cb6bd-kwtt7"
Sep 11 19:07:38.443727 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:38.443577    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0eff5cf7-789d-4dbf-9894-568df3b2ba47-workload-socket\") pod \"e2e-shared-b-w3-3b7391-openshift-default-7cd68cb6bd-kwtt7\" (UID: \"0eff5cf7-789d-4dbf-9894-568df3b2ba47\") " pod="openshift-ingress/e2e-shared-b-w3-3b7391-openshift-default-7cd68cb6bd-kwtt7"
Sep 11 19:07:38.443727 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:38.443634    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0eff5cf7-789d-4dbf-9894-568df3b2ba47-credential-socket\") pod \"e2e-shared-b-w3-3b7391-openshift-default-7cd68cb6bd-kwtt7\" (UID: \"0eff5cf7-789d-4dbf-9894-568df3b2ba47\") " pod="openshift-ingress/e2e-shared-b-w3-3b7391-openshift-default-7cd68cb6bd-kwtt7"
Sep 11 19:07:38.443727 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:38.443653    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0eff5cf7-789d-4dbf-9894-568df3b2ba47-workload-certs\") pod \"e2e-shared-b-w3-3b7391-openshift-default-7cd68cb6bd-kwtt7\" (UID: \"0eff5cf7-789d-4dbf-9894-568df3b2ba47\") " pod="openshift-ingress/e2e-shared-b-w3-3b7391-openshift-default-7cd68cb6bd-kwtt7"
Sep 11 19:07:38.444249 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:38.444224    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0eff5cf7-789d-4dbf-9894-568df3b2ba47-istiod-ca-cert\") pod \"e2e-shared-b-w3-3b7391-openshift-default-7cd68cb6bd-kwtt7\" (UID: \"0eff5cf7-789d-4dbf-9894-568df3b2ba47\") " pod="openshift-ingress/e2e-shared-b-w3-3b7391-openshift-default-7cd68cb6bd-kwtt7"
Sep 11 19:07:38.445756 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:38.445734    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0eff5cf7-789d-4dbf-9894-568df3b2ba47-istio-envoy\") pod \"e2e-shared-b-w3-3b7391-openshift-default-7cd68cb6bd-kwtt7\" (UID: \"0eff5cf7-789d-4dbf-9894-568df3b2ba47\") " pod="openshift-ingress/e2e-shared-b-w3-3b7391-openshift-default-7cd68cb6bd-kwtt7"
Sep 11 19:07:38.445958 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:38.445937    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0eff5cf7-789d-4dbf-9894-568df3b2ba47-istio-podinfo\") pod \"e2e-shared-b-w3-3b7391-openshift-default-7cd68cb6bd-kwtt7\" (UID: \"0eff5cf7-789d-4dbf-9894-568df3b2ba47\") " pod="openshift-ingress/e2e-shared-b-w3-3b7391-openshift-default-7cd68cb6bd-kwtt7"
Sep 11 19:07:38.451352 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:38.451319    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0eff5cf7-789d-4dbf-9894-568df3b2ba47-istio-token\") pod \"e2e-shared-b-w3-3b7391-openshift-default-7cd68cb6bd-kwtt7\" (UID: \"0eff5cf7-789d-4dbf-9894-568df3b2ba47\") " pod="openshift-ingress/e2e-shared-b-w3-3b7391-openshift-default-7cd68cb6bd-kwtt7"
Sep 11 19:07:38.452203 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:38.452184    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59wr7\" (UniqueName: \"kubernetes.io/projected/0eff5cf7-789d-4dbf-9894-568df3b2ba47-kube-api-access-59wr7\") pod \"e2e-shared-b-w3-3b7391-openshift-default-7cd68cb6bd-kwtt7\" (UID: \"0eff5cf7-789d-4dbf-9894-568df3b2ba47\") " pod="openshift-ingress/e2e-shared-b-w3-3b7391-openshift-default-7cd68cb6bd-kwtt7"
Sep 11 19:07:38.557226 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:38.557117    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-3b7391-openshift-default-7cd68cb6bd-kwtt7"
Sep 11 19:07:38.839872 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:38.839846    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-3b7391-openshift-default-7cd68cb6bd-kwtt7"]
Sep 11 19:07:38.842048 ip-10-0-134-189 kubenswrapper[2578]: W0911 19:07:38.842023    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0eff5cf7_789d_4dbf_9894_568df3b2ba47.slice/crio-09babab9da5ac7c41f9126300f1a457567f43559d14485784f2912318e77eb3a WatchSource:0}: Error finding container 09babab9da5ac7c41f9126300f1a457567f43559d14485784f2912318e77eb3a: Status 404 returned error can't find the container with id 09babab9da5ac7c41f9126300f1a457567f43559d14485784f2912318e77eb3a
Sep 11 19:07:38.844928 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:38.844891    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 19:07:38.845042 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:38.844961    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 19:07:38.845042 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:38.844989    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 19:07:38.876239 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:38.876211    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7ad53a9d-8c87b79f4-hzpqk"]
Sep 11 19:07:38.880936 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:38.880917    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7ad53a9d-8c87b79f4-hzpqk"
Sep 11 19:07:38.883374 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:38.883345    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-7ad53a9d-dockercfg-qs2kj\""
Sep 11 19:07:38.883509 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:38.883492    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-7ad53a9d\""
Sep 11 19:07:38.894394 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:38.894329    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7ad53a9d-8c87b79f4-hzpqk"]
Sep 11 19:07:38.947293 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:38.947243    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c67d229b-1c73-44f8-953b-8f4e79a6e3a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7ad53a9d-8c87b79f4-hzpqk\" (UID: \"c67d229b-1c73-44f8-953b-8f4e79a6e3a1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7ad53a9d-8c87b79f4-hzpqk"
Sep 11 19:07:38.947480 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:38.947415    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8qj7\" (UniqueName: \"kubernetes.io/projected/c67d229b-1c73-44f8-953b-8f4e79a6e3a1-kube-api-access-b8qj7\") pod \"payload-pre-processing-e2e-mt-7ad53a9d-8c87b79f4-hzpqk\" (UID: \"c67d229b-1c73-44f8-953b-8f4e79a6e3a1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7ad53a9d-8c87b79f4-hzpqk"
Sep 11 19:07:38.962397 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:38.962358    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7ad53a9d-57cd7b475c-mzpjg"]
Sep 11 19:07:38.966074 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:38.966046    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7ad53a9d-57cd7b475c-mzpjg"
Sep 11 19:07:38.977737 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:38.977710    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7ad53a9d-57cd7b475c-mzpjg"]
Sep 11 19:07:39.048085 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:39.048046    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c67d229b-1c73-44f8-953b-8f4e79a6e3a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7ad53a9d-8c87b79f4-hzpqk\" (UID: \"c67d229b-1c73-44f8-953b-8f4e79a6e3a1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7ad53a9d-8c87b79f4-hzpqk"
Sep 11 19:07:39.048259 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:39.048101    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/570f6c2f-c791-4e27-82c1-24b1b4d8b88a-plugins-config-volume\") pod \"payload-processing-e2e-mt-7ad53a9d-57cd7b475c-mzpjg\" (UID: \"570f6c2f-c791-4e27-82c1-24b1b4d8b88a\") " pod="openshift-ingress/payload-processing-e2e-mt-7ad53a9d-57cd7b475c-mzpjg"
Sep 11 19:07:39.048259 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:39.048161    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nq94h\" (UniqueName: \"kubernetes.io/projected/570f6c2f-c791-4e27-82c1-24b1b4d8b88a-kube-api-access-nq94h\") pod \"payload-processing-e2e-mt-7ad53a9d-57cd7b475c-mzpjg\" (UID: \"570f6c2f-c791-4e27-82c1-24b1b4d8b88a\") " pod="openshift-ingress/payload-processing-e2e-mt-7ad53a9d-57cd7b475c-mzpjg"
Sep 11 19:07:39.048259 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:39.048239    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8qj7\" (UniqueName: \"kubernetes.io/projected/c67d229b-1c73-44f8-953b-8f4e79a6e3a1-kube-api-access-b8qj7\") pod \"payload-pre-processing-e2e-mt-7ad53a9d-8c87b79f4-hzpqk\" (UID: \"c67d229b-1c73-44f8-953b-8f4e79a6e3a1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7ad53a9d-8c87b79f4-hzpqk"
Sep 11 19:07:39.048945 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:39.048925    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c67d229b-1c73-44f8-953b-8f4e79a6e3a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7ad53a9d-8c87b79f4-hzpqk\" (UID: \"c67d229b-1c73-44f8-953b-8f4e79a6e3a1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7ad53a9d-8c87b79f4-hzpqk"
Sep 11 19:07:39.058024 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:39.057990    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8qj7\" (UniqueName: \"kubernetes.io/projected/c67d229b-1c73-44f8-953b-8f4e79a6e3a1-kube-api-access-b8qj7\") pod \"payload-pre-processing-e2e-mt-7ad53a9d-8c87b79f4-hzpqk\" (UID: \"c67d229b-1c73-44f8-953b-8f4e79a6e3a1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7ad53a9d-8c87b79f4-hzpqk"
Sep 11 19:07:39.149919 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:39.149828    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/570f6c2f-c791-4e27-82c1-24b1b4d8b88a-plugins-config-volume\") pod \"payload-processing-e2e-mt-7ad53a9d-57cd7b475c-mzpjg\" (UID: \"570f6c2f-c791-4e27-82c1-24b1b4d8b88a\") " pod="openshift-ingress/payload-processing-e2e-mt-7ad53a9d-57cd7b475c-mzpjg"
Sep 11 19:07:39.149919 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:39.149895    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq94h\" (UniqueName: \"kubernetes.io/projected/570f6c2f-c791-4e27-82c1-24b1b4d8b88a-kube-api-access-nq94h\") pod \"payload-processing-e2e-mt-7ad53a9d-57cd7b475c-mzpjg\" (UID: \"570f6c2f-c791-4e27-82c1-24b1b4d8b88a\") " pod="openshift-ingress/payload-processing-e2e-mt-7ad53a9d-57cd7b475c-mzpjg"
Sep 11 19:07:39.150553 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:39.150530    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/570f6c2f-c791-4e27-82c1-24b1b4d8b88a-plugins-config-volume\") pod \"payload-processing-e2e-mt-7ad53a9d-57cd7b475c-mzpjg\" (UID: \"570f6c2f-c791-4e27-82c1-24b1b4d8b88a\") " pod="openshift-ingress/payload-processing-e2e-mt-7ad53a9d-57cd7b475c-mzpjg"
Sep 11 19:07:39.161106 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:39.161080    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nq94h\" (UniqueName: \"kubernetes.io/projected/570f6c2f-c791-4e27-82c1-24b1b4d8b88a-kube-api-access-nq94h\") pod \"payload-processing-e2e-mt-7ad53a9d-57cd7b475c-mzpjg\" (UID: \"570f6c2f-c791-4e27-82c1-24b1b4d8b88a\") " pod="openshift-ingress/payload-processing-e2e-mt-7ad53a9d-57cd7b475c-mzpjg"
Sep 11 19:07:39.223846 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:39.223803    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7ad53a9d-8c87b79f4-hzpqk"
Sep 11 19:07:39.278504 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:39.278473    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7ad53a9d-57cd7b475c-mzpjg"
Sep 11 19:07:39.516552 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:39.516519    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7ad53a9d-8c87b79f4-hzpqk"]
Sep 11 19:07:39.517341 ip-10-0-134-189 kubenswrapper[2578]: W0911 19:07:39.517315    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc67d229b_1c73_44f8_953b_8f4e79a6e3a1.slice/crio-227a7b7f18baac42b0941c256deb9e1f5f2ed3539736e6a8a7d37174ab8c31fa WatchSource:0}: Error finding container 227a7b7f18baac42b0941c256deb9e1f5f2ed3539736e6a8a7d37174ab8c31fa: Status 404 returned error can't find the container with id 227a7b7f18baac42b0941c256deb9e1f5f2ed3539736e6a8a7d37174ab8c31fa
Sep 11 19:07:39.565258 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:39.565228    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7ad53a9d-57cd7b475c-mzpjg"]
Sep 11 19:07:39.568589 ip-10-0-134-189 kubenswrapper[2578]: W0911 19:07:39.568560    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod570f6c2f_c791_4e27_82c1_24b1b4d8b88a.slice/crio-7e6878f1e5b46e5bdcd0dab85814fdb834df539b9e0c3213203c861bfdb53c4b WatchSource:0}: Error finding container 7e6878f1e5b46e5bdcd0dab85814fdb834df539b9e0c3213203c861bfdb53c4b: Status 404 returned error can't find the container with id 7e6878f1e5b46e5bdcd0dab85814fdb834df539b9e0c3213203c861bfdb53c4b
Sep 11 19:07:39.619839 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:39.619808    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7ad53a9d-8c87b79f4-hzpqk" event={"ID":"c67d229b-1c73-44f8-953b-8f4e79a6e3a1","Type":"ContainerStarted","Data":"f8b7940ff55253f30b4d3b85022c12d870c042e8b6c14d6404bc60d497834226"}
Sep 11 19:07:39.620005 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:39.619864    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7ad53a9d-8c87b79f4-hzpqk" event={"ID":"c67d229b-1c73-44f8-953b-8f4e79a6e3a1","Type":"ContainerStarted","Data":"227a7b7f18baac42b0941c256deb9e1f5f2ed3539736e6a8a7d37174ab8c31fa"}
Sep 11 19:07:39.620133 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:39.620106    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-7ad53a9d-8c87b79f4-hzpqk"
Sep 11 19:07:39.622369 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:39.622330    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-3b7391-openshift-default-7cd68cb6bd-kwtt7" event={"ID":"0eff5cf7-789d-4dbf-9894-568df3b2ba47","Type":"ContainerStarted","Data":"f0218df372cea8a1db10419b3e77115741f0fdbb3c28f548de95847dc679d8f7"}
Sep 11 19:07:39.622544 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:39.622385    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-3b7391-openshift-default-7cd68cb6bd-kwtt7" event={"ID":"0eff5cf7-789d-4dbf-9894-568df3b2ba47","Type":"ContainerStarted","Data":"09babab9da5ac7c41f9126300f1a457567f43559d14485784f2912318e77eb3a"}
Sep 11 19:07:39.623963 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:39.623943    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7ad53a9d-57cd7b475c-mzpjg" event={"ID":"570f6c2f-c791-4e27-82c1-24b1b4d8b88a","Type":"ContainerStarted","Data":"7e6878f1e5b46e5bdcd0dab85814fdb834df539b9e0c3213203c861bfdb53c4b"}
Sep 11 19:07:39.642633 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:39.642541    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-7ad53a9d-8c87b79f4-hzpqk" podStartSLOduration=1.6425183620000001 podStartE2EDuration="1.642518362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:07:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:07:39.641716815 +0000 UTC m=+1135.719307404" watchObservedRunningTime="2026-09-11 19:07:39.642518362 +0000 UTC m=+1135.720108918"
Sep 11 19:07:39.669372 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:39.669319    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-3b7391-openshift-default-7cd68cb6bd-kwtt7" podStartSLOduration=1.66929929 podStartE2EDuration="1.66929929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:07:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:07:39.66809977 +0000 UTC m=+1135.745690374" watchObservedRunningTime="2026-09-11 19:07:39.66929929 +0000 UTC m=+1135.746889835"
Sep 11 19:07:40.456193 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:40.456149    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-f70e4e/noten-model-f70e4e-kserve-554f77bdc5-5dl76"]
Sep 11 19:07:40.460175 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:40.460147    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-f70e4e/noten-model-f70e4e-kserve-554f77bdc5-5dl76"
Sep 11 19:07:40.462882 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:40.462852    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-f70e4e\"/\"openshift-service-ca.crt\""
Sep 11 19:07:40.463125 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:40.463107    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-f70e4e\"/\"kube-root-ca.crt\""
Sep 11 19:07:40.463236 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:40.463121    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-f70e4e\"/\"default-dockercfg-w58f7\""
Sep 11 19:07:40.463236 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:40.463137    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-f70e4e\"/\"noten-model-f70e4e-kserve-self-signed-certs\""
Sep 11 19:07:40.469991 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:40.469964    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-f70e4e/noten-model-f70e4e-kserve-554f77bdc5-5dl76"]
Sep 11 19:07:40.557615 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:40.557560    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-3b7391-openshift-default-7cd68cb6bd-kwtt7"
Sep 11 19:07:40.562615 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:40.562565    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29bc306e-6ad5-45ee-8d9e-d919dd5e0756-model-cache\") pod \"noten-model-f70e4e-kserve-554f77bdc5-5dl76\" (UID: \"29bc306e-6ad5-45ee-8d9e-d919dd5e0756\") " pod="ai-tenant-e2e-noten-f70e4e/noten-model-f70e4e-kserve-554f77bdc5-5dl76"
Sep 11 19:07:40.562785 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:40.562683    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mjrp\" (UniqueName: \"kubernetes.io/projected/29bc306e-6ad5-45ee-8d9e-d919dd5e0756-kube-api-access-5mjrp\") pod \"noten-model-f70e4e-kserve-554f77bdc5-5dl76\" (UID: \"29bc306e-6ad5-45ee-8d9e-d919dd5e0756\") " pod="ai-tenant-e2e-noten-f70e4e/noten-model-f70e4e-kserve-554f77bdc5-5dl76"
Sep 11 19:07:40.562785 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:40.562731    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29bc306e-6ad5-45ee-8d9e-d919dd5e0756-home\") pod \"noten-model-f70e4e-kserve-554f77bdc5-5dl76\" (UID: \"29bc306e-6ad5-45ee-8d9e-d919dd5e0756\") " pod="ai-tenant-e2e-noten-f70e4e/noten-model-f70e4e-kserve-554f77bdc5-5dl76"
Sep 11 19:07:40.562882 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:40.562785    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29bc306e-6ad5-45ee-8d9e-d919dd5e0756-tls-certs\") pod \"noten-model-f70e4e-kserve-554f77bdc5-5dl76\" (UID: \"29bc306e-6ad5-45ee-8d9e-d919dd5e0756\") " pod="ai-tenant-e2e-noten-f70e4e/noten-model-f70e4e-kserve-554f77bdc5-5dl76"
Sep 11 19:07:40.562882 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:40.562867    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29bc306e-6ad5-45ee-8d9e-d919dd5e0756-tmp-dir\") pod \"noten-model-f70e4e-kserve-554f77bdc5-5dl76\" (UID: \"29bc306e-6ad5-45ee-8d9e-d919dd5e0756\") " pod="ai-tenant-e2e-noten-f70e4e/noten-model-f70e4e-kserve-554f77bdc5-5dl76"
Sep 11 19:07:40.562953 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:40.562921    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29bc306e-6ad5-45ee-8d9e-d919dd5e0756-dshm\") pod \"noten-model-f70e4e-kserve-554f77bdc5-5dl76\" (UID: \"29bc306e-6ad5-45ee-8d9e-d919dd5e0756\") " pod="ai-tenant-e2e-noten-f70e4e/noten-model-f70e4e-kserve-554f77bdc5-5dl76"
Sep 11 19:07:40.563169 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:40.563150    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-3b7391-openshift-default-7cd68cb6bd-kwtt7"
Sep 11 19:07:40.631378 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:40.631328    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7ad53a9d-57cd7b475c-mzpjg" event={"ID":"570f6c2f-c791-4e27-82c1-24b1b4d8b88a","Type":"ContainerStarted","Data":"974f2960dbc5ae11248431564f5a97b16e8bd1b140f70e9d54e82f5f9b3ed5f7"}
Sep 11 19:07:40.631683 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:40.631660    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-7ad53a9d-57cd7b475c-mzpjg"
Sep 11 19:07:40.631799 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:40.631717    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-3b7391-openshift-default-7cd68cb6bd-kwtt7"
Sep 11 19:07:40.632497 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:40.632478    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-3b7391-openshift-default-7cd68cb6bd-kwtt7"
Sep 11 19:07:40.650787 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:40.650721    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-7ad53a9d-57cd7b475c-mzpjg" podStartSLOduration=2.650703806 podStartE2EDuration="2.650703806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:07:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:07:40.649021222 +0000 UTC m=+1136.726611776" watchObservedRunningTime="2026-09-11 19:07:40.650703806 +0000 UTC m=+1136.728294361"
Sep 11 19:07:40.664477 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:40.664434    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29bc306e-6ad5-45ee-8d9e-d919dd5e0756-home\") pod \"noten-model-f70e4e-kserve-554f77bdc5-5dl76\" (UID: \"29bc306e-6ad5-45ee-8d9e-d919dd5e0756\") " pod="ai-tenant-e2e-noten-f70e4e/noten-model-f70e4e-kserve-554f77bdc5-5dl76"
Sep 11 19:07:40.664709 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:40.664661    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29bc306e-6ad5-45ee-8d9e-d919dd5e0756-tls-certs\") pod \"noten-model-f70e4e-kserve-554f77bdc5-5dl76\" (UID: \"29bc306e-6ad5-45ee-8d9e-d919dd5e0756\") " pod="ai-tenant-e2e-noten-f70e4e/noten-model-f70e4e-kserve-554f77bdc5-5dl76"
Sep 11 19:07:40.664798 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:40.664750    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29bc306e-6ad5-45ee-8d9e-d919dd5e0756-tmp-dir\") pod \"noten-model-f70e4e-kserve-554f77bdc5-5dl76\" (UID: \"29bc306e-6ad5-45ee-8d9e-d919dd5e0756\") " pod="ai-tenant-e2e-noten-f70e4e/noten-model-f70e4e-kserve-554f77bdc5-5dl76"
Sep 11 19:07:40.664869 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:40.664821    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29bc306e-6ad5-45ee-8d9e-d919dd5e0756-dshm\") pod \"noten-model-f70e4e-kserve-554f77bdc5-5dl76\" (UID: \"29bc306e-6ad5-45ee-8d9e-d919dd5e0756\") " pod="ai-tenant-e2e-noten-f70e4e/noten-model-f70e4e-kserve-554f77bdc5-5dl76"
Sep 11 19:07:40.664935 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:40.664903    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29bc306e-6ad5-45ee-8d9e-d919dd5e0756-model-cache\") pod \"noten-model-f70e4e-kserve-554f77bdc5-5dl76\" (UID: \"29bc306e-6ad5-45ee-8d9e-d919dd5e0756\") " pod="ai-tenant-e2e-noten-f70e4e/noten-model-f70e4e-kserve-554f77bdc5-5dl76"
Sep 11 19:07:40.664997 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:40.664955    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29bc306e-6ad5-45ee-8d9e-d919dd5e0756-home\") pod \"noten-model-f70e4e-kserve-554f77bdc5-5dl76\" (UID: \"29bc306e-6ad5-45ee-8d9e-d919dd5e0756\") " pod="ai-tenant-e2e-noten-f70e4e/noten-model-f70e4e-kserve-554f77bdc5-5dl76"
Sep 11 19:07:40.665059 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:40.664997    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29bc306e-6ad5-45ee-8d9e-d919dd5e0756-tmp-dir\") pod \"noten-model-f70e4e-kserve-554f77bdc5-5dl76\" (UID: \"29bc306e-6ad5-45ee-8d9e-d919dd5e0756\") " pod="ai-tenant-e2e-noten-f70e4e/noten-model-f70e4e-kserve-554f77bdc5-5dl76"
Sep 11 19:07:40.665059 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:40.665047    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mjrp\" (UniqueName: \"kubernetes.io/projected/29bc306e-6ad5-45ee-8d9e-d919dd5e0756-kube-api-access-5mjrp\") pod \"noten-model-f70e4e-kserve-554f77bdc5-5dl76\" (UID: \"29bc306e-6ad5-45ee-8d9e-d919dd5e0756\") " pod="ai-tenant-e2e-noten-f70e4e/noten-model-f70e4e-kserve-554f77bdc5-5dl76"
Sep 11 19:07:40.665244 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:40.665219    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29bc306e-6ad5-45ee-8d9e-d919dd5e0756-model-cache\") pod \"noten-model-f70e4e-kserve-554f77bdc5-5dl76\" (UID: \"29bc306e-6ad5-45ee-8d9e-d919dd5e0756\") " pod="ai-tenant-e2e-noten-f70e4e/noten-model-f70e4e-kserve-554f77bdc5-5dl76"
Sep 11 19:07:40.667424 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:40.667401    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29bc306e-6ad5-45ee-8d9e-d919dd5e0756-dshm\") pod \"noten-model-f70e4e-kserve-554f77bdc5-5dl76\" (UID: \"29bc306e-6ad5-45ee-8d9e-d919dd5e0756\") " pod="ai-tenant-e2e-noten-f70e4e/noten-model-f70e4e-kserve-554f77bdc5-5dl76"
Sep 11 19:07:40.667691 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:40.667673    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29bc306e-6ad5-45ee-8d9e-d919dd5e0756-tls-certs\") pod \"noten-model-f70e4e-kserve-554f77bdc5-5dl76\" (UID: \"29bc306e-6ad5-45ee-8d9e-d919dd5e0756\") " pod="ai-tenant-e2e-noten-f70e4e/noten-model-f70e4e-kserve-554f77bdc5-5dl76"
Sep 11 19:07:40.677493 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:40.677466    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mjrp\" (UniqueName: \"kubernetes.io/projected/29bc306e-6ad5-45ee-8d9e-d919dd5e0756-kube-api-access-5mjrp\") pod \"noten-model-f70e4e-kserve-554f77bdc5-5dl76\" (UID: \"29bc306e-6ad5-45ee-8d9e-d919dd5e0756\") " pod="ai-tenant-e2e-noten-f70e4e/noten-model-f70e4e-kserve-554f77bdc5-5dl76"
Sep 11 19:07:40.771071 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:40.770982    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-f70e4e/noten-model-f70e4e-kserve-554f77bdc5-5dl76"
Sep 11 19:07:41.051059 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:41.051013    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-f70e4e/noten-model-f70e4e-kserve-554f77bdc5-5dl76"]
Sep 11 19:07:41.053256 ip-10-0-134-189 kubenswrapper[2578]: W0911 19:07:41.053226    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod29bc306e_6ad5_45ee_8d9e_d919dd5e0756.slice/crio-6300d9d09b48288eaeb7b4569960896e1a226b5bb74229329e0a0aba82bbe744 WatchSource:0}: Error finding container 6300d9d09b48288eaeb7b4569960896e1a226b5bb74229329e0a0aba82bbe744: Status 404 returned error can't find the container with id 6300d9d09b48288eaeb7b4569960896e1a226b5bb74229329e0a0aba82bbe744
Sep 11 19:07:41.639443 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:41.639403    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-f70e4e/noten-model-f70e4e-kserve-554f77bdc5-5dl76" event={"ID":"29bc306e-6ad5-45ee-8d9e-d919dd5e0756","Type":"ContainerStarted","Data":"54bc17073897aefd19046ddc4e114583dd5ff1ea311edea6ee9b0facf4523a03"}
Sep 11 19:07:41.639856 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:41.639459    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-f70e4e/noten-model-f70e4e-kserve-554f77bdc5-5dl76" event={"ID":"29bc306e-6ad5-45ee-8d9e-d919dd5e0756","Type":"ContainerStarted","Data":"6300d9d09b48288eaeb7b4569960896e1a226b5bb74229329e0a0aba82bbe744"}
Sep 11 19:07:41.664111 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:41.664043    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-f70e4e/noten-model-f70e4e-kserve-554f77bdc5-5dl76" podStartSLOduration=1.6640263979999999 podStartE2EDuration="1.664026398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:07:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:07:41.662785518 +0000 UTC m=+1137.740376088" watchObservedRunningTime="2026-09-11 19:07:41.664026398 +0000 UTC m=+1137.741616950"
Sep 11 19:07:50.633535 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:50.633501    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-7ad53a9d-8c87b79f4-hzpqk"
Sep 11 19:07:50.771154 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:50.771123    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-f70e4e/noten-model-f70e4e-kserve-554f77bdc5-5dl76"
Sep 11 19:07:50.771355 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:50.771174    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-f70e4e/noten-model-f70e4e-kserve-554f77bdc5-5dl76"
Sep 11 19:07:50.783995 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:50.783965    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-f70e4e/noten-model-f70e4e-kserve-554f77bdc5-5dl76"
Sep 11 19:07:51.642420 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:51.642391    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-7ad53a9d-57cd7b475c-mzpjg"
Sep 11 19:07:51.706888 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:07:51.706858    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-f70e4e/noten-model-f70e4e-kserve-554f77bdc5-5dl76"
Sep 11 19:08:15.336490 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:15.336455    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-62499f/distinct-2-w0-62499f-kserve-76c9886944-rptqp"]
Sep 11 19:08:15.339968 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:15.339948    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-62499f/distinct-2-w0-62499f-kserve-76c9886944-rptqp"
Sep 11 19:08:15.342314 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:15.342281    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-62499f\"/\"distinct-2-w0-62499f-kserve-self-signed-certs\""
Sep 11 19:08:15.349966 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:15.349939    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-62499f/distinct-2-w0-62499f-kserve-76c9886944-rptqp"]
Sep 11 19:08:15.433905 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:15.433849    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/060d3a3b-64a2-44a8-9d0f-2e0388c60cee-home\") pod \"distinct-2-w0-62499f-kserve-76c9886944-rptqp\" (UID: \"060d3a3b-64a2-44a8-9d0f-2e0388c60cee\") " pod="e2e-models-e2e-worker-w0-62499f/distinct-2-w0-62499f-kserve-76c9886944-rptqp"
Sep 11 19:08:15.434110 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:15.433927    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/060d3a3b-64a2-44a8-9d0f-2e0388c60cee-tls-certs\") pod \"distinct-2-w0-62499f-kserve-76c9886944-rptqp\" (UID: \"060d3a3b-64a2-44a8-9d0f-2e0388c60cee\") " pod="e2e-models-e2e-worker-w0-62499f/distinct-2-w0-62499f-kserve-76c9886944-rptqp"
Sep 11 19:08:15.434110 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:15.433976    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cbxj\" (UniqueName: \"kubernetes.io/projected/060d3a3b-64a2-44a8-9d0f-2e0388c60cee-kube-api-access-5cbxj\") pod \"distinct-2-w0-62499f-kserve-76c9886944-rptqp\" (UID: \"060d3a3b-64a2-44a8-9d0f-2e0388c60cee\") " pod="e2e-models-e2e-worker-w0-62499f/distinct-2-w0-62499f-kserve-76c9886944-rptqp"
Sep 11 19:08:15.434110 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:15.434045    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/060d3a3b-64a2-44a8-9d0f-2e0388c60cee-tmp-dir\") pod \"distinct-2-w0-62499f-kserve-76c9886944-rptqp\" (UID: \"060d3a3b-64a2-44a8-9d0f-2e0388c60cee\") " pod="e2e-models-e2e-worker-w0-62499f/distinct-2-w0-62499f-kserve-76c9886944-rptqp"
Sep 11 19:08:15.434110 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:15.434080    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/060d3a3b-64a2-44a8-9d0f-2e0388c60cee-dshm\") pod \"distinct-2-w0-62499f-kserve-76c9886944-rptqp\" (UID: \"060d3a3b-64a2-44a8-9d0f-2e0388c60cee\") " pod="e2e-models-e2e-worker-w0-62499f/distinct-2-w0-62499f-kserve-76c9886944-rptqp"
Sep 11 19:08:15.434278 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:15.434203    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/060d3a3b-64a2-44a8-9d0f-2e0388c60cee-model-cache\") pod \"distinct-2-w0-62499f-kserve-76c9886944-rptqp\" (UID: \"060d3a3b-64a2-44a8-9d0f-2e0388c60cee\") " pod="e2e-models-e2e-worker-w0-62499f/distinct-2-w0-62499f-kserve-76c9886944-rptqp"
Sep 11 19:08:15.534965 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:15.534928    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/060d3a3b-64a2-44a8-9d0f-2e0388c60cee-model-cache\") pod \"distinct-2-w0-62499f-kserve-76c9886944-rptqp\" (UID: \"060d3a3b-64a2-44a8-9d0f-2e0388c60cee\") " pod="e2e-models-e2e-worker-w0-62499f/distinct-2-w0-62499f-kserve-76c9886944-rptqp"
Sep 11 19:08:15.535172 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:15.534995    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/060d3a3b-64a2-44a8-9d0f-2e0388c60cee-home\") pod \"distinct-2-w0-62499f-kserve-76c9886944-rptqp\" (UID: \"060d3a3b-64a2-44a8-9d0f-2e0388c60cee\") " pod="e2e-models-e2e-worker-w0-62499f/distinct-2-w0-62499f-kserve-76c9886944-rptqp"
Sep 11 19:08:15.535172 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:15.535060    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/060d3a3b-64a2-44a8-9d0f-2e0388c60cee-tls-certs\") pod \"distinct-2-w0-62499f-kserve-76c9886944-rptqp\" (UID: \"060d3a3b-64a2-44a8-9d0f-2e0388c60cee\") " pod="e2e-models-e2e-worker-w0-62499f/distinct-2-w0-62499f-kserve-76c9886944-rptqp"
Sep 11 19:08:15.535172 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:15.535120    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cbxj\" (UniqueName: \"kubernetes.io/projected/060d3a3b-64a2-44a8-9d0f-2e0388c60cee-kube-api-access-5cbxj\") pod \"distinct-2-w0-62499f-kserve-76c9886944-rptqp\" (UID: \"060d3a3b-64a2-44a8-9d0f-2e0388c60cee\") " pod="e2e-models-e2e-worker-w0-62499f/distinct-2-w0-62499f-kserve-76c9886944-rptqp"
Sep 11 19:08:15.535360 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:15.535179    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/060d3a3b-64a2-44a8-9d0f-2e0388c60cee-tmp-dir\") pod \"distinct-2-w0-62499f-kserve-76c9886944-rptqp\" (UID: \"060d3a3b-64a2-44a8-9d0f-2e0388c60cee\") " pod="e2e-models-e2e-worker-w0-62499f/distinct-2-w0-62499f-kserve-76c9886944-rptqp"
Sep 11 19:08:15.535360 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:15.535220    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/060d3a3b-64a2-44a8-9d0f-2e0388c60cee-dshm\") pod \"distinct-2-w0-62499f-kserve-76c9886944-rptqp\" (UID: \"060d3a3b-64a2-44a8-9d0f-2e0388c60cee\") " pod="e2e-models-e2e-worker-w0-62499f/distinct-2-w0-62499f-kserve-76c9886944-rptqp"
Sep 11 19:08:15.535471 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:15.535360    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/060d3a3b-64a2-44a8-9d0f-2e0388c60cee-model-cache\") pod \"distinct-2-w0-62499f-kserve-76c9886944-rptqp\" (UID: \"060d3a3b-64a2-44a8-9d0f-2e0388c60cee\") " pod="e2e-models-e2e-worker-w0-62499f/distinct-2-w0-62499f-kserve-76c9886944-rptqp"
Sep 11 19:08:15.535471 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:15.535402    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/060d3a3b-64a2-44a8-9d0f-2e0388c60cee-home\") pod \"distinct-2-w0-62499f-kserve-76c9886944-rptqp\" (UID: \"060d3a3b-64a2-44a8-9d0f-2e0388c60cee\") " pod="e2e-models-e2e-worker-w0-62499f/distinct-2-w0-62499f-kserve-76c9886944-rptqp"
Sep 11 19:08:15.535697 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:15.535677    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/060d3a3b-64a2-44a8-9d0f-2e0388c60cee-tmp-dir\") pod \"distinct-2-w0-62499f-kserve-76c9886944-rptqp\" (UID: \"060d3a3b-64a2-44a8-9d0f-2e0388c60cee\") " pod="e2e-models-e2e-worker-w0-62499f/distinct-2-w0-62499f-kserve-76c9886944-rptqp"
Sep 11 19:08:15.537868 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:15.537845    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/060d3a3b-64a2-44a8-9d0f-2e0388c60cee-tls-certs\") pod \"distinct-2-w0-62499f-kserve-76c9886944-rptqp\" (UID: \"060d3a3b-64a2-44a8-9d0f-2e0388c60cee\") " pod="e2e-models-e2e-worker-w0-62499f/distinct-2-w0-62499f-kserve-76c9886944-rptqp"
Sep 11 19:08:15.538066 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:15.538048    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/060d3a3b-64a2-44a8-9d0f-2e0388c60cee-dshm\") pod \"distinct-2-w0-62499f-kserve-76c9886944-rptqp\" (UID: \"060d3a3b-64a2-44a8-9d0f-2e0388c60cee\") " pod="e2e-models-e2e-worker-w0-62499f/distinct-2-w0-62499f-kserve-76c9886944-rptqp"
Sep 11 19:08:15.544187 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:15.544164    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cbxj\" (UniqueName: \"kubernetes.io/projected/060d3a3b-64a2-44a8-9d0f-2e0388c60cee-kube-api-access-5cbxj\") pod \"distinct-2-w0-62499f-kserve-76c9886944-rptqp\" (UID: \"060d3a3b-64a2-44a8-9d0f-2e0388c60cee\") " pod="e2e-models-e2e-worker-w0-62499f/distinct-2-w0-62499f-kserve-76c9886944-rptqp"
Sep 11 19:08:15.651029 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:15.650940    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-62499f/distinct-2-w0-62499f-kserve-76c9886944-rptqp"
Sep 11 19:08:15.931429 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:15.931402    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-62499f/distinct-2-w0-62499f-kserve-76c9886944-rptqp"]
Sep 11 19:08:15.933895 ip-10-0-134-189 kubenswrapper[2578]: W0911 19:08:15.933865    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod060d3a3b_64a2_44a8_9d0f_2e0388c60cee.slice/crio-e00af61e1a3af139579581e782a6aa098732052b01a211e314cf0430363a8697 WatchSource:0}: Error finding container e00af61e1a3af139579581e782a6aa098732052b01a211e314cf0430363a8697: Status 404 returned error can't find the container with id e00af61e1a3af139579581e782a6aa098732052b01a211e314cf0430363a8697
Sep 11 19:08:16.835057 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:16.835016    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-62499f/distinct-2-w0-62499f-kserve-76c9886944-rptqp" event={"ID":"060d3a3b-64a2-44a8-9d0f-2e0388c60cee","Type":"ContainerStarted","Data":"3169786a632b7d075dcc34efd1945462c5eb73037893c2b1ec9e7aae300caa3d"}
Sep 11 19:08:16.835057 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:16.835060    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-62499f/distinct-2-w0-62499f-kserve-76c9886944-rptqp" event={"ID":"060d3a3b-64a2-44a8-9d0f-2e0388c60cee","Type":"ContainerStarted","Data":"e00af61e1a3af139579581e782a6aa098732052b01a211e314cf0430363a8697"}
Sep 11 19:08:16.860697 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:16.860626    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-62499f/distinct-2-w0-62499f-kserve-76c9886944-rptqp" podStartSLOduration=1.860582918 podStartE2EDuration="1.860582918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:08:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:08:16.858892166 +0000 UTC m=+1172.936482720" watchObservedRunningTime="2026-09-11 19:08:16.860582918 +0000 UTC m=+1172.938173473"
Sep 11 19:08:17.841798 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:17.841766    2578 generic.go:356] "Generic (PLEG): container finished" podID="0eff5cf7-789d-4dbf-9894-568df3b2ba47" containerID="f0218df372cea8a1db10419b3e77115741f0fdbb3c28f548de95847dc679d8f7" exitCode=0
Sep 11 19:08:17.842238 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:17.841840    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-3b7391-openshift-default-7cd68cb6bd-kwtt7" event={"ID":"0eff5cf7-789d-4dbf-9894-568df3b2ba47","Type":"ContainerDied","Data":"f0218df372cea8a1db10419b3e77115741f0fdbb3c28f548de95847dc679d8f7"}
Sep 11 19:08:17.842391 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:17.842376    2578 scope.go:117] "RemoveContainer" containerID="f0218df372cea8a1db10419b3e77115741f0fdbb3c28f548de95847dc679d8f7"
Sep 11 19:08:17.843488 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:17.843464    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 19:08:17.843544 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:17.843534    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 19:08:17.843580 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:17.843565    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 19:08:18.849431 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:18.849384    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-3b7391-openshift-default-7cd68cb6bd-kwtt7" event={"ID":"0eff5cf7-789d-4dbf-9894-568df3b2ba47","Type":"ContainerStarted","Data":"6100cc7be28085a781dc7153cc6e10dc7ddcb9676cb77e7a9b1cf4b720946445"}
Sep 11 19:08:19.557811 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:19.557775    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-3b7391-openshift-default-7cd68cb6bd-kwtt7"
Sep 11 19:08:19.558023 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:19.558001    2578 patch_prober.go:28] interesting pod/e2e-shared-b-w3-3b7391-openshift-default-7cd68cb6bd-kwtt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Sep 11 19:08:19.558093 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:19.558066    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-3b7391-openshift-default-7cd68cb6bd-kwtt7" podUID="0eff5cf7-789d-4dbf-9894-568df3b2ba47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Sep 11 19:08:20.065621 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:20.060286    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-f70e4e/noten-model-f70e4e-kserve-554f77bdc5-5dl76"]
Sep 11 19:08:20.558496 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:20.558462    2578 patch_prober.go:28] interesting pod/e2e-shared-b-w3-3b7391-openshift-default-7cd68cb6bd-kwtt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Sep 11 19:08:20.558785 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:20.558519    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-3b7391-openshift-default-7cd68cb6bd-kwtt7" podUID="0eff5cf7-789d-4dbf-9894-568df3b2ba47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Sep 11 19:08:21.557685 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:21.557648    2578 patch_prober.go:28] interesting pod/e2e-shared-b-w3-3b7391-openshift-default-7cd68cb6bd-kwtt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Sep 11 19:08:21.558084 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:21.557705    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-3b7391-openshift-default-7cd68cb6bd-kwtt7" podUID="0eff5cf7-789d-4dbf-9894-568df3b2ba47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Sep 11 19:08:22.557523 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:22.557486    2578 patch_prober.go:28] interesting pod/e2e-shared-b-w3-3b7391-openshift-default-7cd68cb6bd-kwtt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Sep 11 19:08:22.557745 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:22.557559    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-3b7391-openshift-default-7cd68cb6bd-kwtt7" podUID="0eff5cf7-789d-4dbf-9894-568df3b2ba47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Sep 11 19:08:23.557764 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:23.557728    2578 patch_prober.go:28] interesting pod/e2e-shared-b-w3-3b7391-openshift-default-7cd68cb6bd-kwtt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Sep 11 19:08:23.558153 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:23.557785    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-3b7391-openshift-default-7cd68cb6bd-kwtt7" podUID="0eff5cf7-789d-4dbf-9894-568df3b2ba47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Sep 11 19:08:24.562215 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:24.562183    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-3b7391-openshift-default-7cd68cb6bd-kwtt7"
Sep 11 19:08:24.562695 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:24.562475    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-3b7391-openshift-default-7cd68cb6bd-kwtt7"
Sep 11 19:08:24.563168 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:24.563149    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-3b7391-openshift-default-7cd68cb6bd-kwtt7"
Sep 11 19:08:25.651959 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:25.651925    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-62499f/distinct-2-w0-62499f-kserve-76c9886944-rptqp"
Sep 11 19:08:25.652433 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:25.651975    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-62499f/distinct-2-w0-62499f-kserve-76c9886944-rptqp"
Sep 11 19:08:25.664410 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:25.664383    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-62499f/distinct-2-w0-62499f-kserve-76c9886944-rptqp"
Sep 11 19:08:25.907260 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:25.907170    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-62499f/distinct-2-w0-62499f-kserve-76c9886944-rptqp"
Sep 11 19:08:35.101567 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:35.101499    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-f70e4e/noten-model-f70e4e-kserve-554f77bdc5-5dl76" podUID="29bc306e-6ad5-45ee-8d9e-d919dd5e0756" containerName="main" containerID="cri-o://54bc17073897aefd19046ddc4e114583dd5ff1ea311edea6ee9b0facf4523a03" gracePeriod=45
Sep 11 19:08:35.348764 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:35.348738    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-f70e4e/noten-model-f70e4e-kserve-554f77bdc5-5dl76"
Sep 11 19:08:35.441431 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:35.441291    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29bc306e-6ad5-45ee-8d9e-d919dd5e0756-tls-certs\") pod \"29bc306e-6ad5-45ee-8d9e-d919dd5e0756\" (UID: \"29bc306e-6ad5-45ee-8d9e-d919dd5e0756\") "
Sep 11 19:08:35.441431 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:35.441413    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29bc306e-6ad5-45ee-8d9e-d919dd5e0756-home\") pod \"29bc306e-6ad5-45ee-8d9e-d919dd5e0756\" (UID: \"29bc306e-6ad5-45ee-8d9e-d919dd5e0756\") "
Sep 11 19:08:35.441676 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:35.441444    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5mjrp\" (UniqueName: \"kubernetes.io/projected/29bc306e-6ad5-45ee-8d9e-d919dd5e0756-kube-api-access-5mjrp\") pod \"29bc306e-6ad5-45ee-8d9e-d919dd5e0756\" (UID: \"29bc306e-6ad5-45ee-8d9e-d919dd5e0756\") "
Sep 11 19:08:35.441676 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:35.441462    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29bc306e-6ad5-45ee-8d9e-d919dd5e0756-model-cache\") pod \"29bc306e-6ad5-45ee-8d9e-d919dd5e0756\" (UID: \"29bc306e-6ad5-45ee-8d9e-d919dd5e0756\") "
Sep 11 19:08:35.441676 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:35.441487    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29bc306e-6ad5-45ee-8d9e-d919dd5e0756-tmp-dir\") pod \"29bc306e-6ad5-45ee-8d9e-d919dd5e0756\" (UID: \"29bc306e-6ad5-45ee-8d9e-d919dd5e0756\") "
Sep 11 19:08:35.441676 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:35.441647    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29bc306e-6ad5-45ee-8d9e-d919dd5e0756-dshm\") pod \"29bc306e-6ad5-45ee-8d9e-d919dd5e0756\" (UID: \"29bc306e-6ad5-45ee-8d9e-d919dd5e0756\") "
Sep 11 19:08:35.441896 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:35.441721    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29bc306e-6ad5-45ee-8d9e-d919dd5e0756-home" (OuterVolumeSpecName: "home") pod "29bc306e-6ad5-45ee-8d9e-d919dd5e0756" (UID: "29bc306e-6ad5-45ee-8d9e-d919dd5e0756"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:08:35.441896 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:35.441761    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29bc306e-6ad5-45ee-8d9e-d919dd5e0756-model-cache" (OuterVolumeSpecName: "model-cache") pod "29bc306e-6ad5-45ee-8d9e-d919dd5e0756" (UID: "29bc306e-6ad5-45ee-8d9e-d919dd5e0756"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:08:35.441896 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:35.441870    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29bc306e-6ad5-45ee-8d9e-d919dd5e0756-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "29bc306e-6ad5-45ee-8d9e-d919dd5e0756" (UID: "29bc306e-6ad5-45ee-8d9e-d919dd5e0756"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:08:35.442315 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:35.442241    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29bc306e-6ad5-45ee-8d9e-d919dd5e0756-home\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:08:35.442315 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:35.442269    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29bc306e-6ad5-45ee-8d9e-d919dd5e0756-model-cache\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:08:35.442315 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:35.442285    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29bc306e-6ad5-45ee-8d9e-d919dd5e0756-tmp-dir\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:08:35.443742 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:35.443712    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29bc306e-6ad5-45ee-8d9e-d919dd5e0756-kube-api-access-5mjrp" (OuterVolumeSpecName: "kube-api-access-5mjrp") pod "29bc306e-6ad5-45ee-8d9e-d919dd5e0756" (UID: "29bc306e-6ad5-45ee-8d9e-d919dd5e0756"). InnerVolumeSpecName "kube-api-access-5mjrp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:08:35.443848 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:35.443747    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29bc306e-6ad5-45ee-8d9e-d919dd5e0756-dshm" (OuterVolumeSpecName: "dshm") pod "29bc306e-6ad5-45ee-8d9e-d919dd5e0756" (UID: "29bc306e-6ad5-45ee-8d9e-d919dd5e0756"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:08:35.443848 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:35.443756    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29bc306e-6ad5-45ee-8d9e-d919dd5e0756-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "29bc306e-6ad5-45ee-8d9e-d919dd5e0756" (UID: "29bc306e-6ad5-45ee-8d9e-d919dd5e0756"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 19:08:35.543232 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:35.543185    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29bc306e-6ad5-45ee-8d9e-d919dd5e0756-dshm\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:08:35.543232 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:35.543223    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29bc306e-6ad5-45ee-8d9e-d919dd5e0756-tls-certs\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:08:35.543232 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:35.543238    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mjrp\" (UniqueName: \"kubernetes.io/projected/29bc306e-6ad5-45ee-8d9e-d919dd5e0756-kube-api-access-5mjrp\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:08:35.949013 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:35.948977    2578 generic.go:356] "Generic (PLEG): container finished" podID="29bc306e-6ad5-45ee-8d9e-d919dd5e0756" containerID="54bc17073897aefd19046ddc4e114583dd5ff1ea311edea6ee9b0facf4523a03" exitCode=0
Sep 11 19:08:35.949013 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:35.949014    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-f70e4e/noten-model-f70e4e-kserve-554f77bdc5-5dl76"
Sep 11 19:08:35.949294 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:35.949064    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-f70e4e/noten-model-f70e4e-kserve-554f77bdc5-5dl76" event={"ID":"29bc306e-6ad5-45ee-8d9e-d919dd5e0756","Type":"ContainerDied","Data":"54bc17073897aefd19046ddc4e114583dd5ff1ea311edea6ee9b0facf4523a03"}
Sep 11 19:08:35.949294 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:35.949112    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-f70e4e/noten-model-f70e4e-kserve-554f77bdc5-5dl76" event={"ID":"29bc306e-6ad5-45ee-8d9e-d919dd5e0756","Type":"ContainerDied","Data":"6300d9d09b48288eaeb7b4569960896e1a226b5bb74229329e0a0aba82bbe744"}
Sep 11 19:08:35.949294 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:35.949135    2578 scope.go:117] "RemoveContainer" containerID="54bc17073897aefd19046ddc4e114583dd5ff1ea311edea6ee9b0facf4523a03"
Sep 11 19:08:35.958638 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:35.958621    2578 scope.go:117] "RemoveContainer" containerID="54bc17073897aefd19046ddc4e114583dd5ff1ea311edea6ee9b0facf4523a03"
Sep 11 19:08:35.958944 ip-10-0-134-189 kubenswrapper[2578]: E0911 19:08:35.958921    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54bc17073897aefd19046ddc4e114583dd5ff1ea311edea6ee9b0facf4523a03\": container with ID starting with 54bc17073897aefd19046ddc4e114583dd5ff1ea311edea6ee9b0facf4523a03 not found: ID does not exist" containerID="54bc17073897aefd19046ddc4e114583dd5ff1ea311edea6ee9b0facf4523a03"
Sep 11 19:08:35.958999 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:35.958955    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54bc17073897aefd19046ddc4e114583dd5ff1ea311edea6ee9b0facf4523a03"} err="failed to get container status \"54bc17073897aefd19046ddc4e114583dd5ff1ea311edea6ee9b0facf4523a03\": rpc error: code = NotFound desc = could not find container \"54bc17073897aefd19046ddc4e114583dd5ff1ea311edea6ee9b0facf4523a03\": container with ID starting with 54bc17073897aefd19046ddc4e114583dd5ff1ea311edea6ee9b0facf4523a03 not found: ID does not exist"
Sep 11 19:08:35.976947 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:35.976917    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-f70e4e/noten-model-f70e4e-kserve-554f77bdc5-5dl76"]
Sep 11 19:08:35.989027 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:35.989001    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-f70e4e/noten-model-f70e4e-kserve-554f77bdc5-5dl76"]
Sep 11 19:08:36.397224 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:36.397188    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29bc306e-6ad5-45ee-8d9e-d919dd5e0756" path="/var/lib/kubelet/pods/29bc306e-6ad5-45ee-8d9e-d919dd5e0756/volumes"
Sep 11 19:08:44.368678 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:44.368643    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-hsbsf_d0880abe-ec36-48e4-a511-ec663873fffa/console-operator/1.log"
Sep 11 19:08:44.371805 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:44.371779    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-hsbsf_d0880abe-ec36-48e4-a511-ec663873fffa/console-operator/1.log"
Sep 11 19:08:44.458429 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:44.458399    2578 scope.go:117] "RemoveContainer" containerID="ab95400cf93d4f634d06a12cb2bb9bbfce2fc9152df07e45c2e08411a93c54fc"
Sep 11 19:08:59.424215 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:59.424174    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-7ad53a9d/e2e-lifecycle-model-7ad53a9d-kserve-86f545dbc9-zlh6b"]
Sep 11 19:08:59.424817 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:59.424799    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29bc306e-6ad5-45ee-8d9e-d919dd5e0756" containerName="main"
Sep 11 19:08:59.424868 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:59.424821    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="29bc306e-6ad5-45ee-8d9e-d919dd5e0756" containerName="main"
Sep 11 19:08:59.424937 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:59.424926    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="29bc306e-6ad5-45ee-8d9e-d919dd5e0756" containerName="main"
Sep 11 19:08:59.429856 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:59.429834    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-7ad53a9d/e2e-lifecycle-model-7ad53a9d-kserve-86f545dbc9-zlh6b"
Sep 11 19:08:59.432347 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:59.432323    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-7ad53a9d\"/\"e2e-lifecycle-model-7ad53a9d-kserve-self-signed-certs\""
Sep 11 19:08:59.433104 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:59.433080    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-7ad53a9d\"/\"kube-root-ca.crt\""
Sep 11 19:08:59.433235 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:59.433089    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-7ad53a9d\"/\"openshift-service-ca.crt\""
Sep 11 19:08:59.433235 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:59.433171    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-7ad53a9d\"/\"default-dockercfg-tdbgw\""
Sep 11 19:08:59.438963 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:59.438936    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-7ad53a9d/e2e-lifecycle-model-7ad53a9d-kserve-86f545dbc9-zlh6b"]
Sep 11 19:08:59.596417 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:59.596373    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljgmv\" (UniqueName: \"kubernetes.io/projected/f915305d-9ebc-4e8e-8e0a-9748f0389d0b-kube-api-access-ljgmv\") pod \"e2e-lifecycle-model-7ad53a9d-kserve-86f545dbc9-zlh6b\" (UID: \"f915305d-9ebc-4e8e-8e0a-9748f0389d0b\") " pod="ai-tenant-e2e-mt-7ad53a9d/e2e-lifecycle-model-7ad53a9d-kserve-86f545dbc9-zlh6b"
Sep 11 19:08:59.596671 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:59.596429    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f915305d-9ebc-4e8e-8e0a-9748f0389d0b-tls-certs\") pod \"e2e-lifecycle-model-7ad53a9d-kserve-86f545dbc9-zlh6b\" (UID: \"f915305d-9ebc-4e8e-8e0a-9748f0389d0b\") " pod="ai-tenant-e2e-mt-7ad53a9d/e2e-lifecycle-model-7ad53a9d-kserve-86f545dbc9-zlh6b"
Sep 11 19:08:59.596671 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:59.596579    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f915305d-9ebc-4e8e-8e0a-9748f0389d0b-model-cache\") pod \"e2e-lifecycle-model-7ad53a9d-kserve-86f545dbc9-zlh6b\" (UID: \"f915305d-9ebc-4e8e-8e0a-9748f0389d0b\") " pod="ai-tenant-e2e-mt-7ad53a9d/e2e-lifecycle-model-7ad53a9d-kserve-86f545dbc9-zlh6b"
Sep 11 19:08:59.596671 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:59.596645    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f915305d-9ebc-4e8e-8e0a-9748f0389d0b-home\") pod \"e2e-lifecycle-model-7ad53a9d-kserve-86f545dbc9-zlh6b\" (UID: \"f915305d-9ebc-4e8e-8e0a-9748f0389d0b\") " pod="ai-tenant-e2e-mt-7ad53a9d/e2e-lifecycle-model-7ad53a9d-kserve-86f545dbc9-zlh6b"
Sep 11 19:08:59.596822 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:59.596684    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f915305d-9ebc-4e8e-8e0a-9748f0389d0b-dshm\") pod \"e2e-lifecycle-model-7ad53a9d-kserve-86f545dbc9-zlh6b\" (UID: \"f915305d-9ebc-4e8e-8e0a-9748f0389d0b\") " pod="ai-tenant-e2e-mt-7ad53a9d/e2e-lifecycle-model-7ad53a9d-kserve-86f545dbc9-zlh6b"
Sep 11 19:08:59.596822 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:59.596708    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f915305d-9ebc-4e8e-8e0a-9748f0389d0b-tmp-dir\") pod \"e2e-lifecycle-model-7ad53a9d-kserve-86f545dbc9-zlh6b\" (UID: \"f915305d-9ebc-4e8e-8e0a-9748f0389d0b\") " pod="ai-tenant-e2e-mt-7ad53a9d/e2e-lifecycle-model-7ad53a9d-kserve-86f545dbc9-zlh6b"
Sep 11 19:08:59.697901 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:59.697806    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f915305d-9ebc-4e8e-8e0a-9748f0389d0b-model-cache\") pod \"e2e-lifecycle-model-7ad53a9d-kserve-86f545dbc9-zlh6b\" (UID: \"f915305d-9ebc-4e8e-8e0a-9748f0389d0b\") " pod="ai-tenant-e2e-mt-7ad53a9d/e2e-lifecycle-model-7ad53a9d-kserve-86f545dbc9-zlh6b"
Sep 11 19:08:59.697901 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:59.697849    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f915305d-9ebc-4e8e-8e0a-9748f0389d0b-home\") pod \"e2e-lifecycle-model-7ad53a9d-kserve-86f545dbc9-zlh6b\" (UID: \"f915305d-9ebc-4e8e-8e0a-9748f0389d0b\") " pod="ai-tenant-e2e-mt-7ad53a9d/e2e-lifecycle-model-7ad53a9d-kserve-86f545dbc9-zlh6b"
Sep 11 19:08:59.697901 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:59.697874    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f915305d-9ebc-4e8e-8e0a-9748f0389d0b-dshm\") pod \"e2e-lifecycle-model-7ad53a9d-kserve-86f545dbc9-zlh6b\" (UID: \"f915305d-9ebc-4e8e-8e0a-9748f0389d0b\") " pod="ai-tenant-e2e-mt-7ad53a9d/e2e-lifecycle-model-7ad53a9d-kserve-86f545dbc9-zlh6b"
Sep 11 19:08:59.698197 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:59.697906    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f915305d-9ebc-4e8e-8e0a-9748f0389d0b-tmp-dir\") pod \"e2e-lifecycle-model-7ad53a9d-kserve-86f545dbc9-zlh6b\" (UID: \"f915305d-9ebc-4e8e-8e0a-9748f0389d0b\") " pod="ai-tenant-e2e-mt-7ad53a9d/e2e-lifecycle-model-7ad53a9d-kserve-86f545dbc9-zlh6b"
Sep 11 19:08:59.698197 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:59.697988    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljgmv\" (UniqueName: \"kubernetes.io/projected/f915305d-9ebc-4e8e-8e0a-9748f0389d0b-kube-api-access-ljgmv\") pod \"e2e-lifecycle-model-7ad53a9d-kserve-86f545dbc9-zlh6b\" (UID: \"f915305d-9ebc-4e8e-8e0a-9748f0389d0b\") " pod="ai-tenant-e2e-mt-7ad53a9d/e2e-lifecycle-model-7ad53a9d-kserve-86f545dbc9-zlh6b"
Sep 11 19:08:59.698197 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:59.698019    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f915305d-9ebc-4e8e-8e0a-9748f0389d0b-tls-certs\") pod \"e2e-lifecycle-model-7ad53a9d-kserve-86f545dbc9-zlh6b\" (UID: \"f915305d-9ebc-4e8e-8e0a-9748f0389d0b\") " pod="ai-tenant-e2e-mt-7ad53a9d/e2e-lifecycle-model-7ad53a9d-kserve-86f545dbc9-zlh6b"
Sep 11 19:08:59.698378 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:59.698284    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f915305d-9ebc-4e8e-8e0a-9748f0389d0b-model-cache\") pod \"e2e-lifecycle-model-7ad53a9d-kserve-86f545dbc9-zlh6b\" (UID: \"f915305d-9ebc-4e8e-8e0a-9748f0389d0b\") " pod="ai-tenant-e2e-mt-7ad53a9d/e2e-lifecycle-model-7ad53a9d-kserve-86f545dbc9-zlh6b"
Sep 11 19:08:59.698378 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:59.698347    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f915305d-9ebc-4e8e-8e0a-9748f0389d0b-tmp-dir\") pod \"e2e-lifecycle-model-7ad53a9d-kserve-86f545dbc9-zlh6b\" (UID: \"f915305d-9ebc-4e8e-8e0a-9748f0389d0b\") " pod="ai-tenant-e2e-mt-7ad53a9d/e2e-lifecycle-model-7ad53a9d-kserve-86f545dbc9-zlh6b"
Sep 11 19:08:59.698573 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:59.698552    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f915305d-9ebc-4e8e-8e0a-9748f0389d0b-home\") pod \"e2e-lifecycle-model-7ad53a9d-kserve-86f545dbc9-zlh6b\" (UID: \"f915305d-9ebc-4e8e-8e0a-9748f0389d0b\") " pod="ai-tenant-e2e-mt-7ad53a9d/e2e-lifecycle-model-7ad53a9d-kserve-86f545dbc9-zlh6b"
Sep 11 19:08:59.700629 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:59.700574    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f915305d-9ebc-4e8e-8e0a-9748f0389d0b-dshm\") pod \"e2e-lifecycle-model-7ad53a9d-kserve-86f545dbc9-zlh6b\" (UID: \"f915305d-9ebc-4e8e-8e0a-9748f0389d0b\") " pod="ai-tenant-e2e-mt-7ad53a9d/e2e-lifecycle-model-7ad53a9d-kserve-86f545dbc9-zlh6b"
Sep 11 19:08:59.700860 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:59.700839    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f915305d-9ebc-4e8e-8e0a-9748f0389d0b-tls-certs\") pod \"e2e-lifecycle-model-7ad53a9d-kserve-86f545dbc9-zlh6b\" (UID: \"f915305d-9ebc-4e8e-8e0a-9748f0389d0b\") " pod="ai-tenant-e2e-mt-7ad53a9d/e2e-lifecycle-model-7ad53a9d-kserve-86f545dbc9-zlh6b"
Sep 11 19:08:59.708902 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:59.708866    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljgmv\" (UniqueName: \"kubernetes.io/projected/f915305d-9ebc-4e8e-8e0a-9748f0389d0b-kube-api-access-ljgmv\") pod \"e2e-lifecycle-model-7ad53a9d-kserve-86f545dbc9-zlh6b\" (UID: \"f915305d-9ebc-4e8e-8e0a-9748f0389d0b\") " pod="ai-tenant-e2e-mt-7ad53a9d/e2e-lifecycle-model-7ad53a9d-kserve-86f545dbc9-zlh6b"
Sep 11 19:08:59.741499 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:08:59.741454    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-7ad53a9d/e2e-lifecycle-model-7ad53a9d-kserve-86f545dbc9-zlh6b"
Sep 11 19:09:00.035819 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:09:00.035638    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-7ad53a9d/e2e-lifecycle-model-7ad53a9d-kserve-86f545dbc9-zlh6b"]
Sep 11 19:09:00.038799 ip-10-0-134-189 kubenswrapper[2578]: W0911 19:09:00.038774    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf915305d_9ebc_4e8e_8e0a_9748f0389d0b.slice/crio-63aa3beeaaa04b045b052d05a682c6219c08f8e189a99ce05f17bb68054436e3 WatchSource:0}: Error finding container 63aa3beeaaa04b045b052d05a682c6219c08f8e189a99ce05f17bb68054436e3: Status 404 returned error can't find the container with id 63aa3beeaaa04b045b052d05a682c6219c08f8e189a99ce05f17bb68054436e3
Sep 11 19:09:00.076488 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:09:00.076459    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-7ad53a9d/e2e-lifecycle-model-7ad53a9d-kserve-86f545dbc9-zlh6b" event={"ID":"f915305d-9ebc-4e8e-8e0a-9748f0389d0b","Type":"ContainerStarted","Data":"63aa3beeaaa04b045b052d05a682c6219c08f8e189a99ce05f17bb68054436e3"}
Sep 11 19:09:01.083055 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:09:01.083014    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-7ad53a9d/e2e-lifecycle-model-7ad53a9d-kserve-86f545dbc9-zlh6b" event={"ID":"f915305d-9ebc-4e8e-8e0a-9748f0389d0b","Type":"ContainerStarted","Data":"11b8973361b7b61fff7c40446d1d1e052657731011b89a871897a52140c281b5"}
Sep 11 19:09:01.101322 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:09:01.101234    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-7ad53a9d/e2e-lifecycle-model-7ad53a9d-kserve-86f545dbc9-zlh6b" podStartSLOduration=2.101212894 podStartE2EDuration="2.101212894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:08:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:09:01.10064058 +0000 UTC m=+1217.178231143" watchObservedRunningTime="2026-09-11 19:09:01.101212894 +0000 UTC m=+1217.178803462"
Sep 11 19:09:09.742373 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:09:09.742329    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-7ad53a9d/e2e-lifecycle-model-7ad53a9d-kserve-86f545dbc9-zlh6b"
Sep 11 19:09:09.742896 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:09:09.742398    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-7ad53a9d/e2e-lifecycle-model-7ad53a9d-kserve-86f545dbc9-zlh6b"
Sep 11 19:09:09.754903 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:09:09.754876    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-7ad53a9d/e2e-lifecycle-model-7ad53a9d-kserve-86f545dbc9-zlh6b"
Sep 11 19:09:10.141872 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:09:10.141786    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-7ad53a9d/e2e-lifecycle-model-7ad53a9d-kserve-86f545dbc9-zlh6b"
Sep 11 19:09:43.287014 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:09:43.286926    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-62499f-db4mp"]
Sep 11 19:09:43.290587 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:09:43.290563    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-62499f-db4mp"
Sep 11 19:09:43.293610 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:09:43.293572    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ln295\""
Sep 11 19:09:43.293849 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:09:43.293831    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 19:09:43.300284 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:09:43.300252    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-62499f-db4mp"]
Sep 11 19:09:43.355639 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:09:43.355576    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1ad35af-b5d3-4e95-9bc9-e275d0f5fa26-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-62499f-db4mp\" (UID: \"d1ad35af-b5d3-4e95-9bc9-e275d0f5fa26\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-62499f-db4mp"
Sep 11 19:09:43.456964 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:09:43.456921    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1ad35af-b5d3-4e95-9bc9-e275d0f5fa26-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-62499f-db4mp\" (UID: \"d1ad35af-b5d3-4e95-9bc9-e275d0f5fa26\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-62499f-db4mp"
Sep 11 19:09:43.457569 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:09:43.457550    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1ad35af-b5d3-4e95-9bc9-e275d0f5fa26-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-62499f-db4mp\" (UID: \"d1ad35af-b5d3-4e95-9bc9-e275d0f5fa26\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-62499f-db4mp"
Sep 11 19:09:43.603200 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:09:43.603119    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-62499f-db4mp"
Sep 11 19:09:43.881469 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:09:43.881444    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-62499f-db4mp"]
Sep 11 19:09:43.883565 ip-10-0-134-189 kubenswrapper[2578]: W0911 19:09:43.883535    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1ad35af_b5d3_4e95_9bc9_e275d0f5fa26.slice/crio-79f880c0e6d4c0e12b128f71de7d8b0e18e11a40ba03e306e3e1f33d1eb593f0 WatchSource:0}: Error finding container 79f880c0e6d4c0e12b128f71de7d8b0e18e11a40ba03e306e3e1f33d1eb593f0: Status 404 returned error can't find the container with id 79f880c0e6d4c0e12b128f71de7d8b0e18e11a40ba03e306e3e1f33d1eb593f0
Sep 11 19:09:44.311548 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:09:44.311510    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-62499f-db4mp" event={"ID":"d1ad35af-b5d3-4e95-9bc9-e275d0f5fa26","Type":"ContainerStarted","Data":"ac3ab7aa9b5c632ec4ea75ecaa8eb6f8eeab53507b6544a4ef8ad8738c6af40b"}
Sep 11 19:09:44.311954 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:09:44.311571    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-62499f-db4mp" event={"ID":"d1ad35af-b5d3-4e95-9bc9-e275d0f5fa26","Type":"ContainerStarted","Data":"79f880c0e6d4c0e12b128f71de7d8b0e18e11a40ba03e306e3e1f33d1eb593f0"}
Sep 11 19:09:44.330649 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:09:44.330571    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-62499f-db4mp" podStartSLOduration=1.3305536629999999 podStartE2EDuration="1.330553663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:09:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:09:44.32851402 +0000 UTC m=+1260.406104594" watchObservedRunningTime="2026-09-11 19:09:44.330553663 +0000 UTC m=+1260.408144267"
Sep 11 19:09:44.482569 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:09:44.482537    2578 scope.go:117] "RemoveContainer" containerID="de0b866fe122d80118e9994082cdb1fc5c52a6ae94fdba83918053e47ca8d98e"
Sep 11 19:09:44.491574 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:09:44.491551    2578 scope.go:117] "RemoveContainer" containerID="9f6ce7c1bc63cb265bf0bb4a833cb3585e2c96027d77be50948afd31d03d7af2"
Sep 11 19:09:45.319044 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:09:45.319005    2578 generic.go:356] "Generic (PLEG): container finished" podID="d1ad35af-b5d3-4e95-9bc9-e275d0f5fa26" containerID="ac3ab7aa9b5c632ec4ea75ecaa8eb6f8eeab53507b6544a4ef8ad8738c6af40b" exitCode=0
Sep 11 19:09:45.319436 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:09:45.319087    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-62499f-db4mp" event={"ID":"d1ad35af-b5d3-4e95-9bc9-e275d0f5fa26","Type":"ContainerDied","Data":"ac3ab7aa9b5c632ec4ea75ecaa8eb6f8eeab53507b6544a4ef8ad8738c6af40b"}
Sep 11 19:09:46.470805 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:09:46.470775    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-62499f-db4mp"
Sep 11 19:09:46.590505 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:09:46.590348    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1ad35af-b5d3-4e95-9bc9-e275d0f5fa26-maas-api-service-ca\") pod \"d1ad35af-b5d3-4e95-9bc9-e275d0f5fa26\" (UID: \"d1ad35af-b5d3-4e95-9bc9-e275d0f5fa26\") "
Sep 11 19:09:46.590876 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:09:46.590849    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1ad35af-b5d3-4e95-9bc9-e275d0f5fa26-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d1ad35af-b5d3-4e95-9bc9-e275d0f5fa26" (UID: "d1ad35af-b5d3-4e95-9bc9-e275d0f5fa26"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 19:09:46.691716 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:09:46.691676    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1ad35af-b5d3-4e95-9bc9-e275d0f5fa26-maas-api-service-ca\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:09:47.331481 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:09:47.331453    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-62499f-db4mp"
Sep 11 19:09:47.331680 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:09:47.331450    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-62499f-db4mp" event={"ID":"d1ad35af-b5d3-4e95-9bc9-e275d0f5fa26","Type":"ContainerDied","Data":"79f880c0e6d4c0e12b128f71de7d8b0e18e11a40ba03e306e3e1f33d1eb593f0"}
Sep 11 19:09:47.331680 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:09:47.331589    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="79f880c0e6d4c0e12b128f71de7d8b0e18e11a40ba03e306e3e1f33d1eb593f0"
Sep 11 19:09:48.159941 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:09:48.159906    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-62499f/distinct-2-w0-62499f-kserve-76c9886944-rptqp"]
Sep 11 19:09:48.184069 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:09:48.184036    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-62499f/premium-w0-62499f-kserve-5f94979576-n2zc8"]
Sep 11 19:09:48.196334 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:09:48.196303    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-62499f/simulator-w0-62499f-kserve-6f58d77b77-h7rvf"]
Sep 11 19:09:48.724275 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:09:48.724243    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c1aebc/e2e-conflict-b-bb3c1a9d-kserve-6d468cf4c6-tmthf"]
Sep 11 19:09:59.547457 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:09:59.547420    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-62499f-db4mp"]
Sep 11 19:09:59.552839 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:09:59.552814    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-62499f-db4mp"]
Sep 11 19:10:00.396792 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:00.396752    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1ad35af-b5d3-4e95-9bc9-e275d0f5fa26" path="/var/lib/kubelet/pods/d1ad35af-b5d3-4e95-9bc9-e275d0f5fa26/volumes"
Sep 11 19:10:03.180563 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:03.180497    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-62499f/distinct-2-w0-62499f-kserve-76c9886944-rptqp" podUID="060d3a3b-64a2-44a8-9d0f-2e0388c60cee" containerName="main" containerID="cri-o://3169786a632b7d075dcc34efd1945462c5eb73037893c2b1ec9e7aae300caa3d" gracePeriod=45
Sep 11 19:10:03.204844 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:03.204779    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-62499f/premium-w0-62499f-kserve-5f94979576-n2zc8" podUID="e8034fa2-7642-4160-894e-c7ebbe3f7937" containerName="main" containerID="cri-o://2ff6d9dd3a45e134b32d1d2932f4bc2e6e6aae1a9cd0c6abb5235d67fc496cfe" gracePeriod=45
Sep 11 19:10:03.216007 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:03.215971    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-62499f/simulator-w0-62499f-kserve-6f58d77b77-h7rvf" podUID="64cba910-3466-40e1-bfc1-f5169c831845" containerName="main" containerID="cri-o://4dd2ac6e2972631acd0e508a7360751268470bf1c808b1ec06018a5bc3a84ced" gracePeriod=45
Sep 11 19:10:03.421641 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:03.421526    2578 generic.go:356] "Generic (PLEG): container finished" podID="e8034fa2-7642-4160-894e-c7ebbe3f7937" containerID="2ff6d9dd3a45e134b32d1d2932f4bc2e6e6aae1a9cd0c6abb5235d67fc496cfe" exitCode=0
Sep 11 19:10:03.421641 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:03.421549    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-62499f/premium-w0-62499f-kserve-5f94979576-n2zc8" event={"ID":"e8034fa2-7642-4160-894e-c7ebbe3f7937","Type":"ContainerDied","Data":"2ff6d9dd3a45e134b32d1d2932f4bc2e6e6aae1a9cd0c6abb5235d67fc496cfe"}
Sep 11 19:10:03.423657 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:03.423629    2578 generic.go:356] "Generic (PLEG): container finished" podID="060d3a3b-64a2-44a8-9d0f-2e0388c60cee" containerID="3169786a632b7d075dcc34efd1945462c5eb73037893c2b1ec9e7aae300caa3d" exitCode=0
Sep 11 19:10:03.423787 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:03.423708    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-62499f/distinct-2-w0-62499f-kserve-76c9886944-rptqp" event={"ID":"060d3a3b-64a2-44a8-9d0f-2e0388c60cee","Type":"ContainerDied","Data":"3169786a632b7d075dcc34efd1945462c5eb73037893c2b1ec9e7aae300caa3d"}
Sep 11 19:10:03.425608 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:03.425567    2578 generic.go:356] "Generic (PLEG): container finished" podID="64cba910-3466-40e1-bfc1-f5169c831845" containerID="4dd2ac6e2972631acd0e508a7360751268470bf1c808b1ec06018a5bc3a84ced" exitCode=0
Sep 11 19:10:03.425745 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:03.425614    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-62499f/simulator-w0-62499f-kserve-6f58d77b77-h7rvf" event={"ID":"64cba910-3466-40e1-bfc1-f5169c831845","Type":"ContainerDied","Data":"4dd2ac6e2972631acd0e508a7360751268470bf1c808b1ec06018a5bc3a84ced"}
Sep 11 19:10:03.485788 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:03.485757    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-62499f/distinct-2-w0-62499f-kserve-76c9886944-rptqp"
Sep 11 19:10:03.521562 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:03.521532    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-62499f/simulator-w0-62499f-kserve-6f58d77b77-h7rvf"
Sep 11 19:10:03.526184 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:03.526158    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-62499f/premium-w0-62499f-kserve-5f94979576-n2zc8"
Sep 11 19:10:03.565860 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:03.565749    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/060d3a3b-64a2-44a8-9d0f-2e0388c60cee-home\") pod \"060d3a3b-64a2-44a8-9d0f-2e0388c60cee\" (UID: \"060d3a3b-64a2-44a8-9d0f-2e0388c60cee\") "
Sep 11 19:10:03.566042 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:03.565870    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5cbxj\" (UniqueName: \"kubernetes.io/projected/060d3a3b-64a2-44a8-9d0f-2e0388c60cee-kube-api-access-5cbxj\") pod \"060d3a3b-64a2-44a8-9d0f-2e0388c60cee\" (UID: \"060d3a3b-64a2-44a8-9d0f-2e0388c60cee\") "
Sep 11 19:10:03.566111 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:03.566038    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/060d3a3b-64a2-44a8-9d0f-2e0388c60cee-home" (OuterVolumeSpecName: "home") pod "060d3a3b-64a2-44a8-9d0f-2e0388c60cee" (UID: "060d3a3b-64a2-44a8-9d0f-2e0388c60cee"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:10:03.566275 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:03.566249    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/060d3a3b-64a2-44a8-9d0f-2e0388c60cee-tls-certs\") pod \"060d3a3b-64a2-44a8-9d0f-2e0388c60cee\" (UID: \"060d3a3b-64a2-44a8-9d0f-2e0388c60cee\") "
Sep 11 19:10:03.566372 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:03.566354    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/060d3a3b-64a2-44a8-9d0f-2e0388c60cee-model-cache\") pod \"060d3a3b-64a2-44a8-9d0f-2e0388c60cee\" (UID: \"060d3a3b-64a2-44a8-9d0f-2e0388c60cee\") "
Sep 11 19:10:03.566682 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:03.566646    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/060d3a3b-64a2-44a8-9d0f-2e0388c60cee-model-cache" (OuterVolumeSpecName: "model-cache") pod "060d3a3b-64a2-44a8-9d0f-2e0388c60cee" (UID: "060d3a3b-64a2-44a8-9d0f-2e0388c60cee"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:10:03.567241 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:03.567212    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/060d3a3b-64a2-44a8-9d0f-2e0388c60cee-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "060d3a3b-64a2-44a8-9d0f-2e0388c60cee" (UID: "060d3a3b-64a2-44a8-9d0f-2e0388c60cee"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:10:03.567355 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:03.567008    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/060d3a3b-64a2-44a8-9d0f-2e0388c60cee-tmp-dir\") pod \"060d3a3b-64a2-44a8-9d0f-2e0388c60cee\" (UID: \"060d3a3b-64a2-44a8-9d0f-2e0388c60cee\") "
Sep 11 19:10:03.567495 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:03.567475    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/060d3a3b-64a2-44a8-9d0f-2e0388c60cee-dshm\") pod \"060d3a3b-64a2-44a8-9d0f-2e0388c60cee\" (UID: \"060d3a3b-64a2-44a8-9d0f-2e0388c60cee\") "
Sep 11 19:10:03.568289 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:03.568211    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/060d3a3b-64a2-44a8-9d0f-2e0388c60cee-model-cache\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:10:03.568289 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:03.568234    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/060d3a3b-64a2-44a8-9d0f-2e0388c60cee-tmp-dir\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:10:03.568289 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:03.568248    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/060d3a3b-64a2-44a8-9d0f-2e0388c60cee-home\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:10:03.568932 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:03.568912    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/060d3a3b-64a2-44a8-9d0f-2e0388c60cee-kube-api-access-5cbxj" (OuterVolumeSpecName: "kube-api-access-5cbxj") pod "060d3a3b-64a2-44a8-9d0f-2e0388c60cee" (UID: "060d3a3b-64a2-44a8-9d0f-2e0388c60cee"). InnerVolumeSpecName "kube-api-access-5cbxj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:10:03.569116 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:03.569089    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/060d3a3b-64a2-44a8-9d0f-2e0388c60cee-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "060d3a3b-64a2-44a8-9d0f-2e0388c60cee" (UID: "060d3a3b-64a2-44a8-9d0f-2e0388c60cee"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 19:10:03.569711 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:03.569683    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/060d3a3b-64a2-44a8-9d0f-2e0388c60cee-dshm" (OuterVolumeSpecName: "dshm") pod "060d3a3b-64a2-44a8-9d0f-2e0388c60cee" (UID: "060d3a3b-64a2-44a8-9d0f-2e0388c60cee"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:10:03.669168 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:03.669112    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ftxpw\" (UniqueName: \"kubernetes.io/projected/e8034fa2-7642-4160-894e-c7ebbe3f7937-kube-api-access-ftxpw\") pod \"e8034fa2-7642-4160-894e-c7ebbe3f7937\" (UID: \"e8034fa2-7642-4160-894e-c7ebbe3f7937\") "
Sep 11 19:10:03.669382 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:03.669182    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8034fa2-7642-4160-894e-c7ebbe3f7937-tls-certs\") pod \"e8034fa2-7642-4160-894e-c7ebbe3f7937\" (UID: \"e8034fa2-7642-4160-894e-c7ebbe3f7937\") "
Sep 11 19:10:03.669382 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:03.669214    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8034fa2-7642-4160-894e-c7ebbe3f7937-dshm\") pod \"e8034fa2-7642-4160-894e-c7ebbe3f7937\" (UID: \"e8034fa2-7642-4160-894e-c7ebbe3f7937\") "
Sep 11 19:10:03.669382 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:03.669249    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8034fa2-7642-4160-894e-c7ebbe3f7937-home\") pod \"e8034fa2-7642-4160-894e-c7ebbe3f7937\" (UID: \"e8034fa2-7642-4160-894e-c7ebbe3f7937\") "
Sep 11 19:10:03.669382 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:03.669329    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64cba910-3466-40e1-bfc1-f5169c831845-dshm\") pod \"64cba910-3466-40e1-bfc1-f5169c831845\" (UID: \"64cba910-3466-40e1-bfc1-f5169c831845\") "
Sep 11 19:10:03.669382 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:03.669369    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64cba910-3466-40e1-bfc1-f5169c831845-home\") pod \"64cba910-3466-40e1-bfc1-f5169c831845\" (UID: \"64cba910-3466-40e1-bfc1-f5169c831845\") "
Sep 11 19:10:03.669699 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:03.669416    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64cba910-3466-40e1-bfc1-f5169c831845-model-cache\") pod \"64cba910-3466-40e1-bfc1-f5169c831845\" (UID: \"64cba910-3466-40e1-bfc1-f5169c831845\") "
Sep 11 19:10:03.669699 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:03.669442    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8034fa2-7642-4160-894e-c7ebbe3f7937-model-cache\") pod \"e8034fa2-7642-4160-894e-c7ebbe3f7937\" (UID: \"e8034fa2-7642-4160-894e-c7ebbe3f7937\") "
Sep 11 19:10:03.669699 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:03.669474    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rsghq\" (UniqueName: \"kubernetes.io/projected/64cba910-3466-40e1-bfc1-f5169c831845-kube-api-access-rsghq\") pod \"64cba910-3466-40e1-bfc1-f5169c831845\" (UID: \"64cba910-3466-40e1-bfc1-f5169c831845\") "
Sep 11 19:10:03.669699 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:03.669498    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64cba910-3466-40e1-bfc1-f5169c831845-tls-certs\") pod \"64cba910-3466-40e1-bfc1-f5169c831845\" (UID: \"64cba910-3466-40e1-bfc1-f5169c831845\") "
Sep 11 19:10:03.669699 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:03.669542    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8034fa2-7642-4160-894e-c7ebbe3f7937-tmp-dir\") pod \"e8034fa2-7642-4160-894e-c7ebbe3f7937\" (UID: \"e8034fa2-7642-4160-894e-c7ebbe3f7937\") "
Sep 11 19:10:03.669699 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:03.669558    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8034fa2-7642-4160-894e-c7ebbe3f7937-home" (OuterVolumeSpecName: "home") pod "e8034fa2-7642-4160-894e-c7ebbe3f7937" (UID: "e8034fa2-7642-4160-894e-c7ebbe3f7937"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:10:03.669699 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:03.669643    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64cba910-3466-40e1-bfc1-f5169c831845-tmp-dir\") pod \"64cba910-3466-40e1-bfc1-f5169c831845\" (UID: \"64cba910-3466-40e1-bfc1-f5169c831845\") "
Sep 11 19:10:03.670046 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:03.669874    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64cba910-3466-40e1-bfc1-f5169c831845-model-cache" (OuterVolumeSpecName: "model-cache") pod "64cba910-3466-40e1-bfc1-f5169c831845" (UID: "64cba910-3466-40e1-bfc1-f5169c831845"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:10:03.670046 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:03.670028    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64cba910-3466-40e1-bfc1-f5169c831845-model-cache\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:10:03.670046 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:03.670045    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/060d3a3b-64a2-44a8-9d0f-2e0388c60cee-dshm\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:10:03.670204 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:03.670057    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8034fa2-7642-4160-894e-c7ebbe3f7937-home\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:10:03.670204 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:03.670070    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cbxj\" (UniqueName: \"kubernetes.io/projected/060d3a3b-64a2-44a8-9d0f-2e0388c60cee-kube-api-access-5cbxj\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:10:03.670204 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:03.670083    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/060d3a3b-64a2-44a8-9d0f-2e0388c60cee-tls-certs\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:10:03.670368 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:03.670249    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64cba910-3466-40e1-bfc1-f5169c831845-home" (OuterVolumeSpecName: "home") pod "64cba910-3466-40e1-bfc1-f5169c831845" (UID: "64cba910-3466-40e1-bfc1-f5169c831845"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:10:03.670368 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:03.670290    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64cba910-3466-40e1-bfc1-f5169c831845-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "64cba910-3466-40e1-bfc1-f5169c831845" (UID: "64cba910-3466-40e1-bfc1-f5169c831845"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:10:03.670473 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:03.670441    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8034fa2-7642-4160-894e-c7ebbe3f7937-model-cache" (OuterVolumeSpecName: "model-cache") pod "e8034fa2-7642-4160-894e-c7ebbe3f7937" (UID: "e8034fa2-7642-4160-894e-c7ebbe3f7937"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:10:03.670557 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:03.670530    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8034fa2-7642-4160-894e-c7ebbe3f7937-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e8034fa2-7642-4160-894e-c7ebbe3f7937" (UID: "e8034fa2-7642-4160-894e-c7ebbe3f7937"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:10:03.672707 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:03.672667    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8034fa2-7642-4160-894e-c7ebbe3f7937-dshm" (OuterVolumeSpecName: "dshm") pod "e8034fa2-7642-4160-894e-c7ebbe3f7937" (UID: "e8034fa2-7642-4160-894e-c7ebbe3f7937"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:10:03.672898 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:03.672665    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8034fa2-7642-4160-894e-c7ebbe3f7937-kube-api-access-ftxpw" (OuterVolumeSpecName: "kube-api-access-ftxpw") pod "e8034fa2-7642-4160-894e-c7ebbe3f7937" (UID: "e8034fa2-7642-4160-894e-c7ebbe3f7937"). InnerVolumeSpecName "kube-api-access-ftxpw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:10:03.672977 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:03.672958    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64cba910-3466-40e1-bfc1-f5169c831845-dshm" (OuterVolumeSpecName: "dshm") pod "64cba910-3466-40e1-bfc1-f5169c831845" (UID: "64cba910-3466-40e1-bfc1-f5169c831845"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:10:03.673088 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:03.673006    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8034fa2-7642-4160-894e-c7ebbe3f7937-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e8034fa2-7642-4160-894e-c7ebbe3f7937" (UID: "e8034fa2-7642-4160-894e-c7ebbe3f7937"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 19:10:03.673330 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:03.673305    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64cba910-3466-40e1-bfc1-f5169c831845-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "64cba910-3466-40e1-bfc1-f5169c831845" (UID: "64cba910-3466-40e1-bfc1-f5169c831845"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 19:10:03.673414 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:03.673353    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64cba910-3466-40e1-bfc1-f5169c831845-kube-api-access-rsghq" (OuterVolumeSpecName: "kube-api-access-rsghq") pod "64cba910-3466-40e1-bfc1-f5169c831845" (UID: "64cba910-3466-40e1-bfc1-f5169c831845"). InnerVolumeSpecName "kube-api-access-rsghq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:10:03.744514 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:03.744474    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-c1aebc/e2e-conflict-b-bb3c1a9d-kserve-6d468cf4c6-tmthf" podUID="f025a72f-bdab-4de7-b958-ddda7786c4fa" containerName="main" containerID="cri-o://c4b2100328493c2106c3d34d490a52ddd64b36060438af794cf61aaefd0de9f1" gracePeriod=45
Sep 11 19:10:03.771342 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:03.771306    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8034fa2-7642-4160-894e-c7ebbe3f7937-model-cache\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:10:03.771342 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:03.771339    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsghq\" (UniqueName: \"kubernetes.io/projected/64cba910-3466-40e1-bfc1-f5169c831845-kube-api-access-rsghq\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:10:03.771342 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:03.771349    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64cba910-3466-40e1-bfc1-f5169c831845-tls-certs\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:10:03.771564 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:03.771357    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8034fa2-7642-4160-894e-c7ebbe3f7937-tmp-dir\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:10:03.771564 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:03.771365    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64cba910-3466-40e1-bfc1-f5169c831845-tmp-dir\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:10:03.771564 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:03.771373    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftxpw\" (UniqueName: \"kubernetes.io/projected/e8034fa2-7642-4160-894e-c7ebbe3f7937-kube-api-access-ftxpw\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:10:03.771564 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:03.771381    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8034fa2-7642-4160-894e-c7ebbe3f7937-tls-certs\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:10:03.771564 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:03.771389    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8034fa2-7642-4160-894e-c7ebbe3f7937-dshm\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:10:03.771564 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:03.771395    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64cba910-3466-40e1-bfc1-f5169c831845-dshm\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:10:03.771564 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:03.771403    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64cba910-3466-40e1-bfc1-f5169c831845-home\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:10:03.988274 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:03.988246    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c1aebc/e2e-conflict-b-bb3c1a9d-kserve-6d468cf4c6-tmthf"
Sep 11 19:10:04.074313 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:04.074205    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f025a72f-bdab-4de7-b958-ddda7786c4fa-model-cache\") pod \"f025a72f-bdab-4de7-b958-ddda7786c4fa\" (UID: \"f025a72f-bdab-4de7-b958-ddda7786c4fa\") "
Sep 11 19:10:04.074313 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:04.074294    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f025a72f-bdab-4de7-b958-ddda7786c4fa-dshm\") pod \"f025a72f-bdab-4de7-b958-ddda7786c4fa\" (UID: \"f025a72f-bdab-4de7-b958-ddda7786c4fa\") "
Sep 11 19:10:04.074573 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:04.074319    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f025a72f-bdab-4de7-b958-ddda7786c4fa-home\") pod \"f025a72f-bdab-4de7-b958-ddda7786c4fa\" (UID: \"f025a72f-bdab-4de7-b958-ddda7786c4fa\") "
Sep 11 19:10:04.074573 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:04.074345    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f025a72f-bdab-4de7-b958-ddda7786c4fa-tmp-dir\") pod \"f025a72f-bdab-4de7-b958-ddda7786c4fa\" (UID: \"f025a72f-bdab-4de7-b958-ddda7786c4fa\") "
Sep 11 19:10:04.074573 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:04.074395    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f025a72f-bdab-4de7-b958-ddda7786c4fa-tls-certs\") pod \"f025a72f-bdab-4de7-b958-ddda7786c4fa\" (UID: \"f025a72f-bdab-4de7-b958-ddda7786c4fa\") "
Sep 11 19:10:04.074573 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:04.074427    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jpqx7\" (UniqueName: \"kubernetes.io/projected/f025a72f-bdab-4de7-b958-ddda7786c4fa-kube-api-access-jpqx7\") pod \"f025a72f-bdab-4de7-b958-ddda7786c4fa\" (UID: \"f025a72f-bdab-4de7-b958-ddda7786c4fa\") "
Sep 11 19:10:04.074573 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:04.074471    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f025a72f-bdab-4de7-b958-ddda7786c4fa-model-cache" (OuterVolumeSpecName: "model-cache") pod "f025a72f-bdab-4de7-b958-ddda7786c4fa" (UID: "f025a72f-bdab-4de7-b958-ddda7786c4fa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:10:04.074875 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:04.074635    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f025a72f-bdab-4de7-b958-ddda7786c4fa-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f025a72f-bdab-4de7-b958-ddda7786c4fa" (UID: "f025a72f-bdab-4de7-b958-ddda7786c4fa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:10:04.074875 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:04.074657    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f025a72f-bdab-4de7-b958-ddda7786c4fa-home" (OuterVolumeSpecName: "home") pod "f025a72f-bdab-4de7-b958-ddda7786c4fa" (UID: "f025a72f-bdab-4de7-b958-ddda7786c4fa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:10:04.074875 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:04.074832    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f025a72f-bdab-4de7-b958-ddda7786c4fa-home\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:10:04.074875 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:04.074846    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f025a72f-bdab-4de7-b958-ddda7786c4fa-tmp-dir\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:10:04.074875 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:04.074856    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f025a72f-bdab-4de7-b958-ddda7786c4fa-model-cache\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:10:04.076807 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:04.076774    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f025a72f-bdab-4de7-b958-ddda7786c4fa-dshm" (OuterVolumeSpecName: "dshm") pod "f025a72f-bdab-4de7-b958-ddda7786c4fa" (UID: "f025a72f-bdab-4de7-b958-ddda7786c4fa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:10:04.076916 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:04.076856    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f025a72f-bdab-4de7-b958-ddda7786c4fa-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f025a72f-bdab-4de7-b958-ddda7786c4fa" (UID: "f025a72f-bdab-4de7-b958-ddda7786c4fa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 19:10:04.076916 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:04.076864    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f025a72f-bdab-4de7-b958-ddda7786c4fa-kube-api-access-jpqx7" (OuterVolumeSpecName: "kube-api-access-jpqx7") pod "f025a72f-bdab-4de7-b958-ddda7786c4fa" (UID: "f025a72f-bdab-4de7-b958-ddda7786c4fa"). InnerVolumeSpecName "kube-api-access-jpqx7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:10:04.175639 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:04.175559    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f025a72f-bdab-4de7-b958-ddda7786c4fa-dshm\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:10:04.175639 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:04.175611    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f025a72f-bdab-4de7-b958-ddda7786c4fa-tls-certs\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:10:04.175639 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:04.175623    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpqx7\" (UniqueName: \"kubernetes.io/projected/f025a72f-bdab-4de7-b958-ddda7786c4fa-kube-api-access-jpqx7\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:10:04.431819 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:04.431792    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-62499f/premium-w0-62499f-kserve-5f94979576-n2zc8"
Sep 11 19:10:04.432283 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:04.431833    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-62499f/premium-w0-62499f-kserve-5f94979576-n2zc8" event={"ID":"e8034fa2-7642-4160-894e-c7ebbe3f7937","Type":"ContainerDied","Data":"f7e48a7fd7e48fd28f8233a154b48c9c6f02539bba96e0947d516b08fb38e025"}
Sep 11 19:10:04.432283 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:04.431886    2578 scope.go:117] "RemoveContainer" containerID="2ff6d9dd3a45e134b32d1d2932f4bc2e6e6aae1a9cd0c6abb5235d67fc496cfe"
Sep 11 19:10:04.433618 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:04.433575    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c1aebc/e2e-conflict-b-bb3c1a9d-kserve-6d468cf4c6-tmthf"
Sep 11 19:10:04.433618 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:04.433565    2578 generic.go:356] "Generic (PLEG): container finished" podID="f025a72f-bdab-4de7-b958-ddda7786c4fa" containerID="c4b2100328493c2106c3d34d490a52ddd64b36060438af794cf61aaefd0de9f1" exitCode=0
Sep 11 19:10:04.433781 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:04.433634    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c1aebc/e2e-conflict-b-bb3c1a9d-kserve-6d468cf4c6-tmthf" event={"ID":"f025a72f-bdab-4de7-b958-ddda7786c4fa","Type":"ContainerDied","Data":"c4b2100328493c2106c3d34d490a52ddd64b36060438af794cf61aaefd0de9f1"}
Sep 11 19:10:04.433781 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:04.433669    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c1aebc/e2e-conflict-b-bb3c1a9d-kserve-6d468cf4c6-tmthf" event={"ID":"f025a72f-bdab-4de7-b958-ddda7786c4fa","Type":"ContainerDied","Data":"825ba7426ee4d1a5d8371ab21fe165d30fa4fd9ee4de1a18182348a712ec5f87"}
Sep 11 19:10:04.435489 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:04.435468    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-62499f/distinct-2-w0-62499f-kserve-76c9886944-rptqp"
Sep 11 19:10:04.435654 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:04.435631    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-62499f/distinct-2-w0-62499f-kserve-76c9886944-rptqp" event={"ID":"060d3a3b-64a2-44a8-9d0f-2e0388c60cee","Type":"ContainerDied","Data":"e00af61e1a3af139579581e782a6aa098732052b01a211e314cf0430363a8697"}
Sep 11 19:10:04.437201 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:04.437173    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-62499f/simulator-w0-62499f-kserve-6f58d77b77-h7rvf"
Sep 11 19:10:04.437201 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:04.437179    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-62499f/simulator-w0-62499f-kserve-6f58d77b77-h7rvf" event={"ID":"64cba910-3466-40e1-bfc1-f5169c831845","Type":"ContainerDied","Data":"fe3c0dea8c84ec21f33493f7e0086dd53adbf6b591fb13bb6093b63901b2514e"}
Sep 11 19:10:04.443878 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:04.443858    2578 scope.go:117] "RemoveContainer" containerID="c4b2100328493c2106c3d34d490a52ddd64b36060438af794cf61aaefd0de9f1"
Sep 11 19:10:04.452217 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:04.452197    2578 scope.go:117] "RemoveContainer" containerID="c4b2100328493c2106c3d34d490a52ddd64b36060438af794cf61aaefd0de9f1"
Sep 11 19:10:04.452510 ip-10-0-134-189 kubenswrapper[2578]: E0911 19:10:04.452491    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4b2100328493c2106c3d34d490a52ddd64b36060438af794cf61aaefd0de9f1\": container with ID starting with c4b2100328493c2106c3d34d490a52ddd64b36060438af794cf61aaefd0de9f1 not found: ID does not exist" containerID="c4b2100328493c2106c3d34d490a52ddd64b36060438af794cf61aaefd0de9f1"
Sep 11 19:10:04.452584 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:04.452536    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4b2100328493c2106c3d34d490a52ddd64b36060438af794cf61aaefd0de9f1"} err="failed to get container status \"c4b2100328493c2106c3d34d490a52ddd64b36060438af794cf61aaefd0de9f1\": rpc error: code = NotFound desc = could not find container \"c4b2100328493c2106c3d34d490a52ddd64b36060438af794cf61aaefd0de9f1\": container with ID starting with c4b2100328493c2106c3d34d490a52ddd64b36060438af794cf61aaefd0de9f1 not found: ID does not exist"
Sep 11 19:10:04.452584 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:04.452553    2578 scope.go:117] "RemoveContainer" containerID="3169786a632b7d075dcc34efd1945462c5eb73037893c2b1ec9e7aae300caa3d"
Sep 11 19:10:04.456652 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:04.456626    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c1aebc/e2e-conflict-b-bb3c1a9d-kserve-6d468cf4c6-tmthf"]
Sep 11 19:10:04.461375 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:04.461282    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-c1aebc/e2e-conflict-b-bb3c1a9d-kserve-6d468cf4c6-tmthf"]
Sep 11 19:10:04.461430 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:04.461420    2578 scope.go:117] "RemoveContainer" containerID="4dd2ac6e2972631acd0e508a7360751268470bf1c808b1ec06018a5bc3a84ced"
Sep 11 19:10:04.485634 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:04.485581    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-62499f/distinct-2-w0-62499f-kserve-76c9886944-rptqp"]
Sep 11 19:10:04.489875 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:04.489847    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-62499f/distinct-2-w0-62499f-kserve-76c9886944-rptqp"]
Sep 11 19:10:04.504927 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:04.504897    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-62499f/simulator-w0-62499f-kserve-6f58d77b77-h7rvf"]
Sep 11 19:10:04.506772 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:04.506749    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-62499f/simulator-w0-62499f-kserve-6f58d77b77-h7rvf"]
Sep 11 19:10:04.517653 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:04.517629    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-62499f/premium-w0-62499f-kserve-5f94979576-n2zc8"]
Sep 11 19:10:04.524683 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:04.524655    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-62499f/premium-w0-62499f-kserve-5f94979576-n2zc8"]
Sep 11 19:10:06.396980 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:06.396929    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="060d3a3b-64a2-44a8-9d0f-2e0388c60cee" path="/var/lib/kubelet/pods/060d3a3b-64a2-44a8-9d0f-2e0388c60cee/volumes"
Sep 11 19:10:06.397393 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:06.397349    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64cba910-3466-40e1-bfc1-f5169c831845" path="/var/lib/kubelet/pods/64cba910-3466-40e1-bfc1-f5169c831845/volumes"
Sep 11 19:10:06.397738 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:06.397723    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8034fa2-7642-4160-894e-c7ebbe3f7937" path="/var/lib/kubelet/pods/e8034fa2-7642-4160-894e-c7ebbe3f7937/volumes"
Sep 11 19:10:06.398380 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:06.398365    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f025a72f-bdab-4de7-b958-ddda7786c4fa" path="/var/lib/kubelet/pods/f025a72f-bdab-4de7-b958-ddda7786c4fa/volumes"
Sep 11 19:10:14.737786 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:14.737750    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c1aebc/e2e-conflict-a-bb3c1a9d-kserve-6b8d5675c8-wzrlw"]
Sep 11 19:10:29.757212 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:29.757158    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-c1aebc/e2e-conflict-a-bb3c1a9d-kserve-6b8d5675c8-wzrlw" podUID="6b6675d9-8462-41df-aa72-45a9d176137c" containerName="main" containerID="cri-o://fbf74543c12723b8596ca48406bef2784161a37e719d1c1c26ba636c93778019" gracePeriod=45
Sep 11 19:10:30.013654 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:30.013567    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c1aebc/e2e-conflict-a-bb3c1a9d-kserve-6b8d5675c8-wzrlw"
Sep 11 19:10:30.123465 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:30.123418    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b6675d9-8462-41df-aa72-45a9d176137c-model-cache\") pod \"6b6675d9-8462-41df-aa72-45a9d176137c\" (UID: \"6b6675d9-8462-41df-aa72-45a9d176137c\") "
Sep 11 19:10:30.123465 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:30.123470    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b6675d9-8462-41df-aa72-45a9d176137c-tls-certs\") pod \"6b6675d9-8462-41df-aa72-45a9d176137c\" (UID: \"6b6675d9-8462-41df-aa72-45a9d176137c\") "
Sep 11 19:10:30.123761 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:30.123507    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b6675d9-8462-41df-aa72-45a9d176137c-tmp-dir\") pod \"6b6675d9-8462-41df-aa72-45a9d176137c\" (UID: \"6b6675d9-8462-41df-aa72-45a9d176137c\") "
Sep 11 19:10:30.123761 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:30.123572    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b6675d9-8462-41df-aa72-45a9d176137c-dshm\") pod \"6b6675d9-8462-41df-aa72-45a9d176137c\" (UID: \"6b6675d9-8462-41df-aa72-45a9d176137c\") "
Sep 11 19:10:30.123761 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:30.123642    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b6675d9-8462-41df-aa72-45a9d176137c-home\") pod \"6b6675d9-8462-41df-aa72-45a9d176137c\" (UID: \"6b6675d9-8462-41df-aa72-45a9d176137c\") "
Sep 11 19:10:30.123761 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:30.123681    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w5nd4\" (UniqueName: \"kubernetes.io/projected/6b6675d9-8462-41df-aa72-45a9d176137c-kube-api-access-w5nd4\") pod \"6b6675d9-8462-41df-aa72-45a9d176137c\" (UID: \"6b6675d9-8462-41df-aa72-45a9d176137c\") "
Sep 11 19:10:30.123761 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:30.123747    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b6675d9-8462-41df-aa72-45a9d176137c-model-cache" (OuterVolumeSpecName: "model-cache") pod "6b6675d9-8462-41df-aa72-45a9d176137c" (UID: "6b6675d9-8462-41df-aa72-45a9d176137c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:10:30.123992 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:30.123843    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b6675d9-8462-41df-aa72-45a9d176137c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6b6675d9-8462-41df-aa72-45a9d176137c" (UID: "6b6675d9-8462-41df-aa72-45a9d176137c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:10:30.123992 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:30.123921    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b6675d9-8462-41df-aa72-45a9d176137c-home" (OuterVolumeSpecName: "home") pod "6b6675d9-8462-41df-aa72-45a9d176137c" (UID: "6b6675d9-8462-41df-aa72-45a9d176137c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:10:30.124068 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:30.124047    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b6675d9-8462-41df-aa72-45a9d176137c-model-cache\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:10:30.124068 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:30.124061    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b6675d9-8462-41df-aa72-45a9d176137c-tmp-dir\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:10:30.124178 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:30.124070    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b6675d9-8462-41df-aa72-45a9d176137c-home\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:10:30.125934 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:30.125901    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b6675d9-8462-41df-aa72-45a9d176137c-dshm" (OuterVolumeSpecName: "dshm") pod "6b6675d9-8462-41df-aa72-45a9d176137c" (UID: "6b6675d9-8462-41df-aa72-45a9d176137c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:10:30.126056 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:30.125933    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b6675d9-8462-41df-aa72-45a9d176137c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6b6675d9-8462-41df-aa72-45a9d176137c" (UID: "6b6675d9-8462-41df-aa72-45a9d176137c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 19:10:30.126056 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:30.125948    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b6675d9-8462-41df-aa72-45a9d176137c-kube-api-access-w5nd4" (OuterVolumeSpecName: "kube-api-access-w5nd4") pod "6b6675d9-8462-41df-aa72-45a9d176137c" (UID: "6b6675d9-8462-41df-aa72-45a9d176137c"). InnerVolumeSpecName "kube-api-access-w5nd4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:10:30.224901 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:30.224857    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b6675d9-8462-41df-aa72-45a9d176137c-tls-certs\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:10:30.224901 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:30.224892    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b6675d9-8462-41df-aa72-45a9d176137c-dshm\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:10:30.224901 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:30.224907    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5nd4\" (UniqueName: \"kubernetes.io/projected/6b6675d9-8462-41df-aa72-45a9d176137c-kube-api-access-w5nd4\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:10:30.470747 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:30.470713    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ad53a9d-ws8mb"]
Sep 11 19:10:30.471104 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:30.471091    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b6675d9-8462-41df-aa72-45a9d176137c" containerName="main"
Sep 11 19:10:30.471153 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:30.471106    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b6675d9-8462-41df-aa72-45a9d176137c" containerName="main"
Sep 11 19:10:30.471153 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:30.471121    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64cba910-3466-40e1-bfc1-f5169c831845" containerName="main"
Sep 11 19:10:30.471153 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:30.471126    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="64cba910-3466-40e1-bfc1-f5169c831845" containerName="main"
Sep 11 19:10:30.471153 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:30.471138    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f025a72f-bdab-4de7-b958-ddda7786c4fa" containerName="main"
Sep 11 19:10:30.471153 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:30.471143    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f025a72f-bdab-4de7-b958-ddda7786c4fa" containerName="main"
Sep 11 19:10:30.471153 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:30.471153    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="060d3a3b-64a2-44a8-9d0f-2e0388c60cee" containerName="main"
Sep 11 19:10:30.471382 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:30.471159    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="060d3a3b-64a2-44a8-9d0f-2e0388c60cee" containerName="main"
Sep 11 19:10:30.471382 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:30.471169    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1ad35af-b5d3-4e95-9bc9-e275d0f5fa26" containerName="revoke-keys"
Sep 11 19:10:30.471382 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:30.471174    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1ad35af-b5d3-4e95-9bc9-e275d0f5fa26" containerName="revoke-keys"
Sep 11 19:10:30.471382 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:30.471180    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8034fa2-7642-4160-894e-c7ebbe3f7937" containerName="main"
Sep 11 19:10:30.471382 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:30.471186    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8034fa2-7642-4160-894e-c7ebbe3f7937" containerName="main"
Sep 11 19:10:30.471382 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:30.471245    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f025a72f-bdab-4de7-b958-ddda7786c4fa" containerName="main"
Sep 11 19:10:30.471382 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:30.471251    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="060d3a3b-64a2-44a8-9d0f-2e0388c60cee" containerName="main"
Sep 11 19:10:30.471382 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:30.471259    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="64cba910-3466-40e1-bfc1-f5169c831845" containerName="main"
Sep 11 19:10:30.471382 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:30.471264    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b6675d9-8462-41df-aa72-45a9d176137c" containerName="main"
Sep 11 19:10:30.471382 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:30.471270    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1ad35af-b5d3-4e95-9bc9-e275d0f5fa26" containerName="revoke-keys"
Sep 11 19:10:30.471382 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:30.471277    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8034fa2-7642-4160-894e-c7ebbe3f7937" containerName="main"
Sep 11 19:10:30.474537 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:30.474519    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ad53a9d-ws8mb"
Sep 11 19:10:30.476777 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:30.476752    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ln295\""
Sep 11 19:10:30.477109 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:30.477089    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 19:10:30.484691 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:30.484663    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ad53a9d-ws8mb"]
Sep 11 19:10:30.528504 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:30.528464    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd867c77-6b43-4691-9cde-0f3165eff7f4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-7ad53a9d-ws8mb\" (UID: \"bd867c77-6b43-4691-9cde-0f3165eff7f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ad53a9d-ws8mb"
Sep 11 19:10:30.576221 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:30.576185    2578 generic.go:356] "Generic (PLEG): container finished" podID="6b6675d9-8462-41df-aa72-45a9d176137c" containerID="fbf74543c12723b8596ca48406bef2784161a37e719d1c1c26ba636c93778019" exitCode=0
Sep 11 19:10:30.576427 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:30.576230    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c1aebc/e2e-conflict-a-bb3c1a9d-kserve-6b8d5675c8-wzrlw"
Sep 11 19:10:30.576427 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:30.576278    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c1aebc/e2e-conflict-a-bb3c1a9d-kserve-6b8d5675c8-wzrlw" event={"ID":"6b6675d9-8462-41df-aa72-45a9d176137c","Type":"ContainerDied","Data":"fbf74543c12723b8596ca48406bef2784161a37e719d1c1c26ba636c93778019"}
Sep 11 19:10:30.576427 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:30.576324    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c1aebc/e2e-conflict-a-bb3c1a9d-kserve-6b8d5675c8-wzrlw" event={"ID":"6b6675d9-8462-41df-aa72-45a9d176137c","Type":"ContainerDied","Data":"64667937c4554c6c90fd28231596b5dee84a5bb4426483675bd6ecb1273c32cd"}
Sep 11 19:10:30.576427 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:30.576347    2578 scope.go:117] "RemoveContainer" containerID="fbf74543c12723b8596ca48406bef2784161a37e719d1c1c26ba636c93778019"
Sep 11 19:10:30.585443 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:30.585425    2578 scope.go:117] "RemoveContainer" containerID="fbf74543c12723b8596ca48406bef2784161a37e719d1c1c26ba636c93778019"
Sep 11 19:10:30.585791 ip-10-0-134-189 kubenswrapper[2578]: E0911 19:10:30.585771    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbf74543c12723b8596ca48406bef2784161a37e719d1c1c26ba636c93778019\": container with ID starting with fbf74543c12723b8596ca48406bef2784161a37e719d1c1c26ba636c93778019 not found: ID does not exist" containerID="fbf74543c12723b8596ca48406bef2784161a37e719d1c1c26ba636c93778019"
Sep 11 19:10:30.585890 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:30.585799    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbf74543c12723b8596ca48406bef2784161a37e719d1c1c26ba636c93778019"} err="failed to get container status \"fbf74543c12723b8596ca48406bef2784161a37e719d1c1c26ba636c93778019\": rpc error: code = NotFound desc = could not find container \"fbf74543c12723b8596ca48406bef2784161a37e719d1c1c26ba636c93778019\": container with ID starting with fbf74543c12723b8596ca48406bef2784161a37e719d1c1c26ba636c93778019 not found: ID does not exist"
Sep 11 19:10:30.595430 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:30.595401    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c1aebc/e2e-conflict-a-bb3c1a9d-kserve-6b8d5675c8-wzrlw"]
Sep 11 19:10:30.602737 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:30.602713    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-c1aebc/e2e-conflict-a-bb3c1a9d-kserve-6b8d5675c8-wzrlw"]
Sep 11 19:10:30.629240 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:30.629209    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd867c77-6b43-4691-9cde-0f3165eff7f4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-7ad53a9d-ws8mb\" (UID: \"bd867c77-6b43-4691-9cde-0f3165eff7f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ad53a9d-ws8mb"
Sep 11 19:10:30.629895 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:30.629878    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd867c77-6b43-4691-9cde-0f3165eff7f4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-7ad53a9d-ws8mb\" (UID: \"bd867c77-6b43-4691-9cde-0f3165eff7f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ad53a9d-ws8mb"
Sep 11 19:10:30.785759 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:30.785717    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ad53a9d-ws8mb"
Sep 11 19:10:31.064245 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:31.064219    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ad53a9d-ws8mb"]
Sep 11 19:10:31.067010 ip-10-0-134-189 kubenswrapper[2578]: W0911 19:10:31.066987    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd867c77_6b43_4691_9cde_0f3165eff7f4.slice/crio-44aee542577f87341e628b4fc4a39a23adb8f4850a2141c7062cd00623525616 WatchSource:0}: Error finding container 44aee542577f87341e628b4fc4a39a23adb8f4850a2141c7062cd00623525616: Status 404 returned error can't find the container with id 44aee542577f87341e628b4fc4a39a23adb8f4850a2141c7062cd00623525616
Sep 11 19:10:31.589626 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:31.589570    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ad53a9d-ws8mb" event={"ID":"bd867c77-6b43-4691-9cde-0f3165eff7f4","Type":"ContainerStarted","Data":"127d82a9f24eb91b1d2f434140c4cf88836516909d426580e99fbfa9ff19ab6f"}
Sep 11 19:10:31.589832 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:31.589685    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ad53a9d-ws8mb" event={"ID":"bd867c77-6b43-4691-9cde-0f3165eff7f4","Type":"ContainerStarted","Data":"44aee542577f87341e628b4fc4a39a23adb8f4850a2141c7062cd00623525616"}
Sep 11 19:10:31.611643 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:31.611573    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ad53a9d-ws8mb" podStartSLOduration=1.611556325 podStartE2EDuration="1.611556325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:10:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:10:31.609627043 +0000 UTC m=+1307.687217609" watchObservedRunningTime="2026-09-11 19:10:31.611556325 +0000 UTC m=+1307.689146928"
Sep 11 19:10:32.397702 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:32.397666    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b6675d9-8462-41df-aa72-45a9d176137c" path="/var/lib/kubelet/pods/6b6675d9-8462-41df-aa72-45a9d176137c/volumes"
Sep 11 19:10:32.597015 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:32.596980    2578 generic.go:356] "Generic (PLEG): container finished" podID="bd867c77-6b43-4691-9cde-0f3165eff7f4" containerID="127d82a9f24eb91b1d2f434140c4cf88836516909d426580e99fbfa9ff19ab6f" exitCode=0
Sep 11 19:10:32.597228 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:32.597037    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ad53a9d-ws8mb" event={"ID":"bd867c77-6b43-4691-9cde-0f3165eff7f4","Type":"ContainerDied","Data":"127d82a9f24eb91b1d2f434140c4cf88836516909d426580e99fbfa9ff19ab6f"}
Sep 11 19:10:33.745667 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:33.745637    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ad53a9d-ws8mb"
Sep 11 19:10:33.861257 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:33.861153    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd867c77-6b43-4691-9cde-0f3165eff7f4-maas-api-service-ca\") pod \"bd867c77-6b43-4691-9cde-0f3165eff7f4\" (UID: \"bd867c77-6b43-4691-9cde-0f3165eff7f4\") "
Sep 11 19:10:33.861517 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:33.861492    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd867c77-6b43-4691-9cde-0f3165eff7f4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bd867c77-6b43-4691-9cde-0f3165eff7f4" (UID: "bd867c77-6b43-4691-9cde-0f3165eff7f4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 19:10:33.962242 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:33.962204    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd867c77-6b43-4691-9cde-0f3165eff7f4-maas-api-service-ca\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:10:34.609848 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:34.609814    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ad53a9d-ws8mb" event={"ID":"bd867c77-6b43-4691-9cde-0f3165eff7f4","Type":"ContainerDied","Data":"44aee542577f87341e628b4fc4a39a23adb8f4850a2141c7062cd00623525616"}
Sep 11 19:10:34.609848 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:34.609839    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ad53a9d-ws8mb"
Sep 11 19:10:34.610106 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:34.609869    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="44aee542577f87341e628b4fc4a39a23adb8f4850a2141c7062cd00623525616"
Sep 11 19:10:46.227035 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:46.226995    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7ad53a9d-57cd7b475c-mzpjg"]
Sep 11 19:10:46.227516 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:46.227269    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-7ad53a9d-57cd7b475c-mzpjg" podUID="570f6c2f-c791-4e27-82c1-24b1b4d8b88a" containerName="payload-processing" containerID="cri-o://974f2960dbc5ae11248431564f5a97b16e8bd1b140f70e9d54e82f5f9b3ed5f7" gracePeriod=30
Sep 11 19:10:46.236530 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:46.236499    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7ad53a9d-8c87b79f4-hzpqk"]
Sep 11 19:10:46.236736 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:46.236715    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-7ad53a9d-8c87b79f4-hzpqk" podUID="c67d229b-1c73-44f8-953b-8f4e79a6e3a1" containerName="payload-pre-processing" containerID="cri-o://f8b7940ff55253f30b4d3b85022c12d870c042e8b6c14d6404bc60d497834226" gracePeriod=30
Sep 11 19:10:46.459520 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:46.459482    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ad53a9d-ws8mb"]
Sep 11 19:10:46.465910 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:46.465877    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ad53a9d-ws8mb"]
Sep 11 19:10:46.497783 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:46.497758    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7ad53a9d-57cd7b475c-mzpjg"
Sep 11 19:10:46.531057 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:46.531029    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7ad53a9d-8c87b79f4-hzpqk"
Sep 11 19:10:46.583903 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:46.583865    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nq94h\" (UniqueName: \"kubernetes.io/projected/570f6c2f-c791-4e27-82c1-24b1b4d8b88a-kube-api-access-nq94h\") pod \"570f6c2f-c791-4e27-82c1-24b1b4d8b88a\" (UID: \"570f6c2f-c791-4e27-82c1-24b1b4d8b88a\") "
Sep 11 19:10:46.584063 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:46.583928    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/570f6c2f-c791-4e27-82c1-24b1b4d8b88a-plugins-config-volume\") pod \"570f6c2f-c791-4e27-82c1-24b1b4d8b88a\" (UID: \"570f6c2f-c791-4e27-82c1-24b1b4d8b88a\") "
Sep 11 19:10:46.584340 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:46.584315    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/570f6c2f-c791-4e27-82c1-24b1b4d8b88a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "570f6c2f-c791-4e27-82c1-24b1b4d8b88a" (UID: "570f6c2f-c791-4e27-82c1-24b1b4d8b88a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 19:10:46.586214 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:46.586195    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/570f6c2f-c791-4e27-82c1-24b1b4d8b88a-kube-api-access-nq94h" (OuterVolumeSpecName: "kube-api-access-nq94h") pod "570f6c2f-c791-4e27-82c1-24b1b4d8b88a" (UID: "570f6c2f-c791-4e27-82c1-24b1b4d8b88a"). InnerVolumeSpecName "kube-api-access-nq94h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:10:46.672506 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:46.672468    2578 generic.go:356] "Generic (PLEG): container finished" podID="c67d229b-1c73-44f8-953b-8f4e79a6e3a1" containerID="f8b7940ff55253f30b4d3b85022c12d870c042e8b6c14d6404bc60d497834226" exitCode=0
Sep 11 19:10:46.672730 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:46.672515    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7ad53a9d-8c87b79f4-hzpqk"
Sep 11 19:10:46.672730 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:46.672546    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7ad53a9d-8c87b79f4-hzpqk" event={"ID":"c67d229b-1c73-44f8-953b-8f4e79a6e3a1","Type":"ContainerDied","Data":"f8b7940ff55253f30b4d3b85022c12d870c042e8b6c14d6404bc60d497834226"}
Sep 11 19:10:46.672730 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:46.672624    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7ad53a9d-8c87b79f4-hzpqk" event={"ID":"c67d229b-1c73-44f8-953b-8f4e79a6e3a1","Type":"ContainerDied","Data":"227a7b7f18baac42b0941c256deb9e1f5f2ed3539736e6a8a7d37174ab8c31fa"}
Sep 11 19:10:46.672730 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:46.672657    2578 scope.go:117] "RemoveContainer" containerID="f8b7940ff55253f30b4d3b85022c12d870c042e8b6c14d6404bc60d497834226"
Sep 11 19:10:46.674179 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:46.674153    2578 generic.go:356] "Generic (PLEG): container finished" podID="570f6c2f-c791-4e27-82c1-24b1b4d8b88a" containerID="974f2960dbc5ae11248431564f5a97b16e8bd1b140f70e9d54e82f5f9b3ed5f7" exitCode=0
Sep 11 19:10:46.674296 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:46.674188    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7ad53a9d-57cd7b475c-mzpjg"
Sep 11 19:10:46.674296 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:46.674247    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7ad53a9d-57cd7b475c-mzpjg" event={"ID":"570f6c2f-c791-4e27-82c1-24b1b4d8b88a","Type":"ContainerDied","Data":"974f2960dbc5ae11248431564f5a97b16e8bd1b140f70e9d54e82f5f9b3ed5f7"}
Sep 11 19:10:46.674389 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:46.674293    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7ad53a9d-57cd7b475c-mzpjg" event={"ID":"570f6c2f-c791-4e27-82c1-24b1b4d8b88a","Type":"ContainerDied","Data":"7e6878f1e5b46e5bdcd0dab85814fdb834df539b9e0c3213203c861bfdb53c4b"}
Sep 11 19:10:46.683179 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:46.683155    2578 scope.go:117] "RemoveContainer" containerID="f8b7940ff55253f30b4d3b85022c12d870c042e8b6c14d6404bc60d497834226"
Sep 11 19:10:46.683527 ip-10-0-134-189 kubenswrapper[2578]: E0911 19:10:46.683488    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8b7940ff55253f30b4d3b85022c12d870c042e8b6c14d6404bc60d497834226\": container with ID starting with f8b7940ff55253f30b4d3b85022c12d870c042e8b6c14d6404bc60d497834226 not found: ID does not exist" containerID="f8b7940ff55253f30b4d3b85022c12d870c042e8b6c14d6404bc60d497834226"
Sep 11 19:10:46.683643 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:46.683534    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8b7940ff55253f30b4d3b85022c12d870c042e8b6c14d6404bc60d497834226"} err="failed to get container status \"f8b7940ff55253f30b4d3b85022c12d870c042e8b6c14d6404bc60d497834226\": rpc error: code = NotFound desc = could not find container \"f8b7940ff55253f30b4d3b85022c12d870c042e8b6c14d6404bc60d497834226\": container with ID starting with f8b7940ff55253f30b4d3b85022c12d870c042e8b6c14d6404bc60d497834226 not found: ID does not exist"
Sep 11 19:10:46.683643 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:46.683552    2578 scope.go:117] "RemoveContainer" containerID="974f2960dbc5ae11248431564f5a97b16e8bd1b140f70e9d54e82f5f9b3ed5f7"
Sep 11 19:10:46.684545 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:46.684530    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c67d229b-1c73-44f8-953b-8f4e79a6e3a1-plugins-config-volume\") pod \"c67d229b-1c73-44f8-953b-8f4e79a6e3a1\" (UID: \"c67d229b-1c73-44f8-953b-8f4e79a6e3a1\") "
Sep 11 19:10:46.684637 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:46.684555    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b8qj7\" (UniqueName: \"kubernetes.io/projected/c67d229b-1c73-44f8-953b-8f4e79a6e3a1-kube-api-access-b8qj7\") pod \"c67d229b-1c73-44f8-953b-8f4e79a6e3a1\" (UID: \"c67d229b-1c73-44f8-953b-8f4e79a6e3a1\") "
Sep 11 19:10:46.684948 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:46.684926    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nq94h\" (UniqueName: \"kubernetes.io/projected/570f6c2f-c791-4e27-82c1-24b1b4d8b88a-kube-api-access-nq94h\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:10:46.684948 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:46.684936    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c67d229b-1c73-44f8-953b-8f4e79a6e3a1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c67d229b-1c73-44f8-953b-8f4e79a6e3a1" (UID: "c67d229b-1c73-44f8-953b-8f4e79a6e3a1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 19:10:46.685077 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:46.684952    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/570f6c2f-c791-4e27-82c1-24b1b4d8b88a-plugins-config-volume\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:10:46.686845 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:46.686821    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c67d229b-1c73-44f8-953b-8f4e79a6e3a1-kube-api-access-b8qj7" (OuterVolumeSpecName: "kube-api-access-b8qj7") pod "c67d229b-1c73-44f8-953b-8f4e79a6e3a1" (UID: "c67d229b-1c73-44f8-953b-8f4e79a6e3a1"). InnerVolumeSpecName "kube-api-access-b8qj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:10:46.691553 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:46.691537    2578 scope.go:117] "RemoveContainer" containerID="974f2960dbc5ae11248431564f5a97b16e8bd1b140f70e9d54e82f5f9b3ed5f7"
Sep 11 19:10:46.691868 ip-10-0-134-189 kubenswrapper[2578]: E0911 19:10:46.691847    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"974f2960dbc5ae11248431564f5a97b16e8bd1b140f70e9d54e82f5f9b3ed5f7\": container with ID starting with 974f2960dbc5ae11248431564f5a97b16e8bd1b140f70e9d54e82f5f9b3ed5f7 not found: ID does not exist" containerID="974f2960dbc5ae11248431564f5a97b16e8bd1b140f70e9d54e82f5f9b3ed5f7"
Sep 11 19:10:46.691932 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:46.691875    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"974f2960dbc5ae11248431564f5a97b16e8bd1b140f70e9d54e82f5f9b3ed5f7"} err="failed to get container status \"974f2960dbc5ae11248431564f5a97b16e8bd1b140f70e9d54e82f5f9b3ed5f7\": rpc error: code = NotFound desc = could not find container \"974f2960dbc5ae11248431564f5a97b16e8bd1b140f70e9d54e82f5f9b3ed5f7\": container with ID starting with 974f2960dbc5ae11248431564f5a97b16e8bd1b140f70e9d54e82f5f9b3ed5f7 not found: ID does not exist"
Sep 11 19:10:46.698130 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:46.698107    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7ad53a9d-57cd7b475c-mzpjg"]
Sep 11 19:10:46.704286 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:46.704261    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7ad53a9d-57cd7b475c-mzpjg"]
Sep 11 19:10:46.785610 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:46.785494    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c67d229b-1c73-44f8-953b-8f4e79a6e3a1-plugins-config-volume\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:10:46.785610 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:46.785529    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8qj7\" (UniqueName: \"kubernetes.io/projected/c67d229b-1c73-44f8-953b-8f4e79a6e3a1-kube-api-access-b8qj7\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:10:46.997519 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:46.997484    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7ad53a9d-8c87b79f4-hzpqk"]
Sep 11 19:10:47.004671 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:47.004643    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7ad53a9d-8c87b79f4-hzpqk"]
Sep 11 19:10:48.397016 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:48.396978    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="570f6c2f-c791-4e27-82c1-24b1b4d8b88a" path="/var/lib/kubelet/pods/570f6c2f-c791-4e27-82c1-24b1b4d8b88a/volumes"
Sep 11 19:10:48.397405 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:48.397312    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd867c77-6b43-4691-9cde-0f3165eff7f4" path="/var/lib/kubelet/pods/bd867c77-6b43-4691-9cde-0f3165eff7f4/volumes"
Sep 11 19:10:48.397644 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:48.397630    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c67d229b-1c73-44f8-953b-8f4e79a6e3a1" path="/var/lib/kubelet/pods/c67d229b-1c73-44f8-953b-8f4e79a6e3a1/volumes"
Sep 11 19:10:54.571929 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:10:54.571891    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-7ad53a9d/e2e-lifecycle-model-7ad53a9d-kserve-86f545dbc9-zlh6b"]
Sep 11 19:11:01.036850 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:01.036817    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-c1aebc/distinct-w4-c1aebc-kserve-6fd464965d-rw6jw"]
Sep 11 19:11:01.037299 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:01.037186    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c67d229b-1c73-44f8-953b-8f4e79a6e3a1" containerName="payload-pre-processing"
Sep 11 19:11:01.037299 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:01.037199    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c67d229b-1c73-44f8-953b-8f4e79a6e3a1" containerName="payload-pre-processing"
Sep 11 19:11:01.037299 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:01.037210    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="570f6c2f-c791-4e27-82c1-24b1b4d8b88a" containerName="payload-processing"
Sep 11 19:11:01.037299 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:01.037217    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="570f6c2f-c791-4e27-82c1-24b1b4d8b88a" containerName="payload-processing"
Sep 11 19:11:01.037299 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:01.037222    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd867c77-6b43-4691-9cde-0f3165eff7f4" containerName="revoke-keys"
Sep 11 19:11:01.037299 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:01.037229    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd867c77-6b43-4691-9cde-0f3165eff7f4" containerName="revoke-keys"
Sep 11 19:11:01.037299 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:01.037302    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c67d229b-1c73-44f8-953b-8f4e79a6e3a1" containerName="payload-pre-processing"
Sep 11 19:11:01.037541 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:01.037317    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd867c77-6b43-4691-9cde-0f3165eff7f4" containerName="revoke-keys"
Sep 11 19:11:01.037541 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:01.037325    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="570f6c2f-c791-4e27-82c1-24b1b4d8b88a" containerName="payload-processing"
Sep 11 19:11:01.041823 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:01.041804    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c1aebc/distinct-w4-c1aebc-kserve-6fd464965d-rw6jw"
Sep 11 19:11:01.044435 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:01.044412    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-c1aebc\"/\"distinct-w4-c1aebc-kserve-self-signed-certs\""
Sep 11 19:11:01.054328 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:01.054302    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c1aebc/distinct-w4-c1aebc-kserve-6fd464965d-rw6jw"]
Sep 11 19:11:01.123789 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:01.123749    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48g7x\" (UniqueName: \"kubernetes.io/projected/373837e6-d918-4371-a7f4-52e52ce098f8-kube-api-access-48g7x\") pod \"distinct-w4-c1aebc-kserve-6fd464965d-rw6jw\" (UID: \"373837e6-d918-4371-a7f4-52e52ce098f8\") " pod="e2e-models-e2e-worker-w4-c1aebc/distinct-w4-c1aebc-kserve-6fd464965d-rw6jw"
Sep 11 19:11:01.123973 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:01.123818    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/373837e6-d918-4371-a7f4-52e52ce098f8-tls-certs\") pod \"distinct-w4-c1aebc-kserve-6fd464965d-rw6jw\" (UID: \"373837e6-d918-4371-a7f4-52e52ce098f8\") " pod="e2e-models-e2e-worker-w4-c1aebc/distinct-w4-c1aebc-kserve-6fd464965d-rw6jw"
Sep 11 19:11:01.123973 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:01.123890    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/373837e6-d918-4371-a7f4-52e52ce098f8-tmp-dir\") pod \"distinct-w4-c1aebc-kserve-6fd464965d-rw6jw\" (UID: \"373837e6-d918-4371-a7f4-52e52ce098f8\") " pod="e2e-models-e2e-worker-w4-c1aebc/distinct-w4-c1aebc-kserve-6fd464965d-rw6jw"
Sep 11 19:11:01.123973 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:01.123919    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/373837e6-d918-4371-a7f4-52e52ce098f8-home\") pod \"distinct-w4-c1aebc-kserve-6fd464965d-rw6jw\" (UID: \"373837e6-d918-4371-a7f4-52e52ce098f8\") " pod="e2e-models-e2e-worker-w4-c1aebc/distinct-w4-c1aebc-kserve-6fd464965d-rw6jw"
Sep 11 19:11:01.123973 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:01.123946    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/373837e6-d918-4371-a7f4-52e52ce098f8-model-cache\") pod \"distinct-w4-c1aebc-kserve-6fd464965d-rw6jw\" (UID: \"373837e6-d918-4371-a7f4-52e52ce098f8\") " pod="e2e-models-e2e-worker-w4-c1aebc/distinct-w4-c1aebc-kserve-6fd464965d-rw6jw"
Sep 11 19:11:01.124120 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:01.124023    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/373837e6-d918-4371-a7f4-52e52ce098f8-dshm\") pod \"distinct-w4-c1aebc-kserve-6fd464965d-rw6jw\" (UID: \"373837e6-d918-4371-a7f4-52e52ce098f8\") " pod="e2e-models-e2e-worker-w4-c1aebc/distinct-w4-c1aebc-kserve-6fd464965d-rw6jw"
Sep 11 19:11:01.225060 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:01.225018    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/373837e6-d918-4371-a7f4-52e52ce098f8-dshm\") pod \"distinct-w4-c1aebc-kserve-6fd464965d-rw6jw\" (UID: \"373837e6-d918-4371-a7f4-52e52ce098f8\") " pod="e2e-models-e2e-worker-w4-c1aebc/distinct-w4-c1aebc-kserve-6fd464965d-rw6jw"
Sep 11 19:11:01.225268 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:01.225072    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48g7x\" (UniqueName: \"kubernetes.io/projected/373837e6-d918-4371-a7f4-52e52ce098f8-kube-api-access-48g7x\") pod \"distinct-w4-c1aebc-kserve-6fd464965d-rw6jw\" (UID: \"373837e6-d918-4371-a7f4-52e52ce098f8\") " pod="e2e-models-e2e-worker-w4-c1aebc/distinct-w4-c1aebc-kserve-6fd464965d-rw6jw"
Sep 11 19:11:01.225268 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:01.225125    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/373837e6-d918-4371-a7f4-52e52ce098f8-tls-certs\") pod \"distinct-w4-c1aebc-kserve-6fd464965d-rw6jw\" (UID: \"373837e6-d918-4371-a7f4-52e52ce098f8\") " pod="e2e-models-e2e-worker-w4-c1aebc/distinct-w4-c1aebc-kserve-6fd464965d-rw6jw"
Sep 11 19:11:01.225268 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:01.225149    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/373837e6-d918-4371-a7f4-52e52ce098f8-tmp-dir\") pod \"distinct-w4-c1aebc-kserve-6fd464965d-rw6jw\" (UID: \"373837e6-d918-4371-a7f4-52e52ce098f8\") " pod="e2e-models-e2e-worker-w4-c1aebc/distinct-w4-c1aebc-kserve-6fd464965d-rw6jw"
Sep 11 19:11:01.225268 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:01.225175    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/373837e6-d918-4371-a7f4-52e52ce098f8-home\") pod \"distinct-w4-c1aebc-kserve-6fd464965d-rw6jw\" (UID: \"373837e6-d918-4371-a7f4-52e52ce098f8\") " pod="e2e-models-e2e-worker-w4-c1aebc/distinct-w4-c1aebc-kserve-6fd464965d-rw6jw"
Sep 11 19:11:01.225268 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:01.225204    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/373837e6-d918-4371-a7f4-52e52ce098f8-model-cache\") pod \"distinct-w4-c1aebc-kserve-6fd464965d-rw6jw\" (UID: \"373837e6-d918-4371-a7f4-52e52ce098f8\") " pod="e2e-models-e2e-worker-w4-c1aebc/distinct-w4-c1aebc-kserve-6fd464965d-rw6jw"
Sep 11 19:11:01.225644 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:01.225583    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/373837e6-d918-4371-a7f4-52e52ce098f8-tmp-dir\") pod \"distinct-w4-c1aebc-kserve-6fd464965d-rw6jw\" (UID: \"373837e6-d918-4371-a7f4-52e52ce098f8\") " pod="e2e-models-e2e-worker-w4-c1aebc/distinct-w4-c1aebc-kserve-6fd464965d-rw6jw"
Sep 11 19:11:01.225773 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:01.225645    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/373837e6-d918-4371-a7f4-52e52ce098f8-model-cache\") pod \"distinct-w4-c1aebc-kserve-6fd464965d-rw6jw\" (UID: \"373837e6-d918-4371-a7f4-52e52ce098f8\") " pod="e2e-models-e2e-worker-w4-c1aebc/distinct-w4-c1aebc-kserve-6fd464965d-rw6jw"
Sep 11 19:11:01.225773 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:01.225621    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/373837e6-d918-4371-a7f4-52e52ce098f8-home\") pod \"distinct-w4-c1aebc-kserve-6fd464965d-rw6jw\" (UID: \"373837e6-d918-4371-a7f4-52e52ce098f8\") " pod="e2e-models-e2e-worker-w4-c1aebc/distinct-w4-c1aebc-kserve-6fd464965d-rw6jw"
Sep 11 19:11:01.227667 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:01.227640    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/373837e6-d918-4371-a7f4-52e52ce098f8-dshm\") pod \"distinct-w4-c1aebc-kserve-6fd464965d-rw6jw\" (UID: \"373837e6-d918-4371-a7f4-52e52ce098f8\") " pod="e2e-models-e2e-worker-w4-c1aebc/distinct-w4-c1aebc-kserve-6fd464965d-rw6jw"
Sep 11 19:11:01.227889 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:01.227870    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/373837e6-d918-4371-a7f4-52e52ce098f8-tls-certs\") pod \"distinct-w4-c1aebc-kserve-6fd464965d-rw6jw\" (UID: \"373837e6-d918-4371-a7f4-52e52ce098f8\") " pod="e2e-models-e2e-worker-w4-c1aebc/distinct-w4-c1aebc-kserve-6fd464965d-rw6jw"
Sep 11 19:11:01.234576 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:01.234556    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48g7x\" (UniqueName: \"kubernetes.io/projected/373837e6-d918-4371-a7f4-52e52ce098f8-kube-api-access-48g7x\") pod \"distinct-w4-c1aebc-kserve-6fd464965d-rw6jw\" (UID: \"373837e6-d918-4371-a7f4-52e52ce098f8\") " pod="e2e-models-e2e-worker-w4-c1aebc/distinct-w4-c1aebc-kserve-6fd464965d-rw6jw"
Sep 11 19:11:01.351255 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:01.351150    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c1aebc/distinct-w4-c1aebc-kserve-6fd464965d-rw6jw"
Sep 11 19:11:01.631797 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:01.631733    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c1aebc/distinct-w4-c1aebc-kserve-6fd464965d-rw6jw"]
Sep 11 19:11:01.634090 ip-10-0-134-189 kubenswrapper[2578]: W0911 19:11:01.634061    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod373837e6_d918_4371_a7f4_52e52ce098f8.slice/crio-21314396e6275dd68e09a3d1e6218737d0401affb7e525ff89e422cebe31ed9a WatchSource:0}: Error finding container 21314396e6275dd68e09a3d1e6218737d0401affb7e525ff89e422cebe31ed9a: Status 404 returned error can't find the container with id 21314396e6275dd68e09a3d1e6218737d0401affb7e525ff89e422cebe31ed9a
Sep 11 19:11:01.752177 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:01.752133    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c1aebc/distinct-w4-c1aebc-kserve-6fd464965d-rw6jw" event={"ID":"373837e6-d918-4371-a7f4-52e52ce098f8","Type":"ContainerStarted","Data":"c1b8a55e6937a23f5257185c9f7b4ed2929cedb0d2ce1f43d2549aa1ee998325"}
Sep 11 19:11:01.752354 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:01.752193    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c1aebc/distinct-w4-c1aebc-kserve-6fd464965d-rw6jw" event={"ID":"373837e6-d918-4371-a7f4-52e52ce098f8","Type":"ContainerStarted","Data":"21314396e6275dd68e09a3d1e6218737d0401affb7e525ff89e422cebe31ed9a"}
Sep 11 19:11:01.777130 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:01.777074    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-c1aebc/distinct-w4-c1aebc-kserve-6fd464965d-rw6jw" podStartSLOduration=0.777056543 podStartE2EDuration="777.056543ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:11:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:11:01.775729117 +0000 UTC m=+1337.853319672" watchObservedRunningTime="2026-09-11 19:11:01.777056543 +0000 UTC m=+1337.854647109"
Sep 11 19:11:05.940775 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:05.940694    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-3b7391/auth-test-model-3b7391-kserve-74db88fc4c-smtts"]
Sep 11 19:11:05.944181 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:05.944162    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3b7391/auth-test-model-3b7391-kserve-74db88fc4c-smtts"
Sep 11 19:11:05.946522 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:05.946495    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-3b7391\"/\"default-dockercfg-296hb\""
Sep 11 19:11:05.947215 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:05.947191    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-3b7391\"/\"openshift-service-ca.crt\""
Sep 11 19:11:05.947318 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:05.947220    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-3b7391\"/\"auth-test-model-3b7391-kserve-self-signed-certs\""
Sep 11 19:11:05.948898 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:05.948873    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-3b7391\"/\"kube-root-ca.crt\""
Sep 11 19:11:05.956459 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:05.956437    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3b7391/auth-test-model-3b7391-kserve-74db88fc4c-smtts"]
Sep 11 19:11:05.971877 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:05.971844    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d0be388-987b-417e-85b1-0eaf6bb428b3-tmp-dir\") pod \"auth-test-model-3b7391-kserve-74db88fc4c-smtts\" (UID: \"3d0be388-987b-417e-85b1-0eaf6bb428b3\") " pod="ai-tenant-e2e-shared-b-w3-3b7391/auth-test-model-3b7391-kserve-74db88fc4c-smtts"
Sep 11 19:11:05.972069 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:05.971887    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d0be388-987b-417e-85b1-0eaf6bb428b3-dshm\") pod \"auth-test-model-3b7391-kserve-74db88fc4c-smtts\" (UID: \"3d0be388-987b-417e-85b1-0eaf6bb428b3\") " pod="ai-tenant-e2e-shared-b-w3-3b7391/auth-test-model-3b7391-kserve-74db88fc4c-smtts"
Sep 11 19:11:05.972069 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:05.971951    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d0be388-987b-417e-85b1-0eaf6bb428b3-tls-certs\") pod \"auth-test-model-3b7391-kserve-74db88fc4c-smtts\" (UID: \"3d0be388-987b-417e-85b1-0eaf6bb428b3\") " pod="ai-tenant-e2e-shared-b-w3-3b7391/auth-test-model-3b7391-kserve-74db88fc4c-smtts"
Sep 11 19:11:05.972069 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:05.971981    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d0be388-987b-417e-85b1-0eaf6bb428b3-model-cache\") pod \"auth-test-model-3b7391-kserve-74db88fc4c-smtts\" (UID: \"3d0be388-987b-417e-85b1-0eaf6bb428b3\") " pod="ai-tenant-e2e-shared-b-w3-3b7391/auth-test-model-3b7391-kserve-74db88fc4c-smtts"
Sep 11 19:11:05.972210 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:05.972080    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzwsq\" (UniqueName: \"kubernetes.io/projected/3d0be388-987b-417e-85b1-0eaf6bb428b3-kube-api-access-kzwsq\") pod \"auth-test-model-3b7391-kserve-74db88fc4c-smtts\" (UID: \"3d0be388-987b-417e-85b1-0eaf6bb428b3\") " pod="ai-tenant-e2e-shared-b-w3-3b7391/auth-test-model-3b7391-kserve-74db88fc4c-smtts"
Sep 11 19:11:05.972210 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:05.972116    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d0be388-987b-417e-85b1-0eaf6bb428b3-home\") pod \"auth-test-model-3b7391-kserve-74db88fc4c-smtts\" (UID: \"3d0be388-987b-417e-85b1-0eaf6bb428b3\") " pod="ai-tenant-e2e-shared-b-w3-3b7391/auth-test-model-3b7391-kserve-74db88fc4c-smtts"
Sep 11 19:11:06.073204 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:06.073148    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d0be388-987b-417e-85b1-0eaf6bb428b3-tls-certs\") pod \"auth-test-model-3b7391-kserve-74db88fc4c-smtts\" (UID: \"3d0be388-987b-417e-85b1-0eaf6bb428b3\") " pod="ai-tenant-e2e-shared-b-w3-3b7391/auth-test-model-3b7391-kserve-74db88fc4c-smtts"
Sep 11 19:11:06.073204 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:06.073203    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d0be388-987b-417e-85b1-0eaf6bb428b3-model-cache\") pod \"auth-test-model-3b7391-kserve-74db88fc4c-smtts\" (UID: \"3d0be388-987b-417e-85b1-0eaf6bb428b3\") " pod="ai-tenant-e2e-shared-b-w3-3b7391/auth-test-model-3b7391-kserve-74db88fc4c-smtts"
Sep 11 19:11:06.073484 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:06.073249    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzwsq\" (UniqueName: \"kubernetes.io/projected/3d0be388-987b-417e-85b1-0eaf6bb428b3-kube-api-access-kzwsq\") pod \"auth-test-model-3b7391-kserve-74db88fc4c-smtts\" (UID: \"3d0be388-987b-417e-85b1-0eaf6bb428b3\") " pod="ai-tenant-e2e-shared-b-w3-3b7391/auth-test-model-3b7391-kserve-74db88fc4c-smtts"
Sep 11 19:11:06.073484 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:06.073271    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d0be388-987b-417e-85b1-0eaf6bb428b3-home\") pod \"auth-test-model-3b7391-kserve-74db88fc4c-smtts\" (UID: \"3d0be388-987b-417e-85b1-0eaf6bb428b3\") " pod="ai-tenant-e2e-shared-b-w3-3b7391/auth-test-model-3b7391-kserve-74db88fc4c-smtts"
Sep 11 19:11:06.073484 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:06.073319    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d0be388-987b-417e-85b1-0eaf6bb428b3-tmp-dir\") pod \"auth-test-model-3b7391-kserve-74db88fc4c-smtts\" (UID: \"3d0be388-987b-417e-85b1-0eaf6bb428b3\") " pod="ai-tenant-e2e-shared-b-w3-3b7391/auth-test-model-3b7391-kserve-74db88fc4c-smtts"
Sep 11 19:11:06.073484 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:06.073339    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d0be388-987b-417e-85b1-0eaf6bb428b3-dshm\") pod \"auth-test-model-3b7391-kserve-74db88fc4c-smtts\" (UID: \"3d0be388-987b-417e-85b1-0eaf6bb428b3\") " pod="ai-tenant-e2e-shared-b-w3-3b7391/auth-test-model-3b7391-kserve-74db88fc4c-smtts"
Sep 11 19:11:06.073743 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:06.073720    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d0be388-987b-417e-85b1-0eaf6bb428b3-model-cache\") pod \"auth-test-model-3b7391-kserve-74db88fc4c-smtts\" (UID: \"3d0be388-987b-417e-85b1-0eaf6bb428b3\") " pod="ai-tenant-e2e-shared-b-w3-3b7391/auth-test-model-3b7391-kserve-74db88fc4c-smtts"
Sep 11 19:11:06.073816 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:06.073795    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d0be388-987b-417e-85b1-0eaf6bb428b3-home\") pod \"auth-test-model-3b7391-kserve-74db88fc4c-smtts\" (UID: \"3d0be388-987b-417e-85b1-0eaf6bb428b3\") " pod="ai-tenant-e2e-shared-b-w3-3b7391/auth-test-model-3b7391-kserve-74db88fc4c-smtts"
Sep 11 19:11:06.073873 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:06.073848    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d0be388-987b-417e-85b1-0eaf6bb428b3-tmp-dir\") pod \"auth-test-model-3b7391-kserve-74db88fc4c-smtts\" (UID: \"3d0be388-987b-417e-85b1-0eaf6bb428b3\") " pod="ai-tenant-e2e-shared-b-w3-3b7391/auth-test-model-3b7391-kserve-74db88fc4c-smtts"
Sep 11 19:11:06.075721 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:06.075700    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d0be388-987b-417e-85b1-0eaf6bb428b3-dshm\") pod \"auth-test-model-3b7391-kserve-74db88fc4c-smtts\" (UID: \"3d0be388-987b-417e-85b1-0eaf6bb428b3\") " pod="ai-tenant-e2e-shared-b-w3-3b7391/auth-test-model-3b7391-kserve-74db88fc4c-smtts"
Sep 11 19:11:06.076005 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:06.075988    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d0be388-987b-417e-85b1-0eaf6bb428b3-tls-certs\") pod \"auth-test-model-3b7391-kserve-74db88fc4c-smtts\" (UID: \"3d0be388-987b-417e-85b1-0eaf6bb428b3\") " pod="ai-tenant-e2e-shared-b-w3-3b7391/auth-test-model-3b7391-kserve-74db88fc4c-smtts"
Sep 11 19:11:06.082221 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:06.082200    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzwsq\" (UniqueName: \"kubernetes.io/projected/3d0be388-987b-417e-85b1-0eaf6bb428b3-kube-api-access-kzwsq\") pod \"auth-test-model-3b7391-kserve-74db88fc4c-smtts\" (UID: \"3d0be388-987b-417e-85b1-0eaf6bb428b3\") " pod="ai-tenant-e2e-shared-b-w3-3b7391/auth-test-model-3b7391-kserve-74db88fc4c-smtts"
Sep 11 19:11:06.258139 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:06.258104    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3b7391/auth-test-model-3b7391-kserve-74db88fc4c-smtts"
Sep 11 19:11:06.534552 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:06.534521    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3b7391/auth-test-model-3b7391-kserve-74db88fc4c-smtts"]
Sep 11 19:11:06.537168 ip-10-0-134-189 kubenswrapper[2578]: W0911 19:11:06.537136    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3d0be388_987b_417e_85b1_0eaf6bb428b3.slice/crio-dff0805a7e90e7f0f34de73d7d46b59d49c257e0700654ae36f8b6f370751849 WatchSource:0}: Error finding container dff0805a7e90e7f0f34de73d7d46b59d49c257e0700654ae36f8b6f370751849: Status 404 returned error can't find the container with id dff0805a7e90e7f0f34de73d7d46b59d49c257e0700654ae36f8b6f370751849
Sep 11 19:11:06.779076 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:06.779035    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3b7391/auth-test-model-3b7391-kserve-74db88fc4c-smtts" event={"ID":"3d0be388-987b-417e-85b1-0eaf6bb428b3","Type":"ContainerStarted","Data":"415397f1a52cbc3d00c0a924afb5eec0f1095d6fb9c6739ab5d54cd54bdee625"}
Sep 11 19:11:06.779253 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:06.779088    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3b7391/auth-test-model-3b7391-kserve-74db88fc4c-smtts" event={"ID":"3d0be388-987b-417e-85b1-0eaf6bb428b3","Type":"ContainerStarted","Data":"dff0805a7e90e7f0f34de73d7d46b59d49c257e0700654ae36f8b6f370751849"}
Sep 11 19:11:06.812238 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:06.812128    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-3b7391/auth-test-model-3b7391-kserve-74db88fc4c-smtts" podStartSLOduration=1.812108346 podStartE2EDuration="1.812108346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:11:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:11:06.810328442 +0000 UTC m=+1342.887919010" watchObservedRunningTime="2026-09-11 19:11:06.812108346 +0000 UTC m=+1342.889698901"
Sep 11 19:11:09.592041 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:09.591998    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-7ad53a9d/e2e-lifecycle-model-7ad53a9d-kserve-86f545dbc9-zlh6b" podUID="f915305d-9ebc-4e8e-8e0a-9748f0389d0b" containerName="main" containerID="cri-o://11b8973361b7b61fff7c40446d1d1e052657731011b89a871897a52140c281b5" gracePeriod=45
Sep 11 19:11:09.798243 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:09.798206    2578 generic.go:356] "Generic (PLEG): container finished" podID="f915305d-9ebc-4e8e-8e0a-9748f0389d0b" containerID="11b8973361b7b61fff7c40446d1d1e052657731011b89a871897a52140c281b5" exitCode=0
Sep 11 19:11:09.798401 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:09.798285    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-7ad53a9d/e2e-lifecycle-model-7ad53a9d-kserve-86f545dbc9-zlh6b" event={"ID":"f915305d-9ebc-4e8e-8e0a-9748f0389d0b","Type":"ContainerDied","Data":"11b8973361b7b61fff7c40446d1d1e052657731011b89a871897a52140c281b5"}
Sep 11 19:11:09.855867 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:09.855804    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-7ad53a9d/e2e-lifecycle-model-7ad53a9d-kserve-86f545dbc9-zlh6b"
Sep 11 19:11:09.912178 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:09.912148    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ljgmv\" (UniqueName: \"kubernetes.io/projected/f915305d-9ebc-4e8e-8e0a-9748f0389d0b-kube-api-access-ljgmv\") pod \"f915305d-9ebc-4e8e-8e0a-9748f0389d0b\" (UID: \"f915305d-9ebc-4e8e-8e0a-9748f0389d0b\") "
Sep 11 19:11:09.912383 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:09.912185    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f915305d-9ebc-4e8e-8e0a-9748f0389d0b-dshm\") pod \"f915305d-9ebc-4e8e-8e0a-9748f0389d0b\" (UID: \"f915305d-9ebc-4e8e-8e0a-9748f0389d0b\") "
Sep 11 19:11:09.912383 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:09.912220    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f915305d-9ebc-4e8e-8e0a-9748f0389d0b-home\") pod \"f915305d-9ebc-4e8e-8e0a-9748f0389d0b\" (UID: \"f915305d-9ebc-4e8e-8e0a-9748f0389d0b\") "
Sep 11 19:11:09.912383 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:09.912338    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f915305d-9ebc-4e8e-8e0a-9748f0389d0b-tls-certs\") pod \"f915305d-9ebc-4e8e-8e0a-9748f0389d0b\" (UID: \"f915305d-9ebc-4e8e-8e0a-9748f0389d0b\") "
Sep 11 19:11:09.912383 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:09.912362    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f915305d-9ebc-4e8e-8e0a-9748f0389d0b-tmp-dir\") pod \"f915305d-9ebc-4e8e-8e0a-9748f0389d0b\" (UID: \"f915305d-9ebc-4e8e-8e0a-9748f0389d0b\") "
Sep 11 19:11:09.912711 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:09.912391    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f915305d-9ebc-4e8e-8e0a-9748f0389d0b-model-cache\") pod \"f915305d-9ebc-4e8e-8e0a-9748f0389d0b\" (UID: \"f915305d-9ebc-4e8e-8e0a-9748f0389d0b\") "
Sep 11 19:11:09.912811 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:09.912761    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f915305d-9ebc-4e8e-8e0a-9748f0389d0b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f915305d-9ebc-4e8e-8e0a-9748f0389d0b" (UID: "f915305d-9ebc-4e8e-8e0a-9748f0389d0b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:11:09.912876 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:09.912831    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f915305d-9ebc-4e8e-8e0a-9748f0389d0b-model-cache" (OuterVolumeSpecName: "model-cache") pod "f915305d-9ebc-4e8e-8e0a-9748f0389d0b" (UID: "f915305d-9ebc-4e8e-8e0a-9748f0389d0b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:11:09.912876 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:09.912851    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f915305d-9ebc-4e8e-8e0a-9748f0389d0b-home" (OuterVolumeSpecName: "home") pod "f915305d-9ebc-4e8e-8e0a-9748f0389d0b" (UID: "f915305d-9ebc-4e8e-8e0a-9748f0389d0b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:11:09.914822 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:09.914786    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f915305d-9ebc-4e8e-8e0a-9748f0389d0b-kube-api-access-ljgmv" (OuterVolumeSpecName: "kube-api-access-ljgmv") pod "f915305d-9ebc-4e8e-8e0a-9748f0389d0b" (UID: "f915305d-9ebc-4e8e-8e0a-9748f0389d0b"). InnerVolumeSpecName "kube-api-access-ljgmv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:11:09.914961 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:09.914786    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f915305d-9ebc-4e8e-8e0a-9748f0389d0b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f915305d-9ebc-4e8e-8e0a-9748f0389d0b" (UID: "f915305d-9ebc-4e8e-8e0a-9748f0389d0b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 19:11:09.915086 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:09.915071    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f915305d-9ebc-4e8e-8e0a-9748f0389d0b-dshm" (OuterVolumeSpecName: "dshm") pod "f915305d-9ebc-4e8e-8e0a-9748f0389d0b" (UID: "f915305d-9ebc-4e8e-8e0a-9748f0389d0b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:11:10.013748 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:10.013712    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f915305d-9ebc-4e8e-8e0a-9748f0389d0b-tls-certs\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:11:10.013748 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:10.013744    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f915305d-9ebc-4e8e-8e0a-9748f0389d0b-tmp-dir\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:11:10.013748 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:10.013756    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f915305d-9ebc-4e8e-8e0a-9748f0389d0b-model-cache\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:11:10.014010 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:10.013768    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljgmv\" (UniqueName: \"kubernetes.io/projected/f915305d-9ebc-4e8e-8e0a-9748f0389d0b-kube-api-access-ljgmv\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:11:10.014010 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:10.013780    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f915305d-9ebc-4e8e-8e0a-9748f0389d0b-dshm\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:11:10.014010 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:10.013791    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f915305d-9ebc-4e8e-8e0a-9748f0389d0b-home\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:11:10.805156 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:10.805124    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-7ad53a9d/e2e-lifecycle-model-7ad53a9d-kserve-86f545dbc9-zlh6b"
Sep 11 19:11:10.805156 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:10.805143    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-7ad53a9d/e2e-lifecycle-model-7ad53a9d-kserve-86f545dbc9-zlh6b" event={"ID":"f915305d-9ebc-4e8e-8e0a-9748f0389d0b","Type":"ContainerDied","Data":"63aa3beeaaa04b045b052d05a682c6219c08f8e189a99ce05f17bb68054436e3"}
Sep 11 19:11:10.805708 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:10.805199    2578 scope.go:117] "RemoveContainer" containerID="11b8973361b7b61fff7c40446d1d1e052657731011b89a871897a52140c281b5"
Sep 11 19:11:10.823217 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:10.823191    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-7ad53a9d/e2e-lifecycle-model-7ad53a9d-kserve-86f545dbc9-zlh6b"]
Sep 11 19:11:10.828370 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:10.828347    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-7ad53a9d/e2e-lifecycle-model-7ad53a9d-kserve-86f545dbc9-zlh6b"]
Sep 11 19:11:11.352031 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:11.351995    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-c1aebc/distinct-w4-c1aebc-kserve-6fd464965d-rw6jw"
Sep 11 19:11:11.352222 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:11.352107    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c1aebc/distinct-w4-c1aebc-kserve-6fd464965d-rw6jw"
Sep 11 19:11:11.365261 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:11.365224    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-c1aebc/distinct-w4-c1aebc-kserve-6fd464965d-rw6jw"
Sep 11 19:11:11.833302 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:11.833270    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c1aebc/distinct-w4-c1aebc-kserve-6fd464965d-rw6jw"
Sep 11 19:11:12.396485 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:12.396450    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f915305d-9ebc-4e8e-8e0a-9748f0389d0b" path="/var/lib/kubelet/pods/f915305d-9ebc-4e8e-8e0a-9748f0389d0b/volumes"
Sep 11 19:11:16.258588 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:16.258547    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-3b7391/auth-test-model-3b7391-kserve-74db88fc4c-smtts"
Sep 11 19:11:16.259089 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:16.258642    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-3b7391/auth-test-model-3b7391-kserve-74db88fc4c-smtts"
Sep 11 19:11:16.271453 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:16.271422    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-3b7391/auth-test-model-3b7391-kserve-74db88fc4c-smtts"
Sep 11 19:11:16.853244 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:16.853216    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-3b7391/auth-test-model-3b7391-kserve-74db88fc4c-smtts"
Sep 11 19:11:29.536434 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:29.536393    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-c1aebc/distinct-2-w4-c1aebc-kserve-57879dd7f7-dfzxs"]
Sep 11 19:11:29.537042 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:29.536798    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f915305d-9ebc-4e8e-8e0a-9748f0389d0b" containerName="main"
Sep 11 19:11:29.537042 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:29.536811    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f915305d-9ebc-4e8e-8e0a-9748f0389d0b" containerName="main"
Sep 11 19:11:29.537042 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:29.536876    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f915305d-9ebc-4e8e-8e0a-9748f0389d0b" containerName="main"
Sep 11 19:11:29.539636 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:29.539619    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c1aebc/distinct-2-w4-c1aebc-kserve-57879dd7f7-dfzxs"
Sep 11 19:11:29.542164 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:29.542144    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-c1aebc\"/\"distinct-2-w4-c1aebc-kserve-self-signed-certs\""
Sep 11 19:11:29.549089 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:29.549065    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c1aebc/distinct-2-w4-c1aebc-kserve-57879dd7f7-dfzxs"]
Sep 11 19:11:29.704009 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:29.703962    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7748ab8-6e1c-4df9-9e28-aa25e75b0b4e-tls-certs\") pod \"distinct-2-w4-c1aebc-kserve-57879dd7f7-dfzxs\" (UID: \"b7748ab8-6e1c-4df9-9e28-aa25e75b0b4e\") " pod="e2e-models-e2e-worker-w4-c1aebc/distinct-2-w4-c1aebc-kserve-57879dd7f7-dfzxs"
Sep 11 19:11:29.704009 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:29.704010    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7748ab8-6e1c-4df9-9e28-aa25e75b0b4e-tmp-dir\") pod \"distinct-2-w4-c1aebc-kserve-57879dd7f7-dfzxs\" (UID: \"b7748ab8-6e1c-4df9-9e28-aa25e75b0b4e\") " pod="e2e-models-e2e-worker-w4-c1aebc/distinct-2-w4-c1aebc-kserve-57879dd7f7-dfzxs"
Sep 11 19:11:29.704235 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:29.704075    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7748ab8-6e1c-4df9-9e28-aa25e75b0b4e-model-cache\") pod \"distinct-2-w4-c1aebc-kserve-57879dd7f7-dfzxs\" (UID: \"b7748ab8-6e1c-4df9-9e28-aa25e75b0b4e\") " pod="e2e-models-e2e-worker-w4-c1aebc/distinct-2-w4-c1aebc-kserve-57879dd7f7-dfzxs"
Sep 11 19:11:29.704235 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:29.704158    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbg8j\" (UniqueName: \"kubernetes.io/projected/b7748ab8-6e1c-4df9-9e28-aa25e75b0b4e-kube-api-access-cbg8j\") pod \"distinct-2-w4-c1aebc-kserve-57879dd7f7-dfzxs\" (UID: \"b7748ab8-6e1c-4df9-9e28-aa25e75b0b4e\") " pod="e2e-models-e2e-worker-w4-c1aebc/distinct-2-w4-c1aebc-kserve-57879dd7f7-dfzxs"
Sep 11 19:11:29.704235 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:29.704179    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7748ab8-6e1c-4df9-9e28-aa25e75b0b4e-dshm\") pod \"distinct-2-w4-c1aebc-kserve-57879dd7f7-dfzxs\" (UID: \"b7748ab8-6e1c-4df9-9e28-aa25e75b0b4e\") " pod="e2e-models-e2e-worker-w4-c1aebc/distinct-2-w4-c1aebc-kserve-57879dd7f7-dfzxs"
Sep 11 19:11:29.704235 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:29.704198    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7748ab8-6e1c-4df9-9e28-aa25e75b0b4e-home\") pod \"distinct-2-w4-c1aebc-kserve-57879dd7f7-dfzxs\" (UID: \"b7748ab8-6e1c-4df9-9e28-aa25e75b0b4e\") " pod="e2e-models-e2e-worker-w4-c1aebc/distinct-2-w4-c1aebc-kserve-57879dd7f7-dfzxs"
Sep 11 19:11:29.805067 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:29.804972    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7748ab8-6e1c-4df9-9e28-aa25e75b0b4e-tls-certs\") pod \"distinct-2-w4-c1aebc-kserve-57879dd7f7-dfzxs\" (UID: \"b7748ab8-6e1c-4df9-9e28-aa25e75b0b4e\") " pod="e2e-models-e2e-worker-w4-c1aebc/distinct-2-w4-c1aebc-kserve-57879dd7f7-dfzxs"
Sep 11 19:11:29.805067 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:29.805026    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7748ab8-6e1c-4df9-9e28-aa25e75b0b4e-tmp-dir\") pod \"distinct-2-w4-c1aebc-kserve-57879dd7f7-dfzxs\" (UID: \"b7748ab8-6e1c-4df9-9e28-aa25e75b0b4e\") " pod="e2e-models-e2e-worker-w4-c1aebc/distinct-2-w4-c1aebc-kserve-57879dd7f7-dfzxs"
Sep 11 19:11:29.805315 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:29.805068    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7748ab8-6e1c-4df9-9e28-aa25e75b0b4e-model-cache\") pod \"distinct-2-w4-c1aebc-kserve-57879dd7f7-dfzxs\" (UID: \"b7748ab8-6e1c-4df9-9e28-aa25e75b0b4e\") " pod="e2e-models-e2e-worker-w4-c1aebc/distinct-2-w4-c1aebc-kserve-57879dd7f7-dfzxs"
Sep 11 19:11:29.805315 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:29.805125    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbg8j\" (UniqueName: \"kubernetes.io/projected/b7748ab8-6e1c-4df9-9e28-aa25e75b0b4e-kube-api-access-cbg8j\") pod \"distinct-2-w4-c1aebc-kserve-57879dd7f7-dfzxs\" (UID: \"b7748ab8-6e1c-4df9-9e28-aa25e75b0b4e\") " pod="e2e-models-e2e-worker-w4-c1aebc/distinct-2-w4-c1aebc-kserve-57879dd7f7-dfzxs"
Sep 11 19:11:29.805315 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:29.805158    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7748ab8-6e1c-4df9-9e28-aa25e75b0b4e-dshm\") pod \"distinct-2-w4-c1aebc-kserve-57879dd7f7-dfzxs\" (UID: \"b7748ab8-6e1c-4df9-9e28-aa25e75b0b4e\") " pod="e2e-models-e2e-worker-w4-c1aebc/distinct-2-w4-c1aebc-kserve-57879dd7f7-dfzxs"
Sep 11 19:11:29.805315 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:29.805191    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7748ab8-6e1c-4df9-9e28-aa25e75b0b4e-home\") pod \"distinct-2-w4-c1aebc-kserve-57879dd7f7-dfzxs\" (UID: \"b7748ab8-6e1c-4df9-9e28-aa25e75b0b4e\") " pod="e2e-models-e2e-worker-w4-c1aebc/distinct-2-w4-c1aebc-kserve-57879dd7f7-dfzxs"
Sep 11 19:11:29.806628 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:29.805496    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7748ab8-6e1c-4df9-9e28-aa25e75b0b4e-tmp-dir\") pod \"distinct-2-w4-c1aebc-kserve-57879dd7f7-dfzxs\" (UID: \"b7748ab8-6e1c-4df9-9e28-aa25e75b0b4e\") " pod="e2e-models-e2e-worker-w4-c1aebc/distinct-2-w4-c1aebc-kserve-57879dd7f7-dfzxs"
Sep 11 19:11:29.806628 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:29.805763    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7748ab8-6e1c-4df9-9e28-aa25e75b0b4e-home\") pod \"distinct-2-w4-c1aebc-kserve-57879dd7f7-dfzxs\" (UID: \"b7748ab8-6e1c-4df9-9e28-aa25e75b0b4e\") " pod="e2e-models-e2e-worker-w4-c1aebc/distinct-2-w4-c1aebc-kserve-57879dd7f7-dfzxs"
Sep 11 19:11:29.806628 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:29.806043    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7748ab8-6e1c-4df9-9e28-aa25e75b0b4e-model-cache\") pod \"distinct-2-w4-c1aebc-kserve-57879dd7f7-dfzxs\" (UID: \"b7748ab8-6e1c-4df9-9e28-aa25e75b0b4e\") " pod="e2e-models-e2e-worker-w4-c1aebc/distinct-2-w4-c1aebc-kserve-57879dd7f7-dfzxs"
Sep 11 19:11:29.808779 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:29.808722    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7748ab8-6e1c-4df9-9e28-aa25e75b0b4e-dshm\") pod \"distinct-2-w4-c1aebc-kserve-57879dd7f7-dfzxs\" (UID: \"b7748ab8-6e1c-4df9-9e28-aa25e75b0b4e\") " pod="e2e-models-e2e-worker-w4-c1aebc/distinct-2-w4-c1aebc-kserve-57879dd7f7-dfzxs"
Sep 11 19:11:29.808901 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:29.808857    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7748ab8-6e1c-4df9-9e28-aa25e75b0b4e-tls-certs\") pod \"distinct-2-w4-c1aebc-kserve-57879dd7f7-dfzxs\" (UID: \"b7748ab8-6e1c-4df9-9e28-aa25e75b0b4e\") " pod="e2e-models-e2e-worker-w4-c1aebc/distinct-2-w4-c1aebc-kserve-57879dd7f7-dfzxs"
Sep 11 19:11:29.823159 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:29.823125    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbg8j\" (UniqueName: \"kubernetes.io/projected/b7748ab8-6e1c-4df9-9e28-aa25e75b0b4e-kube-api-access-cbg8j\") pod \"distinct-2-w4-c1aebc-kserve-57879dd7f7-dfzxs\" (UID: \"b7748ab8-6e1c-4df9-9e28-aa25e75b0b4e\") " pod="e2e-models-e2e-worker-w4-c1aebc/distinct-2-w4-c1aebc-kserve-57879dd7f7-dfzxs"
Sep 11 19:11:29.849850 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:29.849799    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c1aebc/distinct-2-w4-c1aebc-kserve-57879dd7f7-dfzxs"
Sep 11 19:11:30.126294 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:30.126271    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c1aebc/distinct-2-w4-c1aebc-kserve-57879dd7f7-dfzxs"]
Sep 11 19:11:30.127922 ip-10-0-134-189 kubenswrapper[2578]: W0911 19:11:30.127898    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb7748ab8_6e1c_4df9_9e28_aa25e75b0b4e.slice/crio-f21a372f91ac637ba36b1fff2201d435a176fe36c2d98a879bc59048a18cb6eb WatchSource:0}: Error finding container f21a372f91ac637ba36b1fff2201d435a176fe36c2d98a879bc59048a18cb6eb: Status 404 returned error can't find the container with id f21a372f91ac637ba36b1fff2201d435a176fe36c2d98a879bc59048a18cb6eb
Sep 11 19:11:30.914577 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:30.914538    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c1aebc/distinct-2-w4-c1aebc-kserve-57879dd7f7-dfzxs" event={"ID":"b7748ab8-6e1c-4df9-9e28-aa25e75b0b4e","Type":"ContainerStarted","Data":"e07a85729f476965b9dc0f2589eec2152981e61c1304730291691af56cf924d4"}
Sep 11 19:11:30.914577 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:30.914583    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c1aebc/distinct-2-w4-c1aebc-kserve-57879dd7f7-dfzxs" event={"ID":"b7748ab8-6e1c-4df9-9e28-aa25e75b0b4e","Type":"ContainerStarted","Data":"f21a372f91ac637ba36b1fff2201d435a176fe36c2d98a879bc59048a18cb6eb"}
Sep 11 19:11:30.940087 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:30.940038    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-c1aebc/distinct-2-w4-c1aebc-kserve-57879dd7f7-dfzxs" podStartSLOduration=1.940023133 podStartE2EDuration="1.940023133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:11:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:11:30.938851037 +0000 UTC m=+1367.016441612" watchObservedRunningTime="2026-09-11 19:11:30.940023133 +0000 UTC m=+1367.017613689"
Sep 11 19:11:39.849979 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:39.849928    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c1aebc/distinct-2-w4-c1aebc-kserve-57879dd7f7-dfzxs"
Sep 11 19:11:39.850457 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:39.850250    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-c1aebc/distinct-2-w4-c1aebc-kserve-57879dd7f7-dfzxs"
Sep 11 19:11:39.862731 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:39.862696    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-c1aebc/distinct-2-w4-c1aebc-kserve-57879dd7f7-dfzxs"
Sep 11 19:11:39.973122 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:11:39.973088    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c1aebc/distinct-2-w4-c1aebc-kserve-57879dd7f7-dfzxs"
Sep 11 19:12:44.561165 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:12:44.561089    2578 scope.go:117] "RemoveContainer" containerID="66b4b7551633890d0d99bfa1c8660c219eb8e83f0ebc7fb041c4210c2cda6b2d"
Sep 11 19:12:44.569910 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:12:44.569883    2578 scope.go:117] "RemoveContainer" containerID="cc6f7f131a258e4757f8018fae984ac3ee2c7c2fb3db06f5873508d557d06c1f"
Sep 11 19:12:44.791549 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:12:44.791514    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7c10b1-5694f468b6-7bkqx"]
Sep 11 19:12:44.796089 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:12:44.796065    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7c10b1-5694f468b6-7bkqx"
Sep 11 19:12:44.798281 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:12:44.798258    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-7c10b1\""
Sep 11 19:12:44.809134 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:12:44.809102    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7c10b1-5694f468b6-7bkqx"]
Sep 11 19:12:44.820961 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:12:44.820897    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7c10b1-85ffc8cddc-qw49x"]
Sep 11 19:12:44.824149 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:12:44.824128    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-7c10b1-85ffc8cddc-qw49x"
Sep 11 19:12:44.826582 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:12:44.826563    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-7c10b1-dockercfg-jk8gp\""
Sep 11 19:12:44.834754 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:12:44.834730    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7c10b1-85ffc8cddc-qw49x"]
Sep 11 19:12:44.948415 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:12:44.948378    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bs875\" (UniqueName: \"kubernetes.io/projected/3246a64d-dc72-4107-b5dd-6a6f1409a16f-kube-api-access-bs875\") pod \"payload-processing-e2e-api-a-7c10b1-85ffc8cddc-qw49x\" (UID: \"3246a64d-dc72-4107-b5dd-6a6f1409a16f\") " pod="openshift-ingress/payload-processing-e2e-api-a-7c10b1-85ffc8cddc-qw49x"
Sep 11 19:12:44.948636 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:12:44.948431    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3246a64d-dc72-4107-b5dd-6a6f1409a16f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-7c10b1-85ffc8cddc-qw49x\" (UID: \"3246a64d-dc72-4107-b5dd-6a6f1409a16f\") " pod="openshift-ingress/payload-processing-e2e-api-a-7c10b1-85ffc8cddc-qw49x"
Sep 11 19:12:44.948636 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:12:44.948491    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2ljm\" (UniqueName: \"kubernetes.io/projected/4e269dec-ab2f-46f4-90a1-fb628dafcfb7-kube-api-access-z2ljm\") pod \"payload-pre-processing-e2e-api-a-7c10b1-5694f468b6-7bkqx\" (UID: \"4e269dec-ab2f-46f4-90a1-fb628dafcfb7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7c10b1-5694f468b6-7bkqx"
Sep 11 19:12:44.948756 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:12:44.948632    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e269dec-ab2f-46f4-90a1-fb628dafcfb7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-7c10b1-5694f468b6-7bkqx\" (UID: \"4e269dec-ab2f-46f4-90a1-fb628dafcfb7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7c10b1-5694f468b6-7bkqx"
Sep 11 19:12:45.050062 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:12:45.050017    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs875\" (UniqueName: \"kubernetes.io/projected/3246a64d-dc72-4107-b5dd-6a6f1409a16f-kube-api-access-bs875\") pod \"payload-processing-e2e-api-a-7c10b1-85ffc8cddc-qw49x\" (UID: \"3246a64d-dc72-4107-b5dd-6a6f1409a16f\") " pod="openshift-ingress/payload-processing-e2e-api-a-7c10b1-85ffc8cddc-qw49x"
Sep 11 19:12:45.050272 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:12:45.050079    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3246a64d-dc72-4107-b5dd-6a6f1409a16f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-7c10b1-85ffc8cddc-qw49x\" (UID: \"3246a64d-dc72-4107-b5dd-6a6f1409a16f\") " pod="openshift-ingress/payload-processing-e2e-api-a-7c10b1-85ffc8cddc-qw49x"
Sep 11 19:12:45.050272 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:12:45.050119    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2ljm\" (UniqueName: \"kubernetes.io/projected/4e269dec-ab2f-46f4-90a1-fb628dafcfb7-kube-api-access-z2ljm\") pod \"payload-pre-processing-e2e-api-a-7c10b1-5694f468b6-7bkqx\" (UID: \"4e269dec-ab2f-46f4-90a1-fb628dafcfb7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7c10b1-5694f468b6-7bkqx"
Sep 11 19:12:45.050272 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:12:45.050202    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e269dec-ab2f-46f4-90a1-fb628dafcfb7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-7c10b1-5694f468b6-7bkqx\" (UID: \"4e269dec-ab2f-46f4-90a1-fb628dafcfb7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7c10b1-5694f468b6-7bkqx"
Sep 11 19:12:45.050880 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:12:45.050854    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3246a64d-dc72-4107-b5dd-6a6f1409a16f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-7c10b1-85ffc8cddc-qw49x\" (UID: \"3246a64d-dc72-4107-b5dd-6a6f1409a16f\") " pod="openshift-ingress/payload-processing-e2e-api-a-7c10b1-85ffc8cddc-qw49x"
Sep 11 19:12:45.050991 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:12:45.050915    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e269dec-ab2f-46f4-90a1-fb628dafcfb7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-7c10b1-5694f468b6-7bkqx\" (UID: \"4e269dec-ab2f-46f4-90a1-fb628dafcfb7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7c10b1-5694f468b6-7bkqx"
Sep 11 19:12:45.061308 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:12:45.061282    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2ljm\" (UniqueName: \"kubernetes.io/projected/4e269dec-ab2f-46f4-90a1-fb628dafcfb7-kube-api-access-z2ljm\") pod \"payload-pre-processing-e2e-api-a-7c10b1-5694f468b6-7bkqx\" (UID: \"4e269dec-ab2f-46f4-90a1-fb628dafcfb7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7c10b1-5694f468b6-7bkqx"
Sep 11 19:12:45.062021 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:12:45.062003    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs875\" (UniqueName: \"kubernetes.io/projected/3246a64d-dc72-4107-b5dd-6a6f1409a16f-kube-api-access-bs875\") pod \"payload-processing-e2e-api-a-7c10b1-85ffc8cddc-qw49x\" (UID: \"3246a64d-dc72-4107-b5dd-6a6f1409a16f\") " pod="openshift-ingress/payload-processing-e2e-api-a-7c10b1-85ffc8cddc-qw49x"
Sep 11 19:12:45.106536 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:12:45.106432    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7c10b1-5694f468b6-7bkqx"
Sep 11 19:12:45.134886 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:12:45.134855    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-7c10b1-85ffc8cddc-qw49x"
Sep 11 19:12:45.383925 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:12:45.383901    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7c10b1-5694f468b6-7bkqx"]
Sep 11 19:12:45.386018 ip-10-0-134-189 kubenswrapper[2578]: W0911 19:12:45.385992    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e269dec_ab2f_46f4_90a1_fb628dafcfb7.slice/crio-930be5660fbbf2aaed42db9beadd0401e4cabb87ea639847d07c115290523a4b WatchSource:0}: Error finding container 930be5660fbbf2aaed42db9beadd0401e4cabb87ea639847d07c115290523a4b: Status 404 returned error can't find the container with id 930be5660fbbf2aaed42db9beadd0401e4cabb87ea639847d07c115290523a4b
Sep 11 19:12:45.429082 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:12:45.429050    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7c10b1-85ffc8cddc-qw49x"]
Sep 11 19:12:45.431564 ip-10-0-134-189 kubenswrapper[2578]: W0911 19:12:45.431535    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3246a64d_dc72_4107_b5dd_6a6f1409a16f.slice/crio-93717fdc52adf462bb20e7d7396abce9187b375fe7a20392123e1944e58779af WatchSource:0}: Error finding container 93717fdc52adf462bb20e7d7396abce9187b375fe7a20392123e1944e58779af: Status 404 returned error can't find the container with id 93717fdc52adf462bb20e7d7396abce9187b375fe7a20392123e1944e58779af
Sep 11 19:12:46.306172 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:12:46.306108    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7c10b1-5694f468b6-7bkqx" event={"ID":"4e269dec-ab2f-46f4-90a1-fb628dafcfb7","Type":"ContainerStarted","Data":"879a5bb94b9fb94ee68821eb48506ab8410f59bc4267c90e0d5b8011ad05d862"}
Sep 11 19:12:46.306689 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:12:46.306209    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7c10b1-5694f468b6-7bkqx" event={"ID":"4e269dec-ab2f-46f4-90a1-fb628dafcfb7","Type":"ContainerStarted","Data":"930be5660fbbf2aaed42db9beadd0401e4cabb87ea639847d07c115290523a4b"}
Sep 11 19:12:46.306689 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:12:46.306317    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7c10b1-5694f468b6-7bkqx"
Sep 11 19:12:46.307982 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:12:46.307959    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-7c10b1-85ffc8cddc-qw49x" event={"ID":"3246a64d-dc72-4107-b5dd-6a6f1409a16f","Type":"ContainerStarted","Data":"a460c90255a8eb095c65e1130965d467830724d9811fc76cadcce143418da62b"}
Sep 11 19:12:46.308095 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:12:46.308005    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-7c10b1-85ffc8cddc-qw49x" event={"ID":"3246a64d-dc72-4107-b5dd-6a6f1409a16f","Type":"ContainerStarted","Data":"93717fdc52adf462bb20e7d7396abce9187b375fe7a20392123e1944e58779af"}
Sep 11 19:12:46.308095 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:12:46.308035    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-7c10b1-85ffc8cddc-qw49x"
Sep 11 19:12:46.324064 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:12:46.324007    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7c10b1-5694f468b6-7bkqx" podStartSLOduration=2.323987847 podStartE2EDuration="2.323987847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:12:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:12:46.322632285 +0000 UTC m=+1442.400222849" watchObservedRunningTime="2026-09-11 19:12:46.323987847 +0000 UTC m=+1442.401578401"
Sep 11 19:12:46.341101 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:12:46.341042    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-7c10b1-85ffc8cddc-qw49x" podStartSLOduration=2.341024333 podStartE2EDuration="2.341024333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:12:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:12:46.339678388 +0000 UTC m=+1442.417268965" watchObservedRunningTime="2026-09-11 19:12:46.341024333 +0000 UTC m=+1442.418614914"
Sep 11 19:12:57.315463 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:12:57.315428    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-7c10b1-85ffc8cddc-qw49x"
Sep 11 19:12:57.316065 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:12:57.315657    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7c10b1-5694f468b6-7bkqx"
Sep 11 19:13:15.423313 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:13:15.423271    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-852059-55b7677bf4-qmwk7"]
Sep 11 19:13:15.426947 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:13:15.426929    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-852059-55b7677bf4-qmwk7"
Sep 11 19:13:15.429324 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:13:15.429298    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-852059\""
Sep 11 19:13:15.429465 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:13:15.429394    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-852059-dockercfg-twtgw\""
Sep 11 19:13:15.439621 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:13:15.439577    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-852059-55b7677bf4-qmwk7"]
Sep 11 19:13:15.561097 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:13:15.561054    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c471430b-31b5-4ca4-9149-fd894fe3f14b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-852059-55b7677bf4-qmwk7\" (UID: \"c471430b-31b5-4ca4-9149-fd894fe3f14b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-852059-55b7677bf4-qmwk7"
Sep 11 19:13:15.561310 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:13:15.561166    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxlw2\" (UniqueName: \"kubernetes.io/projected/c471430b-31b5-4ca4-9149-fd894fe3f14b-kube-api-access-qxlw2\") pod \"payload-pre-processing-e2e-api-b-852059-55b7677bf4-qmwk7\" (UID: \"c471430b-31b5-4ca4-9149-fd894fe3f14b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-852059-55b7677bf4-qmwk7"
Sep 11 19:13:15.661935 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:13:15.661899    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxlw2\" (UniqueName: \"kubernetes.io/projected/c471430b-31b5-4ca4-9149-fd894fe3f14b-kube-api-access-qxlw2\") pod \"payload-pre-processing-e2e-api-b-852059-55b7677bf4-qmwk7\" (UID: \"c471430b-31b5-4ca4-9149-fd894fe3f14b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-852059-55b7677bf4-qmwk7"
Sep 11 19:13:15.662149 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:13:15.662027    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c471430b-31b5-4ca4-9149-fd894fe3f14b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-852059-55b7677bf4-qmwk7\" (UID: \"c471430b-31b5-4ca4-9149-fd894fe3f14b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-852059-55b7677bf4-qmwk7"
Sep 11 19:13:15.662721 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:13:15.662701    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c471430b-31b5-4ca4-9149-fd894fe3f14b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-852059-55b7677bf4-qmwk7\" (UID: \"c471430b-31b5-4ca4-9149-fd894fe3f14b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-852059-55b7677bf4-qmwk7"
Sep 11 19:13:15.675621 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:13:15.675512    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxlw2\" (UniqueName: \"kubernetes.io/projected/c471430b-31b5-4ca4-9149-fd894fe3f14b-kube-api-access-qxlw2\") pod \"payload-pre-processing-e2e-api-b-852059-55b7677bf4-qmwk7\" (UID: \"c471430b-31b5-4ca4-9149-fd894fe3f14b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-852059-55b7677bf4-qmwk7"
Sep 11 19:13:15.737438 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:13:15.737398    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-852059-55b7677bf4-qmwk7"
Sep 11 19:13:16.016443 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:13:16.016408    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-852059-55b7677bf4-qmwk7"]
Sep 11 19:13:16.017724 ip-10-0-134-189 kubenswrapper[2578]: W0911 19:13:16.017694    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc471430b_31b5_4ca4_9149_fd894fe3f14b.slice/crio-61f36373939b8e9e917570bec0ea514e1de1c3f9f24ecaa52088a25ee2ebd23b WatchSource:0}: Error finding container 61f36373939b8e9e917570bec0ea514e1de1c3f9f24ecaa52088a25ee2ebd23b: Status 404 returned error can't find the container with id 61f36373939b8e9e917570bec0ea514e1de1c3f9f24ecaa52088a25ee2ebd23b
Sep 11 19:13:16.464154 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:13:16.464120    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-852059-55b7677bf4-qmwk7" event={"ID":"c471430b-31b5-4ca4-9149-fd894fe3f14b","Type":"ContainerStarted","Data":"3989dbaf7c6a2659224a774deb92331398009924cebe8f4838287ffe65efad00"}
Sep 11 19:13:16.464558 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:13:16.464173    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-852059-55b7677bf4-qmwk7" event={"ID":"c471430b-31b5-4ca4-9149-fd894fe3f14b","Type":"ContainerStarted","Data":"61f36373939b8e9e917570bec0ea514e1de1c3f9f24ecaa52088a25ee2ebd23b"}
Sep 11 19:13:16.464558 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:13:16.464228    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-852059-55b7677bf4-qmwk7"
Sep 11 19:13:16.483983 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:13:16.483926    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-852059-55b7677bf4-qmwk7" podStartSLOduration=1.483910118 podStartE2EDuration="1.483910118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:13:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:13:16.481648775 +0000 UTC m=+1472.559239349" watchObservedRunningTime="2026-09-11 19:13:16.483910118 +0000 UTC m=+1472.561500673"
Sep 11 19:13:27.472668 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:13:27.472631    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-852059-55b7677bf4-qmwk7"
Sep 11 19:13:41.121751 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:13:41.121708    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7c10b1-qffzf"]
Sep 11 19:13:41.125252 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:13:41.125228    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7c10b1-qffzf"
Sep 11 19:13:41.127363 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:13:41.127344    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 19:13:41.127496 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:13:41.127344    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ln295\""
Sep 11 19:13:41.133582 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:13:41.133556    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7c10b1-qffzf"]
Sep 11 19:13:41.222017 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:13:41.221973    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a63183b0-e6af-4055-9abf-94cd949a6eef-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-7c10b1-qffzf\" (UID: \"a63183b0-e6af-4055-9abf-94cd949a6eef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7c10b1-qffzf"
Sep 11 19:13:41.323470 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:13:41.323430    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a63183b0-e6af-4055-9abf-94cd949a6eef-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-7c10b1-qffzf\" (UID: \"a63183b0-e6af-4055-9abf-94cd949a6eef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7c10b1-qffzf"
Sep 11 19:13:41.324080 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:13:41.324062    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a63183b0-e6af-4055-9abf-94cd949a6eef-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-7c10b1-qffzf\" (UID: \"a63183b0-e6af-4055-9abf-94cd949a6eef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7c10b1-qffzf"
Sep 11 19:13:41.437012 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:13:41.436920    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7c10b1-qffzf"
Sep 11 19:13:41.710920 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:13:41.710887    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7c10b1-qffzf"]
Sep 11 19:13:41.712638 ip-10-0-134-189 kubenswrapper[2578]: W0911 19:13:41.712607    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda63183b0_e6af_4055_9abf_94cd949a6eef.slice/crio-feb057e62a47fa5682dfe2a15e921773804d704d5641a2833bceea970511d9b0 WatchSource:0}: Error finding container feb057e62a47fa5682dfe2a15e921773804d704d5641a2833bceea970511d9b0: Status 404 returned error can't find the container with id feb057e62a47fa5682dfe2a15e921773804d704d5641a2833bceea970511d9b0
Sep 11 19:13:42.603009 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:13:42.602975    2578 generic.go:356] "Generic (PLEG): container finished" podID="a63183b0-e6af-4055-9abf-94cd949a6eef" containerID="75f0ef66ac0d888aff1c36d5ecedbedd976887df2752f8694e17698a6f78690b" exitCode=0
Sep 11 19:13:42.603408 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:13:42.603056    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7c10b1-qffzf" event={"ID":"a63183b0-e6af-4055-9abf-94cd949a6eef","Type":"ContainerDied","Data":"75f0ef66ac0d888aff1c36d5ecedbedd976887df2752f8694e17698a6f78690b"}
Sep 11 19:13:42.603408 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:13:42.603107    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7c10b1-qffzf" event={"ID":"a63183b0-e6af-4055-9abf-94cd949a6eef","Type":"ContainerStarted","Data":"feb057e62a47fa5682dfe2a15e921773804d704d5641a2833bceea970511d9b0"}
Sep 11 19:13:43.741843 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:13:43.741818    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7c10b1-qffzf"
Sep 11 19:13:43.853238 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:13:43.853187    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a63183b0-e6af-4055-9abf-94cd949a6eef-maas-api-service-ca\") pod \"a63183b0-e6af-4055-9abf-94cd949a6eef\" (UID: \"a63183b0-e6af-4055-9abf-94cd949a6eef\") "
Sep 11 19:13:43.853579 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:13:43.853552    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a63183b0-e6af-4055-9abf-94cd949a6eef-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a63183b0-e6af-4055-9abf-94cd949a6eef" (UID: "a63183b0-e6af-4055-9abf-94cd949a6eef"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 19:13:43.954209 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:13:43.954173    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a63183b0-e6af-4055-9abf-94cd949a6eef-maas-api-service-ca\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:13:44.399842 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:13:44.399766    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-hsbsf_d0880abe-ec36-48e4-a511-ec663873fffa/console-operator/1.log"
Sep 11 19:13:44.405076 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:13:44.405042    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-hsbsf_d0880abe-ec36-48e4-a511-ec663873fffa/console-operator/1.log"
Sep 11 19:13:44.592909 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:13:44.592875    2578 scope.go:117] "RemoveContainer" containerID="42bf6979d1825340b0266da137bf14b110f83dbe57ed6d9916771046f09d40b5"
Sep 11 19:13:44.614636 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:13:44.614588    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7c10b1-qffzf"
Sep 11 19:13:44.614763 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:13:44.614636    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7c10b1-qffzf" event={"ID":"a63183b0-e6af-4055-9abf-94cd949a6eef","Type":"ContainerDied","Data":"feb057e62a47fa5682dfe2a15e921773804d704d5641a2833bceea970511d9b0"}
Sep 11 19:13:44.614763 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:13:44.614687    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="feb057e62a47fa5682dfe2a15e921773804d704d5641a2833bceea970511d9b0"
Sep 11 19:13:57.051578 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:13:57.051542    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7c10b1-85ffc8cddc-qw49x"]
Sep 11 19:13:57.052130 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:13:57.051836    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-7c10b1-85ffc8cddc-qw49x" podUID="3246a64d-dc72-4107-b5dd-6a6f1409a16f" containerName="payload-processing" containerID="cri-o://a460c90255a8eb095c65e1130965d467830724d9811fc76cadcce143418da62b" gracePeriod=30
Sep 11 19:13:57.059213 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:13:57.059177    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7c10b1-5694f468b6-7bkqx"]
Sep 11 19:13:57.059781 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:13:57.059422    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7c10b1-5694f468b6-7bkqx" podUID="4e269dec-ab2f-46f4-90a1-fb628dafcfb7" containerName="payload-pre-processing" containerID="cri-o://879a5bb94b9fb94ee68821eb48506ab8410f59bc4267c90e0d5b8011ad05d862" gracePeriod=30
Sep 11 19:13:57.263629 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:13:57.262024    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7c10b1-qffzf"]
Sep 11 19:13:57.265756 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:13:57.265723    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7c10b1-qffzf"]
Sep 11 19:13:57.322568 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:13:57.322540    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-7c10b1-85ffc8cddc-qw49x"
Sep 11 19:13:57.353187 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:13:57.353163    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7c10b1-5694f468b6-7bkqx"
Sep 11 19:13:57.378929 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:13:57.378903    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bs875\" (UniqueName: \"kubernetes.io/projected/3246a64d-dc72-4107-b5dd-6a6f1409a16f-kube-api-access-bs875\") pod \"3246a64d-dc72-4107-b5dd-6a6f1409a16f\" (UID: \"3246a64d-dc72-4107-b5dd-6a6f1409a16f\") "
Sep 11 19:13:57.378929 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:13:57.378934    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z2ljm\" (UniqueName: \"kubernetes.io/projected/4e269dec-ab2f-46f4-90a1-fb628dafcfb7-kube-api-access-z2ljm\") pod \"4e269dec-ab2f-46f4-90a1-fb628dafcfb7\" (UID: \"4e269dec-ab2f-46f4-90a1-fb628dafcfb7\") "
Sep 11 19:13:57.379185 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:13:57.378960    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e269dec-ab2f-46f4-90a1-fb628dafcfb7-plugins-config-volume\") pod \"4e269dec-ab2f-46f4-90a1-fb628dafcfb7\" (UID: \"4e269dec-ab2f-46f4-90a1-fb628dafcfb7\") "
Sep 11 19:13:57.379185 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:13:57.379027    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3246a64d-dc72-4107-b5dd-6a6f1409a16f-plugins-config-volume\") pod \"3246a64d-dc72-4107-b5dd-6a6f1409a16f\" (UID: \"3246a64d-dc72-4107-b5dd-6a6f1409a16f\") "
Sep 11 19:13:57.379444 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:13:57.379413    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e269dec-ab2f-46f4-90a1-fb628dafcfb7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4e269dec-ab2f-46f4-90a1-fb628dafcfb7" (UID: "4e269dec-ab2f-46f4-90a1-fb628dafcfb7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 19:13:57.379564 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:13:57.379456    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3246a64d-dc72-4107-b5dd-6a6f1409a16f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3246a64d-dc72-4107-b5dd-6a6f1409a16f" (UID: "3246a64d-dc72-4107-b5dd-6a6f1409a16f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 19:13:57.381159 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:13:57.381135    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3246a64d-dc72-4107-b5dd-6a6f1409a16f-kube-api-access-bs875" (OuterVolumeSpecName: "kube-api-access-bs875") pod "3246a64d-dc72-4107-b5dd-6a6f1409a16f" (UID: "3246a64d-dc72-4107-b5dd-6a6f1409a16f"). InnerVolumeSpecName "kube-api-access-bs875". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:13:57.381265 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:13:57.381221    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e269dec-ab2f-46f4-90a1-fb628dafcfb7-kube-api-access-z2ljm" (OuterVolumeSpecName: "kube-api-access-z2ljm") pod "4e269dec-ab2f-46f4-90a1-fb628dafcfb7" (UID: "4e269dec-ab2f-46f4-90a1-fb628dafcfb7"). InnerVolumeSpecName "kube-api-access-z2ljm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:13:57.480836 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:13:57.480783    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bs875\" (UniqueName: \"kubernetes.io/projected/3246a64d-dc72-4107-b5dd-6a6f1409a16f-kube-api-access-bs875\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:13:57.480836 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:13:57.480823    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2ljm\" (UniqueName: \"kubernetes.io/projected/4e269dec-ab2f-46f4-90a1-fb628dafcfb7-kube-api-access-z2ljm\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:13:57.480836 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:13:57.480838    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e269dec-ab2f-46f4-90a1-fb628dafcfb7-plugins-config-volume\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:13:57.481123 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:13:57.480852    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3246a64d-dc72-4107-b5dd-6a6f1409a16f-plugins-config-volume\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:13:57.690632 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:13:57.690517    2578 generic.go:356] "Generic (PLEG): container finished" podID="3246a64d-dc72-4107-b5dd-6a6f1409a16f" containerID="a460c90255a8eb095c65e1130965d467830724d9811fc76cadcce143418da62b" exitCode=0
Sep 11 19:13:57.690632 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:13:57.690572    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-7c10b1-85ffc8cddc-qw49x"
Sep 11 19:13:57.690870 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:13:57.690624    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-7c10b1-85ffc8cddc-qw49x" event={"ID":"3246a64d-dc72-4107-b5dd-6a6f1409a16f","Type":"ContainerDied","Data":"a460c90255a8eb095c65e1130965d467830724d9811fc76cadcce143418da62b"}
Sep 11 19:13:57.690870 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:13:57.690679    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-7c10b1-85ffc8cddc-qw49x" event={"ID":"3246a64d-dc72-4107-b5dd-6a6f1409a16f","Type":"ContainerDied","Data":"93717fdc52adf462bb20e7d7396abce9187b375fe7a20392123e1944e58779af"}
Sep 11 19:13:57.690870 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:13:57.690706    2578 scope.go:117] "RemoveContainer" containerID="a460c90255a8eb095c65e1130965d467830724d9811fc76cadcce143418da62b"
Sep 11 19:13:57.692093 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:13:57.692074    2578 generic.go:356] "Generic (PLEG): container finished" podID="4e269dec-ab2f-46f4-90a1-fb628dafcfb7" containerID="879a5bb94b9fb94ee68821eb48506ab8410f59bc4267c90e0d5b8011ad05d862" exitCode=0
Sep 11 19:13:57.692204 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:13:57.692113    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7c10b1-5694f468b6-7bkqx" event={"ID":"4e269dec-ab2f-46f4-90a1-fb628dafcfb7","Type":"ContainerDied","Data":"879a5bb94b9fb94ee68821eb48506ab8410f59bc4267c90e0d5b8011ad05d862"}
Sep 11 19:13:57.692204 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:13:57.692119    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7c10b1-5694f468b6-7bkqx"
Sep 11 19:13:57.692204 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:13:57.692146    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7c10b1-5694f468b6-7bkqx" event={"ID":"4e269dec-ab2f-46f4-90a1-fb628dafcfb7","Type":"ContainerDied","Data":"930be5660fbbf2aaed42db9beadd0401e4cabb87ea639847d07c115290523a4b"}
Sep 11 19:13:57.700741 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:13:57.700723    2578 scope.go:117] "RemoveContainer" containerID="a460c90255a8eb095c65e1130965d467830724d9811fc76cadcce143418da62b"
Sep 11 19:13:57.701070 ip-10-0-134-189 kubenswrapper[2578]: E0911 19:13:57.701046    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a460c90255a8eb095c65e1130965d467830724d9811fc76cadcce143418da62b\": container with ID starting with a460c90255a8eb095c65e1130965d467830724d9811fc76cadcce143418da62b not found: ID does not exist" containerID="a460c90255a8eb095c65e1130965d467830724d9811fc76cadcce143418da62b"
Sep 11 19:13:57.701166 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:13:57.701082    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a460c90255a8eb095c65e1130965d467830724d9811fc76cadcce143418da62b"} err="failed to get container status \"a460c90255a8eb095c65e1130965d467830724d9811fc76cadcce143418da62b\": rpc error: code = NotFound desc = could not find container \"a460c90255a8eb095c65e1130965d467830724d9811fc76cadcce143418da62b\": container with ID starting with a460c90255a8eb095c65e1130965d467830724d9811fc76cadcce143418da62b not found: ID does not exist"
Sep 11 19:13:57.701166 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:13:57.701107    2578 scope.go:117] "RemoveContainer" containerID="879a5bb94b9fb94ee68821eb48506ab8410f59bc4267c90e0d5b8011ad05d862"
Sep 11 19:13:57.712622 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:13:57.712580    2578 scope.go:117] "RemoveContainer" containerID="879a5bb94b9fb94ee68821eb48506ab8410f59bc4267c90e0d5b8011ad05d862"
Sep 11 19:13:57.713229 ip-10-0-134-189 kubenswrapper[2578]: E0911 19:13:57.713204    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"879a5bb94b9fb94ee68821eb48506ab8410f59bc4267c90e0d5b8011ad05d862\": container with ID starting with 879a5bb94b9fb94ee68821eb48506ab8410f59bc4267c90e0d5b8011ad05d862 not found: ID does not exist" containerID="879a5bb94b9fb94ee68821eb48506ab8410f59bc4267c90e0d5b8011ad05d862"
Sep 11 19:13:57.713323 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:13:57.713235    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"879a5bb94b9fb94ee68821eb48506ab8410f59bc4267c90e0d5b8011ad05d862"} err="failed to get container status \"879a5bb94b9fb94ee68821eb48506ab8410f59bc4267c90e0d5b8011ad05d862\": rpc error: code = NotFound desc = could not find container \"879a5bb94b9fb94ee68821eb48506ab8410f59bc4267c90e0d5b8011ad05d862\": container with ID starting with 879a5bb94b9fb94ee68821eb48506ab8410f59bc4267c90e0d5b8011ad05d862 not found: ID does not exist"
Sep 11 19:13:57.714776 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:13:57.714755    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7c10b1-85ffc8cddc-qw49x"]
Sep 11 19:13:57.719173 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:13:57.719151    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7c10b1-85ffc8cddc-qw49x"]
Sep 11 19:13:57.728071 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:13:57.728042    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7c10b1-5694f468b6-7bkqx"]
Sep 11 19:13:57.734013 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:13:57.733985    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7c10b1-5694f468b6-7bkqx"]
Sep 11 19:13:58.314501 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:13:58.314456    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7c10b1-5694f468b6-7bkqx" podUID="4e269dec-ab2f-46f4-90a1-fb628dafcfb7" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.70:9004: i/o timeout"
Sep 11 19:13:58.314949 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:13:58.314457    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-api-a-7c10b1-85ffc8cddc-qw49x" podUID="3246a64d-dc72-4107-b5dd-6a6f1409a16f" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.71:9004: i/o timeout"
Sep 11 19:13:58.401355 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:13:58.401257    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3246a64d-dc72-4107-b5dd-6a6f1409a16f" path="/var/lib/kubelet/pods/3246a64d-dc72-4107-b5dd-6a6f1409a16f/volumes"
Sep 11 19:13:58.401752 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:13:58.401732    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e269dec-ab2f-46f4-90a1-fb628dafcfb7" path="/var/lib/kubelet/pods/4e269dec-ab2f-46f4-90a1-fb628dafcfb7/volumes"
Sep 11 19:13:58.402105 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:13:58.402087    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a63183b0-e6af-4055-9abf-94cd949a6eef" path="/var/lib/kubelet/pods/a63183b0-e6af-4055-9abf-94cd949a6eef/volumes"
Sep 11 19:14:02.913807 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:02.913767    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c1aebc-52kb9"]
Sep 11 19:14:02.914248 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:02.914153    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a63183b0-e6af-4055-9abf-94cd949a6eef" containerName="revoke-keys"
Sep 11 19:14:02.914248 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:02.914166    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a63183b0-e6af-4055-9abf-94cd949a6eef" containerName="revoke-keys"
Sep 11 19:14:02.914248 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:02.914185    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e269dec-ab2f-46f4-90a1-fb628dafcfb7" containerName="payload-pre-processing"
Sep 11 19:14:02.914248 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:02.914192    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e269dec-ab2f-46f4-90a1-fb628dafcfb7" containerName="payload-pre-processing"
Sep 11 19:14:02.914248 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:02.914208    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3246a64d-dc72-4107-b5dd-6a6f1409a16f" containerName="payload-processing"
Sep 11 19:14:02.914248 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:02.914213    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3246a64d-dc72-4107-b5dd-6a6f1409a16f" containerName="payload-processing"
Sep 11 19:14:02.914478 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:02.914291    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e269dec-ab2f-46f4-90a1-fb628dafcfb7" containerName="payload-pre-processing"
Sep 11 19:14:02.914478 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:02.914300    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3246a64d-dc72-4107-b5dd-6a6f1409a16f" containerName="payload-processing"
Sep 11 19:14:02.914478 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:02.914307    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a63183b0-e6af-4055-9abf-94cd949a6eef" containerName="revoke-keys"
Sep 11 19:14:02.919222 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:02.919204    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c1aebc-52kb9"
Sep 11 19:14:02.921657 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:02.921579    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 19:14:02.921809 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:02.921582    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ln295\""
Sep 11 19:14:02.924944 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:02.924917    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c1aebc-52kb9"]
Sep 11 19:14:03.033208 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:03.033169    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/924cf0ba-3f3d-4756-a15b-61c9522b5777-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-c1aebc-52kb9\" (UID: \"924cf0ba-3f3d-4756-a15b-61c9522b5777\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c1aebc-52kb9"
Sep 11 19:14:03.134094 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:03.134055    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/924cf0ba-3f3d-4756-a15b-61c9522b5777-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-c1aebc-52kb9\" (UID: \"924cf0ba-3f3d-4756-a15b-61c9522b5777\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c1aebc-52kb9"
Sep 11 19:14:03.134725 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:03.134706    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/924cf0ba-3f3d-4756-a15b-61c9522b5777-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-c1aebc-52kb9\" (UID: \"924cf0ba-3f3d-4756-a15b-61c9522b5777\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c1aebc-52kb9"
Sep 11 19:14:03.231665 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:03.231590    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c1aebc-52kb9"
Sep 11 19:14:03.505297 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:03.505204    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c1aebc-52kb9"]
Sep 11 19:14:03.508285 ip-10-0-134-189 kubenswrapper[2578]: W0911 19:14:03.508257    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod924cf0ba_3f3d_4756_a15b_61c9522b5777.slice/crio-6df0a618260f99681c14c26b57a93afdbda414da9eddfeb86939a5f5939e3063 WatchSource:0}: Error finding container 6df0a618260f99681c14c26b57a93afdbda414da9eddfeb86939a5f5939e3063: Status 404 returned error can't find the container with id 6df0a618260f99681c14c26b57a93afdbda414da9eddfeb86939a5f5939e3063
Sep 11 19:14:03.725566 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:03.725528    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c1aebc-52kb9" event={"ID":"924cf0ba-3f3d-4756-a15b-61c9522b5777","Type":"ContainerStarted","Data":"6df0a618260f99681c14c26b57a93afdbda414da9eddfeb86939a5f5939e3063"}
Sep 11 19:14:04.118103 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:04.118003    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-852059-l2ccg"]
Sep 11 19:14:04.121808 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:04.121785    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-852059-l2ccg"
Sep 11 19:14:04.128869 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:04.128836    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-852059-l2ccg"]
Sep 11 19:14:04.243702 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:04.243661    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/845fe98f-3c9d-4dd3-b110-206694f4f13a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-852059-l2ccg\" (UID: \"845fe98f-3c9d-4dd3-b110-206694f4f13a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-852059-l2ccg"
Sep 11 19:14:04.344656 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:04.344497    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/845fe98f-3c9d-4dd3-b110-206694f4f13a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-852059-l2ccg\" (UID: \"845fe98f-3c9d-4dd3-b110-206694f4f13a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-852059-l2ccg"
Sep 11 19:14:04.345493 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:04.345467    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/845fe98f-3c9d-4dd3-b110-206694f4f13a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-852059-l2ccg\" (UID: \"845fe98f-3c9d-4dd3-b110-206694f4f13a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-852059-l2ccg"
Sep 11 19:14:04.434641 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:04.434533    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-852059-l2ccg"
Sep 11 19:14:04.708223 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:04.708092    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-852059-l2ccg"]
Sep 11 19:14:04.710741 ip-10-0-134-189 kubenswrapper[2578]: W0911 19:14:04.710716    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod845fe98f_3c9d_4dd3_b110_206694f4f13a.slice/crio-2de240206eabe8697bd0dc0e6753fab41e0afe1bfdddc3881e1e3962349296c2 WatchSource:0}: Error finding container 2de240206eabe8697bd0dc0e6753fab41e0afe1bfdddc3881e1e3962349296c2: Status 404 returned error can't find the container with id 2de240206eabe8697bd0dc0e6753fab41e0afe1bfdddc3881e1e3962349296c2
Sep 11 19:14:04.733680 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:04.733651    2578 generic.go:356] "Generic (PLEG): container finished" podID="924cf0ba-3f3d-4756-a15b-61c9522b5777" containerID="dc688307e065dcb247ad0d164e842205d0a4b2e456733f9dd30cb2bed2af93ac" exitCode=0
Sep 11 19:14:04.733825 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:04.733767    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c1aebc-52kb9" event={"ID":"924cf0ba-3f3d-4756-a15b-61c9522b5777","Type":"ContainerDied","Data":"dc688307e065dcb247ad0d164e842205d0a4b2e456733f9dd30cb2bed2af93ac"}
Sep 11 19:14:04.735266 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:04.735242    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-852059-l2ccg" event={"ID":"845fe98f-3c9d-4dd3-b110-206694f4f13a","Type":"ContainerStarted","Data":"2de240206eabe8697bd0dc0e6753fab41e0afe1bfdddc3881e1e3962349296c2"}
Sep 11 19:14:05.741477 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:05.741447    2578 generic.go:356] "Generic (PLEG): container finished" podID="845fe98f-3c9d-4dd3-b110-206694f4f13a" containerID="1e014b2897b963f4424a7682fd26436bcbdd145318e57dcae72b38453810a5ab" exitCode=0
Sep 11 19:14:05.741951 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:05.741534    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-852059-l2ccg" event={"ID":"845fe98f-3c9d-4dd3-b110-206694f4f13a","Type":"ContainerDied","Data":"1e014b2897b963f4424a7682fd26436bcbdd145318e57dcae72b38453810a5ab"}
Sep 11 19:14:05.880290 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:05.880262    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c1aebc-52kb9"
Sep 11 19:14:05.959798 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:05.959760    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/924cf0ba-3f3d-4756-a15b-61c9522b5777-maas-api-service-ca\") pod \"924cf0ba-3f3d-4756-a15b-61c9522b5777\" (UID: \"924cf0ba-3f3d-4756-a15b-61c9522b5777\") "
Sep 11 19:14:05.960135 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:05.960112    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/924cf0ba-3f3d-4756-a15b-61c9522b5777-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "924cf0ba-3f3d-4756-a15b-61c9522b5777" (UID: "924cf0ba-3f3d-4756-a15b-61c9522b5777"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 19:14:06.061177 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:06.061084    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/924cf0ba-3f3d-4756-a15b-61c9522b5777-maas-api-service-ca\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:14:06.748364 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:06.748334    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c1aebc-52kb9"
Sep 11 19:14:06.748771 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:06.748361    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c1aebc-52kb9" event={"ID":"924cf0ba-3f3d-4756-a15b-61c9522b5777","Type":"ContainerDied","Data":"6df0a618260f99681c14c26b57a93afdbda414da9eddfeb86939a5f5939e3063"}
Sep 11 19:14:06.748771 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:06.748410    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6df0a618260f99681c14c26b57a93afdbda414da9eddfeb86939a5f5939e3063"
Sep 11 19:14:06.886939 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:06.886912    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-852059-l2ccg"
Sep 11 19:14:06.968865 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:06.968835    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/845fe98f-3c9d-4dd3-b110-206694f4f13a-maas-api-service-ca\") pod \"845fe98f-3c9d-4dd3-b110-206694f4f13a\" (UID: \"845fe98f-3c9d-4dd3-b110-206694f4f13a\") "
Sep 11 19:14:06.969208 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:06.969186    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/845fe98f-3c9d-4dd3-b110-206694f4f13a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "845fe98f-3c9d-4dd3-b110-206694f4f13a" (UID: "845fe98f-3c9d-4dd3-b110-206694f4f13a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 19:14:07.070729 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:07.070631    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/845fe98f-3c9d-4dd3-b110-206694f4f13a-maas-api-service-ca\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:14:07.754827 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:07.754786    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-852059-l2ccg"
Sep 11 19:14:07.755204 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:07.754830    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-852059-l2ccg" event={"ID":"845fe98f-3c9d-4dd3-b110-206694f4f13a","Type":"ContainerDied","Data":"2de240206eabe8697bd0dc0e6753fab41e0afe1bfdddc3881e1e3962349296c2"}
Sep 11 19:14:07.755204 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:07.754903    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2de240206eabe8697bd0dc0e6753fab41e0afe1bfdddc3881e1e3962349296c2"
Sep 11 19:14:07.779647 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:07.779617    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c1aebc/distinct-2-w4-c1aebc-kserve-57879dd7f7-dfzxs"]
Sep 11 19:14:07.784713 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:07.784685    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c1aebc/distinct-w4-c1aebc-kserve-6fd464965d-rw6jw"]
Sep 11 19:14:07.789713 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:07.789683    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c1aebc/premium-w4-c1aebc-kserve-5455b5b4c-5jvlw"]
Sep 11 19:14:07.806698 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:07.806657    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c1aebc/simulator-w4-c1aebc-kserve-867479ffb6-76pm2"]
Sep 11 19:14:18.800528 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:18.800492    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c1aebc-52kb9"]
Sep 11 19:14:18.803650 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:18.803622    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c1aebc-52kb9"]
Sep 11 19:14:20.397977 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:20.397942    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="924cf0ba-3f3d-4756-a15b-61c9522b5777" path="/var/lib/kubelet/pods/924cf0ba-3f3d-4756-a15b-61c9522b5777/volumes"
Sep 11 19:14:21.938947 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:21.938886    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-852059-55b7677bf4-qmwk7"]
Sep 11 19:14:21.939658 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:21.939628    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-852059-55b7677bf4-qmwk7" podUID="c471430b-31b5-4ca4-9149-fd894fe3f14b" containerName="payload-pre-processing" containerID="cri-o://3989dbaf7c6a2659224a774deb92331398009924cebe8f4838287ffe65efad00" gracePeriod=30
Sep 11 19:14:22.110927 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:22.110880    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-852059-l2ccg"]
Sep 11 19:14:22.113298 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:22.113267    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-852059-l2ccg"]
Sep 11 19:14:22.194629 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:22.194548    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-852059-55b7677bf4-qmwk7"
Sep 11 19:14:22.326731 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:22.326699    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c471430b-31b5-4ca4-9149-fd894fe3f14b-plugins-config-volume\") pod \"c471430b-31b5-4ca4-9149-fd894fe3f14b\" (UID: \"c471430b-31b5-4ca4-9149-fd894fe3f14b\") "
Sep 11 19:14:22.326925 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:22.326757    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qxlw2\" (UniqueName: \"kubernetes.io/projected/c471430b-31b5-4ca4-9149-fd894fe3f14b-kube-api-access-qxlw2\") pod \"c471430b-31b5-4ca4-9149-fd894fe3f14b\" (UID: \"c471430b-31b5-4ca4-9149-fd894fe3f14b\") "
Sep 11 19:14:22.327118 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:22.327094    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c471430b-31b5-4ca4-9149-fd894fe3f14b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c471430b-31b5-4ca4-9149-fd894fe3f14b" (UID: "c471430b-31b5-4ca4-9149-fd894fe3f14b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 19:14:22.328988 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:22.328967    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c471430b-31b5-4ca4-9149-fd894fe3f14b-kube-api-access-qxlw2" (OuterVolumeSpecName: "kube-api-access-qxlw2") pod "c471430b-31b5-4ca4-9149-fd894fe3f14b" (UID: "c471430b-31b5-4ca4-9149-fd894fe3f14b"). InnerVolumeSpecName "kube-api-access-qxlw2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:14:22.396796 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:22.396757    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="845fe98f-3c9d-4dd3-b110-206694f4f13a" path="/var/lib/kubelet/pods/845fe98f-3c9d-4dd3-b110-206694f4f13a/volumes"
Sep 11 19:14:22.427862 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:22.427826    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c471430b-31b5-4ca4-9149-fd894fe3f14b-plugins-config-volume\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:14:22.427862 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:22.427853    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxlw2\" (UniqueName: \"kubernetes.io/projected/c471430b-31b5-4ca4-9149-fd894fe3f14b-kube-api-access-qxlw2\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:14:22.802447 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:22.802323    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-c1aebc/distinct-2-w4-c1aebc-kserve-57879dd7f7-dfzxs" podUID="b7748ab8-6e1c-4df9-9e28-aa25e75b0b4e" containerName="main" containerID="cri-o://e07a85729f476965b9dc0f2589eec2152981e61c1304730291691af56cf924d4" gracePeriod=45
Sep 11 19:14:22.808572 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:22.808530    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-c1aebc/distinct-w4-c1aebc-kserve-6fd464965d-rw6jw" podUID="373837e6-d918-4371-a7f4-52e52ce098f8" containerName="main" containerID="cri-o://c1b8a55e6937a23f5257185c9f7b4ed2929cedb0d2ce1f43d2549aa1ee998325" gracePeriod=45
Sep 11 19:14:22.818977 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:22.818937    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-c1aebc/premium-w4-c1aebc-kserve-5455b5b4c-5jvlw" podUID="fa97d84e-4f85-429c-9368-635a74daa253" containerName="main" containerID="cri-o://581b4d20d24c8272410ac01651828beb1a02f37f6fe35d9050655cf1342267ce" gracePeriod=45
Sep 11 19:14:22.832937 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:22.832896    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-c1aebc/simulator-w4-c1aebc-kserve-867479ffb6-76pm2" podUID="84d9a7b6-0329-418e-adf5-03ac9a1aa33d" containerName="main" containerID="cri-o://46e9e33c40887bcba635964cd81bf431bb6d3b3e3696fa58799eb2f193a600d9" gracePeriod=45
Sep 11 19:14:22.837284 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:22.837258    2578 generic.go:356] "Generic (PLEG): container finished" podID="c471430b-31b5-4ca4-9149-fd894fe3f14b" containerID="3989dbaf7c6a2659224a774deb92331398009924cebe8f4838287ffe65efad00" exitCode=0
Sep 11 19:14:22.837390 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:22.837309    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-852059-55b7677bf4-qmwk7"
Sep 11 19:14:22.837390 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:22.837346    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-852059-55b7677bf4-qmwk7" event={"ID":"c471430b-31b5-4ca4-9149-fd894fe3f14b","Type":"ContainerDied","Data":"3989dbaf7c6a2659224a774deb92331398009924cebe8f4838287ffe65efad00"}
Sep 11 19:14:22.837475 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:22.837404    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-852059-55b7677bf4-qmwk7" event={"ID":"c471430b-31b5-4ca4-9149-fd894fe3f14b","Type":"ContainerDied","Data":"61f36373939b8e9e917570bec0ea514e1de1c3f9f24ecaa52088a25ee2ebd23b"}
Sep 11 19:14:22.837475 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:22.837426    2578 scope.go:117] "RemoveContainer" containerID="3989dbaf7c6a2659224a774deb92331398009924cebe8f4838287ffe65efad00"
Sep 11 19:14:22.999436 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:22.999337    2578 scope.go:117] "RemoveContainer" containerID="3989dbaf7c6a2659224a774deb92331398009924cebe8f4838287ffe65efad00"
Sep 11 19:14:23.000069 ip-10-0-134-189 kubenswrapper[2578]: E0911 19:14:22.999987    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3989dbaf7c6a2659224a774deb92331398009924cebe8f4838287ffe65efad00\": container with ID starting with 3989dbaf7c6a2659224a774deb92331398009924cebe8f4838287ffe65efad00 not found: ID does not exist" containerID="3989dbaf7c6a2659224a774deb92331398009924cebe8f4838287ffe65efad00"
Sep 11 19:14:23.000069 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.000029    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3989dbaf7c6a2659224a774deb92331398009924cebe8f4838287ffe65efad00"} err="failed to get container status \"3989dbaf7c6a2659224a774deb92331398009924cebe8f4838287ffe65efad00\": rpc error: code = NotFound desc = could not find container \"3989dbaf7c6a2659224a774deb92331398009924cebe8f4838287ffe65efad00\": container with ID starting with 3989dbaf7c6a2659224a774deb92331398009924cebe8f4838287ffe65efad00 not found: ID does not exist"
Sep 11 19:14:23.007523 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.007424    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-852059-55b7677bf4-qmwk7"]
Sep 11 19:14:23.009768 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.009732    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-852059-55b7677bf4-qmwk7"]
Sep 11 19:14:23.147114 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.147084    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c1aebc/distinct-2-w4-c1aebc-kserve-57879dd7f7-dfzxs"
Sep 11 19:14:23.194116 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.194086    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c1aebc/distinct-w4-c1aebc-kserve-6fd464965d-rw6jw"
Sep 11 19:14:23.197198 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.197171    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c1aebc/premium-w4-c1aebc-kserve-5455b5b4c-5jvlw"
Sep 11 19:14:23.201103 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.201078    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c1aebc/simulator-w4-c1aebc-kserve-867479ffb6-76pm2"
Sep 11 19:14:23.236421 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.236388    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7748ab8-6e1c-4df9-9e28-aa25e75b0b4e-dshm\") pod \"b7748ab8-6e1c-4df9-9e28-aa25e75b0b4e\" (UID: \"b7748ab8-6e1c-4df9-9e28-aa25e75b0b4e\") "
Sep 11 19:14:23.236421 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.236431    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7748ab8-6e1c-4df9-9e28-aa25e75b0b4e-model-cache\") pod \"b7748ab8-6e1c-4df9-9e28-aa25e75b0b4e\" (UID: \"b7748ab8-6e1c-4df9-9e28-aa25e75b0b4e\") "
Sep 11 19:14:23.236654 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.236475    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7748ab8-6e1c-4df9-9e28-aa25e75b0b4e-tmp-dir\") pod \"b7748ab8-6e1c-4df9-9e28-aa25e75b0b4e\" (UID: \"b7748ab8-6e1c-4df9-9e28-aa25e75b0b4e\") "
Sep 11 19:14:23.236654 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.236521    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7748ab8-6e1c-4df9-9e28-aa25e75b0b4e-home\") pod \"b7748ab8-6e1c-4df9-9e28-aa25e75b0b4e\" (UID: \"b7748ab8-6e1c-4df9-9e28-aa25e75b0b4e\") "
Sep 11 19:14:23.236654 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.236630    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cbg8j\" (UniqueName: \"kubernetes.io/projected/b7748ab8-6e1c-4df9-9e28-aa25e75b0b4e-kube-api-access-cbg8j\") pod \"b7748ab8-6e1c-4df9-9e28-aa25e75b0b4e\" (UID: \"b7748ab8-6e1c-4df9-9e28-aa25e75b0b4e\") "
Sep 11 19:14:23.236756 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.236732    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7748ab8-6e1c-4df9-9e28-aa25e75b0b4e-tls-certs\") pod \"b7748ab8-6e1c-4df9-9e28-aa25e75b0b4e\" (UID: \"b7748ab8-6e1c-4df9-9e28-aa25e75b0b4e\") "
Sep 11 19:14:23.236797 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.236755    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7748ab8-6e1c-4df9-9e28-aa25e75b0b4e-model-cache" (OuterVolumeSpecName: "model-cache") pod "b7748ab8-6e1c-4df9-9e28-aa25e75b0b4e" (UID: "b7748ab8-6e1c-4df9-9e28-aa25e75b0b4e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:14:23.237044 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.237000    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7748ab8-6e1c-4df9-9e28-aa25e75b0b4e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b7748ab8-6e1c-4df9-9e28-aa25e75b0b4e" (UID: "b7748ab8-6e1c-4df9-9e28-aa25e75b0b4e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:14:23.237354 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.237082    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7748ab8-6e1c-4df9-9e28-aa25e75b0b4e-model-cache\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:14:23.237354 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.237102    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7748ab8-6e1c-4df9-9e28-aa25e75b0b4e-tmp-dir\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:14:23.237354 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.237323    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7748ab8-6e1c-4df9-9e28-aa25e75b0b4e-home" (OuterVolumeSpecName: "home") pod "b7748ab8-6e1c-4df9-9e28-aa25e75b0b4e" (UID: "b7748ab8-6e1c-4df9-9e28-aa25e75b0b4e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:14:23.239002 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.238967    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7748ab8-6e1c-4df9-9e28-aa25e75b0b4e-dshm" (OuterVolumeSpecName: "dshm") pod "b7748ab8-6e1c-4df9-9e28-aa25e75b0b4e" (UID: "b7748ab8-6e1c-4df9-9e28-aa25e75b0b4e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:14:23.239469 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.239441    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7748ab8-6e1c-4df9-9e28-aa25e75b0b4e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b7748ab8-6e1c-4df9-9e28-aa25e75b0b4e" (UID: "b7748ab8-6e1c-4df9-9e28-aa25e75b0b4e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 19:14:23.239831 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.239808    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7748ab8-6e1c-4df9-9e28-aa25e75b0b4e-kube-api-access-cbg8j" (OuterVolumeSpecName: "kube-api-access-cbg8j") pod "b7748ab8-6e1c-4df9-9e28-aa25e75b0b4e" (UID: "b7748ab8-6e1c-4df9-9e28-aa25e75b0b4e"). InnerVolumeSpecName "kube-api-access-cbg8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:14:23.338473 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.338374    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa97d84e-4f85-429c-9368-635a74daa253-tls-certs\") pod \"fa97d84e-4f85-429c-9368-635a74daa253\" (UID: \"fa97d84e-4f85-429c-9368-635a74daa253\") "
Sep 11 19:14:23.338473 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.338423    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/373837e6-d918-4371-a7f4-52e52ce098f8-dshm\") pod \"373837e6-d918-4371-a7f4-52e52ce098f8\" (UID: \"373837e6-d918-4371-a7f4-52e52ce098f8\") "
Sep 11 19:14:23.338473 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.338450    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84d9a7b6-0329-418e-adf5-03ac9a1aa33d-tls-certs\") pod \"84d9a7b6-0329-418e-adf5-03ac9a1aa33d\" (UID: \"84d9a7b6-0329-418e-adf5-03ac9a1aa33d\") "
Sep 11 19:14:23.338814 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.338474    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nplqg\" (UniqueName: \"kubernetes.io/projected/84d9a7b6-0329-418e-adf5-03ac9a1aa33d-kube-api-access-nplqg\") pod \"84d9a7b6-0329-418e-adf5-03ac9a1aa33d\" (UID: \"84d9a7b6-0329-418e-adf5-03ac9a1aa33d\") "
Sep 11 19:14:23.338814 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.338524    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa97d84e-4f85-429c-9368-635a74daa253-tmp-dir\") pod \"fa97d84e-4f85-429c-9368-635a74daa253\" (UID: \"fa97d84e-4f85-429c-9368-635a74daa253\") "
Sep 11 19:14:23.338814 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.338564    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-48g7x\" (UniqueName: \"kubernetes.io/projected/373837e6-d918-4371-a7f4-52e52ce098f8-kube-api-access-48g7x\") pod \"373837e6-d918-4371-a7f4-52e52ce098f8\" (UID: \"373837e6-d918-4371-a7f4-52e52ce098f8\") "
Sep 11 19:14:23.338814 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.338649    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa97d84e-4f85-429c-9368-635a74daa253-home\") pod \"fa97d84e-4f85-429c-9368-635a74daa253\" (UID: \"fa97d84e-4f85-429c-9368-635a74daa253\") "
Sep 11 19:14:23.338814 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.338675    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wlfwn\" (UniqueName: \"kubernetes.io/projected/fa97d84e-4f85-429c-9368-635a74daa253-kube-api-access-wlfwn\") pod \"fa97d84e-4f85-429c-9368-635a74daa253\" (UID: \"fa97d84e-4f85-429c-9368-635a74daa253\") "
Sep 11 19:14:23.339086 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.338819    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa97d84e-4f85-429c-9368-635a74daa253-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fa97d84e-4f85-429c-9368-635a74daa253" (UID: "fa97d84e-4f85-429c-9368-635a74daa253"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:14:23.339086 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.338925    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa97d84e-4f85-429c-9368-635a74daa253-home" (OuterVolumeSpecName: "home") pod "fa97d84e-4f85-429c-9368-635a74daa253" (UID: "fa97d84e-4f85-429c-9368-635a74daa253"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:14:23.339086 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.339000    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84d9a7b6-0329-418e-adf5-03ac9a1aa33d-tmp-dir\") pod \"84d9a7b6-0329-418e-adf5-03ac9a1aa33d\" (UID: \"84d9a7b6-0329-418e-adf5-03ac9a1aa33d\") "
Sep 11 19:14:23.339330 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.339295    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/373837e6-d918-4371-a7f4-52e52ce098f8-tls-certs\") pod \"373837e6-d918-4371-a7f4-52e52ce098f8\" (UID: \"373837e6-d918-4371-a7f4-52e52ce098f8\") "
Sep 11 19:14:23.339480 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.339465    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/373837e6-d918-4371-a7f4-52e52ce098f8-tmp-dir\") pod \"373837e6-d918-4371-a7f4-52e52ce098f8\" (UID: \"373837e6-d918-4371-a7f4-52e52ce098f8\") "
Sep 11 19:14:23.339803 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.339788    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84d9a7b6-0329-418e-adf5-03ac9a1aa33d-home\") pod \"84d9a7b6-0329-418e-adf5-03ac9a1aa33d\" (UID: \"84d9a7b6-0329-418e-adf5-03ac9a1aa33d\") "
Sep 11 19:14:23.339948 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.339934    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84d9a7b6-0329-418e-adf5-03ac9a1aa33d-dshm\") pod \"84d9a7b6-0329-418e-adf5-03ac9a1aa33d\" (UID: \"84d9a7b6-0329-418e-adf5-03ac9a1aa33d\") "
Sep 11 19:14:23.340068 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.340054    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84d9a7b6-0329-418e-adf5-03ac9a1aa33d-model-cache\") pod \"84d9a7b6-0329-418e-adf5-03ac9a1aa33d\" (UID: \"84d9a7b6-0329-418e-adf5-03ac9a1aa33d\") "
Sep 11 19:14:23.340177 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.340164    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa97d84e-4f85-429c-9368-635a74daa253-model-cache\") pod \"fa97d84e-4f85-429c-9368-635a74daa253\" (UID: \"fa97d84e-4f85-429c-9368-635a74daa253\") "
Sep 11 19:14:23.340285 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.340271    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa97d84e-4f85-429c-9368-635a74daa253-dshm\") pod \"fa97d84e-4f85-429c-9368-635a74daa253\" (UID: \"fa97d84e-4f85-429c-9368-635a74daa253\") "
Sep 11 19:14:23.340388 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.340376    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/373837e6-d918-4371-a7f4-52e52ce098f8-home\") pod \"373837e6-d918-4371-a7f4-52e52ce098f8\" (UID: \"373837e6-d918-4371-a7f4-52e52ce098f8\") "
Sep 11 19:14:23.340487 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.340475    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/373837e6-d918-4371-a7f4-52e52ce098f8-model-cache\") pod \"373837e6-d918-4371-a7f4-52e52ce098f8\" (UID: \"373837e6-d918-4371-a7f4-52e52ce098f8\") "
Sep 11 19:14:23.341000 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.340977    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7748ab8-6e1c-4df9-9e28-aa25e75b0b4e-dshm\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:14:23.341000 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.341000    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7748ab8-6e1c-4df9-9e28-aa25e75b0b4e-home\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:14:23.341181 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.341013    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa97d84e-4f85-429c-9368-635a74daa253-tmp-dir\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:14:23.341181 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.341030    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbg8j\" (UniqueName: \"kubernetes.io/projected/b7748ab8-6e1c-4df9-9e28-aa25e75b0b4e-kube-api-access-cbg8j\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:14:23.341181 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.341042    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa97d84e-4f85-429c-9368-635a74daa253-home\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:14:23.341181 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.341056    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7748ab8-6e1c-4df9-9e28-aa25e75b0b4e-tls-certs\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:14:23.341364 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.339193    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84d9a7b6-0329-418e-adf5-03ac9a1aa33d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "84d9a7b6-0329-418e-adf5-03ac9a1aa33d" (UID: "84d9a7b6-0329-418e-adf5-03ac9a1aa33d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:14:23.341364 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.341276    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/373837e6-d918-4371-a7f4-52e52ce098f8-model-cache" (OuterVolumeSpecName: "model-cache") pod "373837e6-d918-4371-a7f4-52e52ce098f8" (UID: "373837e6-d918-4371-a7f4-52e52ce098f8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:14:23.341475 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.341457    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/373837e6-d918-4371-a7f4-52e52ce098f8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "373837e6-d918-4371-a7f4-52e52ce098f8" (UID: "373837e6-d918-4371-a7f4-52e52ce098f8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:14:23.341842 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.341572    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa97d84e-4f85-429c-9368-635a74daa253-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fa97d84e-4f85-429c-9368-635a74daa253" (UID: "fa97d84e-4f85-429c-9368-635a74daa253"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 19:14:23.341842 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.341583    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84d9a7b6-0329-418e-adf5-03ac9a1aa33d-kube-api-access-nplqg" (OuterVolumeSpecName: "kube-api-access-nplqg") pod "84d9a7b6-0329-418e-adf5-03ac9a1aa33d" (UID: "84d9a7b6-0329-418e-adf5-03ac9a1aa33d"). InnerVolumeSpecName "kube-api-access-nplqg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:14:23.341842 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.341693    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84d9a7b6-0329-418e-adf5-03ac9a1aa33d-home" (OuterVolumeSpecName: "home") pod "84d9a7b6-0329-418e-adf5-03ac9a1aa33d" (UID: "84d9a7b6-0329-418e-adf5-03ac9a1aa33d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:14:23.341842 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.341828    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa97d84e-4f85-429c-9368-635a74daa253-model-cache" (OuterVolumeSpecName: "model-cache") pod "fa97d84e-4f85-429c-9368-635a74daa253" (UID: "fa97d84e-4f85-429c-9368-635a74daa253"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:14:23.342105 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.341886    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa97d84e-4f85-429c-9368-635a74daa253-kube-api-access-wlfwn" (OuterVolumeSpecName: "kube-api-access-wlfwn") pod "fa97d84e-4f85-429c-9368-635a74daa253" (UID: "fa97d84e-4f85-429c-9368-635a74daa253"). InnerVolumeSpecName "kube-api-access-wlfwn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:14:23.342105 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.341931    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/373837e6-d918-4371-a7f4-52e52ce098f8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "373837e6-d918-4371-a7f4-52e52ce098f8" (UID: "373837e6-d918-4371-a7f4-52e52ce098f8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 19:14:23.342105 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.341966    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84d9a7b6-0329-418e-adf5-03ac9a1aa33d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "84d9a7b6-0329-418e-adf5-03ac9a1aa33d" (UID: "84d9a7b6-0329-418e-adf5-03ac9a1aa33d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 19:14:23.342105 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.341951    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/373837e6-d918-4371-a7f4-52e52ce098f8-dshm" (OuterVolumeSpecName: "dshm") pod "373837e6-d918-4371-a7f4-52e52ce098f8" (UID: "373837e6-d918-4371-a7f4-52e52ce098f8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:14:23.342105 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.342020    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84d9a7b6-0329-418e-adf5-03ac9a1aa33d-model-cache" (OuterVolumeSpecName: "model-cache") pod "84d9a7b6-0329-418e-adf5-03ac9a1aa33d" (UID: "84d9a7b6-0329-418e-adf5-03ac9a1aa33d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:14:23.342516 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.342096    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/373837e6-d918-4371-a7f4-52e52ce098f8-home" (OuterVolumeSpecName: "home") pod "373837e6-d918-4371-a7f4-52e52ce098f8" (UID: "373837e6-d918-4371-a7f4-52e52ce098f8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:14:23.342516 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.342241    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/373837e6-d918-4371-a7f4-52e52ce098f8-kube-api-access-48g7x" (OuterVolumeSpecName: "kube-api-access-48g7x") pod "373837e6-d918-4371-a7f4-52e52ce098f8" (UID: "373837e6-d918-4371-a7f4-52e52ce098f8"). InnerVolumeSpecName "kube-api-access-48g7x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:14:23.343687 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.343666    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa97d84e-4f85-429c-9368-635a74daa253-dshm" (OuterVolumeSpecName: "dshm") pod "fa97d84e-4f85-429c-9368-635a74daa253" (UID: "fa97d84e-4f85-429c-9368-635a74daa253"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:14:23.343765 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.343721    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84d9a7b6-0329-418e-adf5-03ac9a1aa33d-dshm" (OuterVolumeSpecName: "dshm") pod "84d9a7b6-0329-418e-adf5-03ac9a1aa33d" (UID: "84d9a7b6-0329-418e-adf5-03ac9a1aa33d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:14:23.442450 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.442413    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48g7x\" (UniqueName: \"kubernetes.io/projected/373837e6-d918-4371-a7f4-52e52ce098f8-kube-api-access-48g7x\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:14:23.442450 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.442444    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlfwn\" (UniqueName: \"kubernetes.io/projected/fa97d84e-4f85-429c-9368-635a74daa253-kube-api-access-wlfwn\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:14:23.442450 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.442455    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84d9a7b6-0329-418e-adf5-03ac9a1aa33d-tmp-dir\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:14:23.442742 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.442467    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/373837e6-d918-4371-a7f4-52e52ce098f8-tls-certs\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:14:23.442742 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.442475    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/373837e6-d918-4371-a7f4-52e52ce098f8-tmp-dir\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:14:23.442742 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.442482    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84d9a7b6-0329-418e-adf5-03ac9a1aa33d-home\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:14:23.442742 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.442489    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84d9a7b6-0329-418e-adf5-03ac9a1aa33d-dshm\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:14:23.442742 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.442496    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84d9a7b6-0329-418e-adf5-03ac9a1aa33d-model-cache\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:14:23.442742 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.442503    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa97d84e-4f85-429c-9368-635a74daa253-model-cache\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:14:23.442742 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.442511    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa97d84e-4f85-429c-9368-635a74daa253-dshm\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:14:23.442742 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.442517    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/373837e6-d918-4371-a7f4-52e52ce098f8-home\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:14:23.442742 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.442524    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/373837e6-d918-4371-a7f4-52e52ce098f8-model-cache\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:14:23.442742 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.442531    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa97d84e-4f85-429c-9368-635a74daa253-tls-certs\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:14:23.442742 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.442538    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/373837e6-d918-4371-a7f4-52e52ce098f8-dshm\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:14:23.442742 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.442545    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84d9a7b6-0329-418e-adf5-03ac9a1aa33d-tls-certs\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:14:23.442742 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.442552    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nplqg\" (UniqueName: \"kubernetes.io/projected/84d9a7b6-0329-418e-adf5-03ac9a1aa33d-kube-api-access-nplqg\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:14:23.770016 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.769984    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7788-c16b48"]
Sep 11 19:14:23.770359 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.770348    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7748ab8-6e1c-4df9-9e28-aa25e75b0b4e" containerName="main"
Sep 11 19:14:23.770406 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.770361    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7748ab8-6e1c-4df9-9e28-aa25e75b0b4e" containerName="main"
Sep 11 19:14:23.770406 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.770370    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="924cf0ba-3f3d-4756-a15b-61c9522b5777" containerName="revoke-keys"
Sep 11 19:14:23.770406 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.770376    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="924cf0ba-3f3d-4756-a15b-61c9522b5777" containerName="revoke-keys"
Sep 11 19:14:23.770406 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.770382    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="845fe98f-3c9d-4dd3-b110-206694f4f13a" containerName="revoke-keys"
Sep 11 19:14:23.770406 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.770388    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="845fe98f-3c9d-4dd3-b110-206694f4f13a" containerName="revoke-keys"
Sep 11 19:14:23.770406 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.770395    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84d9a7b6-0329-418e-adf5-03ac9a1aa33d" containerName="main"
Sep 11 19:14:23.770406 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.770400    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="84d9a7b6-0329-418e-adf5-03ac9a1aa33d" containerName="main"
Sep 11 19:14:23.770406 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.770405    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa97d84e-4f85-429c-9368-635a74daa253" containerName="main"
Sep 11 19:14:23.770659 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.770410    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa97d84e-4f85-429c-9368-635a74daa253" containerName="main"
Sep 11 19:14:23.770659 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.770420    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c471430b-31b5-4ca4-9149-fd894fe3f14b" containerName="payload-pre-processing"
Sep 11 19:14:23.770659 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.770425    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c471430b-31b5-4ca4-9149-fd894fe3f14b" containerName="payload-pre-processing"
Sep 11 19:14:23.770659 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.770433    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="373837e6-d918-4371-a7f4-52e52ce098f8" containerName="main"
Sep 11 19:14:23.770659 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.770438    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="373837e6-d918-4371-a7f4-52e52ce098f8" containerName="main"
Sep 11 19:14:23.770659 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.770487    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="84d9a7b6-0329-418e-adf5-03ac9a1aa33d" containerName="main"
Sep 11 19:14:23.770659 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.770495    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="924cf0ba-3f3d-4756-a15b-61c9522b5777" containerName="revoke-keys"
Sep 11 19:14:23.770659 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.770500    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="845fe98f-3c9d-4dd3-b110-206694f4f13a" containerName="revoke-keys"
Sep 11 19:14:23.770659 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.770505    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7748ab8-6e1c-4df9-9e28-aa25e75b0b4e" containerName="main"
Sep 11 19:14:23.770659 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.770511    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c471430b-31b5-4ca4-9149-fd894fe3f14b" containerName="payload-pre-processing"
Sep 11 19:14:23.770659 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.770516    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa97d84e-4f85-429c-9368-635a74daa253" containerName="main"
Sep 11 19:14:23.770659 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.770522    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="373837e6-d918-4371-a7f4-52e52ce098f8" containerName="main"
Sep 11 19:14:23.774788 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.774771    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7788-c16b48"
Sep 11 19:14:23.778006 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.777979    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-jxjbf\""
Sep 11 19:14:23.783025 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.783000    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7788-c16b48"]
Sep 11 19:14:23.845204 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.845166    2578 generic.go:356] "Generic (PLEG): container finished" podID="84d9a7b6-0329-418e-adf5-03ac9a1aa33d" containerID="46e9e33c40887bcba635964cd81bf431bb6d3b3e3696fa58799eb2f193a600d9" exitCode=0
Sep 11 19:14:23.845419 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.845205    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c1aebc/simulator-w4-c1aebc-kserve-867479ffb6-76pm2" event={"ID":"84d9a7b6-0329-418e-adf5-03ac9a1aa33d","Type":"ContainerDied","Data":"46e9e33c40887bcba635964cd81bf431bb6d3b3e3696fa58799eb2f193a600d9"}
Sep 11 19:14:23.845419 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.845256    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c1aebc/simulator-w4-c1aebc-kserve-867479ffb6-76pm2"
Sep 11 19:14:23.845419 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.845270    2578 scope.go:117] "RemoveContainer" containerID="46e9e33c40887bcba635964cd81bf431bb6d3b3e3696fa58799eb2f193a600d9"
Sep 11 19:14:23.845419 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.845258    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c1aebc/simulator-w4-c1aebc-kserve-867479ffb6-76pm2" event={"ID":"84d9a7b6-0329-418e-adf5-03ac9a1aa33d","Type":"ContainerDied","Data":"8fefb69c375204550b94e6a628407afffe8540446fd17fe4493e0b04611f888e"}
Sep 11 19:14:23.847096 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.846968    2578 generic.go:356] "Generic (PLEG): container finished" podID="fa97d84e-4f85-429c-9368-635a74daa253" containerID="581b4d20d24c8272410ac01651828beb1a02f37f6fe35d9050655cf1342267ce" exitCode=0
Sep 11 19:14:23.847096 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.847039    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c1aebc/premium-w4-c1aebc-kserve-5455b5b4c-5jvlw"
Sep 11 19:14:23.847096 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.847051    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c1aebc/premium-w4-c1aebc-kserve-5455b5b4c-5jvlw" event={"ID":"fa97d84e-4f85-429c-9368-635a74daa253","Type":"ContainerDied","Data":"581b4d20d24c8272410ac01651828beb1a02f37f6fe35d9050655cf1342267ce"}
Sep 11 19:14:23.847096 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.847088    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c1aebc/premium-w4-c1aebc-kserve-5455b5b4c-5jvlw" event={"ID":"fa97d84e-4f85-429c-9368-635a74daa253","Type":"ContainerDied","Data":"ad35bd19801b2113186fcf64db93d21a67f7f37d896e42cf2e54ae190511fdcd"}
Sep 11 19:14:23.848634 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.848608    2578 generic.go:356] "Generic (PLEG): container finished" podID="373837e6-d918-4371-a7f4-52e52ce098f8" containerID="c1b8a55e6937a23f5257185c9f7b4ed2929cedb0d2ce1f43d2549aa1ee998325" exitCode=0
Sep 11 19:14:23.848759 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.848661    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c1aebc/distinct-w4-c1aebc-kserve-6fd464965d-rw6jw"
Sep 11 19:14:23.848759 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.848691    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c1aebc/distinct-w4-c1aebc-kserve-6fd464965d-rw6jw" event={"ID":"373837e6-d918-4371-a7f4-52e52ce098f8","Type":"ContainerDied","Data":"c1b8a55e6937a23f5257185c9f7b4ed2929cedb0d2ce1f43d2549aa1ee998325"}
Sep 11 19:14:23.848759 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.848725    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c1aebc/distinct-w4-c1aebc-kserve-6fd464965d-rw6jw" event={"ID":"373837e6-d918-4371-a7f4-52e52ce098f8","Type":"ContainerDied","Data":"21314396e6275dd68e09a3d1e6218737d0401affb7e525ff89e422cebe31ed9a"}
Sep 11 19:14:23.850332 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.850309    2578 generic.go:356] "Generic (PLEG): container finished" podID="b7748ab8-6e1c-4df9-9e28-aa25e75b0b4e" containerID="e07a85729f476965b9dc0f2589eec2152981e61c1304730291691af56cf924d4" exitCode=0
Sep 11 19:14:23.850457 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.850366    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c1aebc/distinct-2-w4-c1aebc-kserve-57879dd7f7-dfzxs"
Sep 11 19:14:23.850457 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.850388    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c1aebc/distinct-2-w4-c1aebc-kserve-57879dd7f7-dfzxs" event={"ID":"b7748ab8-6e1c-4df9-9e28-aa25e75b0b4e","Type":"ContainerDied","Data":"e07a85729f476965b9dc0f2589eec2152981e61c1304730291691af56cf924d4"}
Sep 11 19:14:23.850457 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.850419    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c1aebc/distinct-2-w4-c1aebc-kserve-57879dd7f7-dfzxs" event={"ID":"b7748ab8-6e1c-4df9-9e28-aa25e75b0b4e","Type":"ContainerDied","Data":"f21a372f91ac637ba36b1fff2201d435a176fe36c2d98a879bc59048a18cb6eb"}
Sep 11 19:14:23.857030 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.856932    2578 scope.go:117] "RemoveContainer" containerID="46e9e33c40887bcba635964cd81bf431bb6d3b3e3696fa58799eb2f193a600d9"
Sep 11 19:14:23.857482 ip-10-0-134-189 kubenswrapper[2578]: E0911 19:14:23.857456    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46e9e33c40887bcba635964cd81bf431bb6d3b3e3696fa58799eb2f193a600d9\": container with ID starting with 46e9e33c40887bcba635964cd81bf431bb6d3b3e3696fa58799eb2f193a600d9 not found: ID does not exist" containerID="46e9e33c40887bcba635964cd81bf431bb6d3b3e3696fa58799eb2f193a600d9"
Sep 11 19:14:23.857576 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.857493    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46e9e33c40887bcba635964cd81bf431bb6d3b3e3696fa58799eb2f193a600d9"} err="failed to get container status \"46e9e33c40887bcba635964cd81bf431bb6d3b3e3696fa58799eb2f193a600d9\": rpc error: code = NotFound desc = could not find container \"46e9e33c40887bcba635964cd81bf431bb6d3b3e3696fa58799eb2f193a600d9\": container with ID starting with 46e9e33c40887bcba635964cd81bf431bb6d3b3e3696fa58799eb2f193a600d9 not found: ID does not exist"
Sep 11 19:14:23.857576 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.857517    2578 scope.go:117] "RemoveContainer" containerID="581b4d20d24c8272410ac01651828beb1a02f37f6fe35d9050655cf1342267ce"
Sep 11 19:14:23.867001 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.866981    2578 scope.go:117] "RemoveContainer" containerID="581b4d20d24c8272410ac01651828beb1a02f37f6fe35d9050655cf1342267ce"
Sep 11 19:14:23.867317 ip-10-0-134-189 kubenswrapper[2578]: E0911 19:14:23.867296    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"581b4d20d24c8272410ac01651828beb1a02f37f6fe35d9050655cf1342267ce\": container with ID starting with 581b4d20d24c8272410ac01651828beb1a02f37f6fe35d9050655cf1342267ce not found: ID does not exist" containerID="581b4d20d24c8272410ac01651828beb1a02f37f6fe35d9050655cf1342267ce"
Sep 11 19:14:23.867364 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.867326    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"581b4d20d24c8272410ac01651828beb1a02f37f6fe35d9050655cf1342267ce"} err="failed to get container status \"581b4d20d24c8272410ac01651828beb1a02f37f6fe35d9050655cf1342267ce\": rpc error: code = NotFound desc = could not find container \"581b4d20d24c8272410ac01651828beb1a02f37f6fe35d9050655cf1342267ce\": container with ID starting with 581b4d20d24c8272410ac01651828beb1a02f37f6fe35d9050655cf1342267ce not found: ID does not exist"
Sep 11 19:14:23.867364 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.867345    2578 scope.go:117] "RemoveContainer" containerID="c1b8a55e6937a23f5257185c9f7b4ed2929cedb0d2ce1f43d2549aa1ee998325"
Sep 11 19:14:23.874692 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.874665    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c1aebc/simulator-w4-c1aebc-kserve-867479ffb6-76pm2"]
Sep 11 19:14:23.876383 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.876361    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-c1aebc/simulator-w4-c1aebc-kserve-867479ffb6-76pm2"]
Sep 11 19:14:23.876815 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.876800    2578 scope.go:117] "RemoveContainer" containerID="c1b8a55e6937a23f5257185c9f7b4ed2929cedb0d2ce1f43d2549aa1ee998325"
Sep 11 19:14:23.877118 ip-10-0-134-189 kubenswrapper[2578]: E0911 19:14:23.877102    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1b8a55e6937a23f5257185c9f7b4ed2929cedb0d2ce1f43d2549aa1ee998325\": container with ID starting with c1b8a55e6937a23f5257185c9f7b4ed2929cedb0d2ce1f43d2549aa1ee998325 not found: ID does not exist" containerID="c1b8a55e6937a23f5257185c9f7b4ed2929cedb0d2ce1f43d2549aa1ee998325"
Sep 11 19:14:23.877163 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.877125    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1b8a55e6937a23f5257185c9f7b4ed2929cedb0d2ce1f43d2549aa1ee998325"} err="failed to get container status \"c1b8a55e6937a23f5257185c9f7b4ed2929cedb0d2ce1f43d2549aa1ee998325\": rpc error: code = NotFound desc = could not find container \"c1b8a55e6937a23f5257185c9f7b4ed2929cedb0d2ce1f43d2549aa1ee998325\": container with ID starting with c1b8a55e6937a23f5257185c9f7b4ed2929cedb0d2ce1f43d2549aa1ee998325 not found: ID does not exist"
Sep 11 19:14:23.877163 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.877143    2578 scope.go:117] "RemoveContainer" containerID="e07a85729f476965b9dc0f2589eec2152981e61c1304730291691af56cf924d4"
Sep 11 19:14:23.885375 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.885337    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c1aebc/distinct-w4-c1aebc-kserve-6fd464965d-rw6jw"]
Sep 11 19:14:23.886055 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.886035    2578 scope.go:117] "RemoveContainer" containerID="e07a85729f476965b9dc0f2589eec2152981e61c1304730291691af56cf924d4"
Sep 11 19:14:23.886411 ip-10-0-134-189 kubenswrapper[2578]: E0911 19:14:23.886391    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e07a85729f476965b9dc0f2589eec2152981e61c1304730291691af56cf924d4\": container with ID starting with e07a85729f476965b9dc0f2589eec2152981e61c1304730291691af56cf924d4 not found: ID does not exist" containerID="e07a85729f476965b9dc0f2589eec2152981e61c1304730291691af56cf924d4"
Sep 11 19:14:23.886487 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.886419    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e07a85729f476965b9dc0f2589eec2152981e61c1304730291691af56cf924d4"} err="failed to get container status \"e07a85729f476965b9dc0f2589eec2152981e61c1304730291691af56cf924d4\": rpc error: code = NotFound desc = could not find container \"e07a85729f476965b9dc0f2589eec2152981e61c1304730291691af56cf924d4\": container with ID starting with e07a85729f476965b9dc0f2589eec2152981e61c1304730291691af56cf924d4 not found: ID does not exist"
Sep 11 19:14:23.887393 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.887290    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-c1aebc/distinct-w4-c1aebc-kserve-6fd464965d-rw6jw"]
Sep 11 19:14:23.896487 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.896439    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c1aebc/premium-w4-c1aebc-kserve-5455b5b4c-5jvlw"]
Sep 11 19:14:23.898462 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.898438    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-c1aebc/premium-w4-c1aebc-kserve-5455b5b4c-5jvlw"]
Sep 11 19:14:23.907169 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.907144    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c1aebc/distinct-2-w4-c1aebc-kserve-57879dd7f7-dfzxs"]
Sep 11 19:14:23.911952 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.911927    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-c1aebc/distinct-2-w4-c1aebc-kserve-57879dd7f7-dfzxs"]
Sep 11 19:14:23.946492 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:23.946434    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz7tq\" (UniqueName: \"kubernetes.io/projected/7c902995-60ae-4f2f-8d98-a78f4d68fa8a-kube-api-access-gz7tq\") pod \"test-curl-7788-c16b48\" (UID: \"7c902995-60ae-4f2f-8d98-a78f4d68fa8a\") " pod="opendatahub/test-curl-7788-c16b48"
Sep 11 19:14:24.047769 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:24.047666    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz7tq\" (UniqueName: \"kubernetes.io/projected/7c902995-60ae-4f2f-8d98-a78f4d68fa8a-kube-api-access-gz7tq\") pod \"test-curl-7788-c16b48\" (UID: \"7c902995-60ae-4f2f-8d98-a78f4d68fa8a\") " pod="opendatahub/test-curl-7788-c16b48"
Sep 11 19:14:24.057546 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:24.057520    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz7tq\" (UniqueName: \"kubernetes.io/projected/7c902995-60ae-4f2f-8d98-a78f4d68fa8a-kube-api-access-gz7tq\") pod \"test-curl-7788-c16b48\" (UID: \"7c902995-60ae-4f2f-8d98-a78f4d68fa8a\") " pod="opendatahub/test-curl-7788-c16b48"
Sep 11 19:14:24.085414 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:24.085361    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7788-c16b48"
Sep 11 19:14:24.360549 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:24.360470    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7788-c16b48"]
Sep 11 19:14:24.364104 ip-10-0-134-189 kubenswrapper[2578]: W0911 19:14:24.364075    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7c902995_60ae_4f2f_8d98_a78f4d68fa8a.slice/crio-6b6afe74ef78e769657e6404897fb2f35db8821f1055fea686e20f53177f58e4 WatchSource:0}: Error finding container 6b6afe74ef78e769657e6404897fb2f35db8821f1055fea686e20f53177f58e4: Status 404 returned error can't find the container with id 6b6afe74ef78e769657e6404897fb2f35db8821f1055fea686e20f53177f58e4
Sep 11 19:14:24.397822 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:24.397794    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="373837e6-d918-4371-a7f4-52e52ce098f8" path="/var/lib/kubelet/pods/373837e6-d918-4371-a7f4-52e52ce098f8/volumes"
Sep 11 19:14:24.398198 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:24.398185    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84d9a7b6-0329-418e-adf5-03ac9a1aa33d" path="/var/lib/kubelet/pods/84d9a7b6-0329-418e-adf5-03ac9a1aa33d/volumes"
Sep 11 19:14:24.398520 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:24.398508    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7748ab8-6e1c-4df9-9e28-aa25e75b0b4e" path="/var/lib/kubelet/pods/b7748ab8-6e1c-4df9-9e28-aa25e75b0b4e/volumes"
Sep 11 19:14:24.399174 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:24.399156    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c471430b-31b5-4ca4-9149-fd894fe3f14b" path="/var/lib/kubelet/pods/c471430b-31b5-4ca4-9149-fd894fe3f14b/volumes"
Sep 11 19:14:24.399523 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:24.399511    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa97d84e-4f85-429c-9368-635a74daa253" path="/var/lib/kubelet/pods/fa97d84e-4f85-429c-9368-635a74daa253/volumes"
Sep 11 19:14:24.860868 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:24.860827    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7788-c16b48" event={"ID":"7c902995-60ae-4f2f-8d98-a78f4d68fa8a","Type":"ContainerStarted","Data":"f2d46957bc87525b3a9e8cb603707afc323cc0f74815b0f84e3a341ddb94ee1d"}
Sep 11 19:14:24.861094 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:24.860876    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7788-c16b48" event={"ID":"7c902995-60ae-4f2f-8d98-a78f4d68fa8a","Type":"ContainerStarted","Data":"6b6afe74ef78e769657e6404897fb2f35db8821f1055fea686e20f53177f58e4"}
Sep 11 19:14:24.876912 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:24.876863    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7788-c16b48" podStartSLOduration=1.876845343 podStartE2EDuration="1.876845343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:14:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:14:24.875849168 +0000 UTC m=+1540.953439722" watchObservedRunningTime="2026-09-11 19:14:24.876845343 +0000 UTC m=+1540.954435896"
Sep 11 19:14:25.454357 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:25.454321    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c1aebc-openshift-default-5c87857866-4s5ww"]
Sep 11 19:14:25.454872 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:25.454585    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-c1aebc-openshift-default-5c87857866-4s5ww" podUID="f9b6e770-8c4d-4531-88d9-ab04cb50effc" containerName="istio-proxy" containerID="cri-o://6a30fd111fb853f849fa04f8a4479d325402f315685099084cac0e054f4d989b" gracePeriod=30
Sep 11 19:14:25.523659 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:25.523615    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7788-c16b48"]
Sep 11 19:14:25.528569 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:25.528535    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7788-c16b48"]
Sep 11 19:14:25.867922 ip-10-0-134-189 kubenswrapper[2578]: E0911 19:14:25.867888    2578 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7788-c16b48" container="test-curl-7788-c16b48"
Sep 11 19:14:26.872772 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:26.872728    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7788-c16b48" podUID="7c902995-60ae-4f2f-8d98-a78f4d68fa8a" containerName="test-curl-7788-c16b48" containerID="cri-o://f2d46957bc87525b3a9e8cb603707afc323cc0f74815b0f84e3a341ddb94ee1d" gracePeriod=2
Sep 11 19:14:28.640361 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:28.640324    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-19647e-n7l8w"]
Sep 11 19:14:28.640913 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:28.640703    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c902995-60ae-4f2f-8d98-a78f4d68fa8a" containerName="test-curl-7788-c16b48"
Sep 11 19:14:28.640913 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:28.640717    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c902995-60ae-4f2f-8d98-a78f4d68fa8a" containerName="test-curl-7788-c16b48"
Sep 11 19:14:28.640913 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:28.640776    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c902995-60ae-4f2f-8d98-a78f4d68fa8a" containerName="test-curl-7788-c16b48"
Sep 11 19:14:28.643908 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:28.643891    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-19647e-n7l8w"
Sep 11 19:14:28.645713 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:28.645683    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 19:14:28.645835 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:28.645779    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ln295\""
Sep 11 19:14:28.650662 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:28.650637    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-19647e-n7l8w"]
Sep 11 19:14:28.690504 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:28.690462    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8b21472-44e3-41bf-a7fd-bc06141fa13a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-19647e-n7l8w\" (UID: \"c8b21472-44e3-41bf-a7fd-bc06141fa13a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-19647e-n7l8w"
Sep 11 19:14:28.791840 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:28.791803    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8b21472-44e3-41bf-a7fd-bc06141fa13a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-19647e-n7l8w\" (UID: \"c8b21472-44e3-41bf-a7fd-bc06141fa13a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-19647e-n7l8w"
Sep 11 19:14:28.792399 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:28.792377    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8b21472-44e3-41bf-a7fd-bc06141fa13a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-19647e-n7l8w\" (UID: \"c8b21472-44e3-41bf-a7fd-bc06141fa13a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-19647e-n7l8w"
Sep 11 19:14:28.955678 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:28.955642    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-19647e-n7l8w"
Sep 11 19:14:29.071714 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:29.071688    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7788-c16b48"
Sep 11 19:14:29.093745 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:29.093715    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gz7tq\" (UniqueName: \"kubernetes.io/projected/7c902995-60ae-4f2f-8d98-a78f4d68fa8a-kube-api-access-gz7tq\") pod \"7c902995-60ae-4f2f-8d98-a78f4d68fa8a\" (UID: \"7c902995-60ae-4f2f-8d98-a78f4d68fa8a\") "
Sep 11 19:14:29.095976 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:29.095950    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c902995-60ae-4f2f-8d98-a78f4d68fa8a-kube-api-access-gz7tq" (OuterVolumeSpecName: "kube-api-access-gz7tq") pod "7c902995-60ae-4f2f-8d98-a78f4d68fa8a" (UID: "7c902995-60ae-4f2f-8d98-a78f4d68fa8a"). InnerVolumeSpecName "kube-api-access-gz7tq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:14:29.194999 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:29.194906    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gz7tq\" (UniqueName: \"kubernetes.io/projected/7c902995-60ae-4f2f-8d98-a78f4d68fa8a-kube-api-access-gz7tq\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:14:29.233279 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:29.233247    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-19647e-n7l8w"]
Sep 11 19:14:29.236373 ip-10-0-134-189 kubenswrapper[2578]: W0911 19:14:29.236346    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8b21472_44e3_41bf_a7fd_bc06141fa13a.slice/crio-ca64a1aeeac34c26e76494b963828434172ef1fb8fa621ab2a7cde6af8e75ade WatchSource:0}: Error finding container ca64a1aeeac34c26e76494b963828434172ef1fb8fa621ab2a7cde6af8e75ade: Status 404 returned error can't find the container with id ca64a1aeeac34c26e76494b963828434172ef1fb8fa621ab2a7cde6af8e75ade
Sep 11 19:14:29.890779 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:29.890746    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-19647e-n7l8w" event={"ID":"c8b21472-44e3-41bf-a7fd-bc06141fa13a","Type":"ContainerStarted","Data":"be6ce4f403a36a7dcb2efd8a002fc1efa6af513c1c5fbafbec0e7c4c0ad460ca"}
Sep 11 19:14:29.891218 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:29.890848    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-19647e-n7l8w" event={"ID":"c8b21472-44e3-41bf-a7fd-bc06141fa13a","Type":"ContainerStarted","Data":"ca64a1aeeac34c26e76494b963828434172ef1fb8fa621ab2a7cde6af8e75ade"}
Sep 11 19:14:29.892112 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:29.892088    2578 generic.go:356] "Generic (PLEG): container finished" podID="7c902995-60ae-4f2f-8d98-a78f4d68fa8a" containerID="f2d46957bc87525b3a9e8cb603707afc323cc0f74815b0f84e3a341ddb94ee1d" exitCode=137
Sep 11 19:14:29.892221 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:29.892133    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7788-c16b48"
Sep 11 19:14:29.892262 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:29.892136    2578 scope.go:117] "RemoveContainer" containerID="f2d46957bc87525b3a9e8cb603707afc323cc0f74815b0f84e3a341ddb94ee1d"
Sep 11 19:14:29.901033 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:29.901018    2578 scope.go:117] "RemoveContainer" containerID="f2d46957bc87525b3a9e8cb603707afc323cc0f74815b0f84e3a341ddb94ee1d"
Sep 11 19:14:29.901306 ip-10-0-134-189 kubenswrapper[2578]: E0911 19:14:29.901285    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2d46957bc87525b3a9e8cb603707afc323cc0f74815b0f84e3a341ddb94ee1d\": container with ID starting with f2d46957bc87525b3a9e8cb603707afc323cc0f74815b0f84e3a341ddb94ee1d not found: ID does not exist" containerID="f2d46957bc87525b3a9e8cb603707afc323cc0f74815b0f84e3a341ddb94ee1d"
Sep 11 19:14:29.901352 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:29.901315    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2d46957bc87525b3a9e8cb603707afc323cc0f74815b0f84e3a341ddb94ee1d"} err="failed to get container status \"f2d46957bc87525b3a9e8cb603707afc323cc0f74815b0f84e3a341ddb94ee1d\": rpc error: code = NotFound desc = could not find container \"f2d46957bc87525b3a9e8cb603707afc323cc0f74815b0f84e3a341ddb94ee1d\": container with ID starting with f2d46957bc87525b3a9e8cb603707afc323cc0f74815b0f84e3a341ddb94ee1d not found: ID does not exist"
Sep 11 19:14:29.904725 ip-10-0-134-189 kubenswrapper[2578]: E0911 19:14:29.904689    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7788-c16b48\" is forbidden: User \"system:node:ip-10-0-134-189.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-189.ec2.internal' and this object" podUID="7c902995-60ae-4f2f-8d98-a78f4d68fa8a" pod="opendatahub/test-curl-7788-c16b48"
Sep 11 19:14:29.904988 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:29.904953    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-19647e-n7l8w" podStartSLOduration=1.904938238 podStartE2EDuration="1.904938238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:14:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:14:29.903329516 +0000 UTC m=+1545.980920082" watchObservedRunningTime="2026-09-11 19:14:29.904938238 +0000 UTC m=+1545.982528793"
Sep 11 19:14:30.397159 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:30.397125    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c902995-60ae-4f2f-8d98-a78f4d68fa8a" path="/var/lib/kubelet/pods/7c902995-60ae-4f2f-8d98-a78f4d68fa8a/volumes"
Sep 11 19:14:30.705661 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:30.705636    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c1aebc-openshift-default-5c87857866-4s5ww"
Sep 11 19:14:30.706000 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:30.705981    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9b6e770-8c4d-4531-88d9-ab04cb50effc-workload-certs\") pod \"f9b6e770-8c4d-4531-88d9-ab04cb50effc\" (UID: \"f9b6e770-8c4d-4531-88d9-ab04cb50effc\") "
Sep 11 19:14:30.706058 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:30.706012    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9b6e770-8c4d-4531-88d9-ab04cb50effc-workload-socket\") pod \"f9b6e770-8c4d-4531-88d9-ab04cb50effc\" (UID: \"f9b6e770-8c4d-4531-88d9-ab04cb50effc\") "
Sep 11 19:14:30.706058 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:30.706048    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9b6e770-8c4d-4531-88d9-ab04cb50effc-istiod-ca-cert\") pod \"f9b6e770-8c4d-4531-88d9-ab04cb50effc\" (UID: \"f9b6e770-8c4d-4531-88d9-ab04cb50effc\") "
Sep 11 19:14:30.706147 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:30.706080    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9b6e770-8c4d-4531-88d9-ab04cb50effc-istio-podinfo\") pod \"f9b6e770-8c4d-4531-88d9-ab04cb50effc\" (UID: \"f9b6e770-8c4d-4531-88d9-ab04cb50effc\") "
Sep 11 19:14:30.706147 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:30.706107    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9b6e770-8c4d-4531-88d9-ab04cb50effc-istio-envoy\") pod \"f9b6e770-8c4d-4531-88d9-ab04cb50effc\" (UID: \"f9b6e770-8c4d-4531-88d9-ab04cb50effc\") "
Sep 11 19:14:30.706147 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:30.706129    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hk67v\" (UniqueName: \"kubernetes.io/projected/f9b6e770-8c4d-4531-88d9-ab04cb50effc-kube-api-access-hk67v\") pod \"f9b6e770-8c4d-4531-88d9-ab04cb50effc\" (UID: \"f9b6e770-8c4d-4531-88d9-ab04cb50effc\") "
Sep 11 19:14:30.706306 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:30.706247    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9b6e770-8c4d-4531-88d9-ab04cb50effc-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f9b6e770-8c4d-4531-88d9-ab04cb50effc" (UID: "f9b6e770-8c4d-4531-88d9-ab04cb50effc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:14:30.706377 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:30.706295    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9b6e770-8c4d-4531-88d9-ab04cb50effc-istio-data\") pod \"f9b6e770-8c4d-4531-88d9-ab04cb50effc\" (UID: \"f9b6e770-8c4d-4531-88d9-ab04cb50effc\") "
Sep 11 19:14:30.706377 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:30.706356    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9b6e770-8c4d-4531-88d9-ab04cb50effc-credential-socket\") pod \"f9b6e770-8c4d-4531-88d9-ab04cb50effc\" (UID: \"f9b6e770-8c4d-4531-88d9-ab04cb50effc\") "
Sep 11 19:14:30.706481 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:30.706375    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9b6e770-8c4d-4531-88d9-ab04cb50effc-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f9b6e770-8c4d-4531-88d9-ab04cb50effc" (UID: "f9b6e770-8c4d-4531-88d9-ab04cb50effc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:14:30.706481 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:30.706407    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9b6e770-8c4d-4531-88d9-ab04cb50effc-istio-token\") pod \"f9b6e770-8c4d-4531-88d9-ab04cb50effc\" (UID: \"f9b6e770-8c4d-4531-88d9-ab04cb50effc\") "
Sep 11 19:14:30.706612 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:30.706573    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9b6e770-8c4d-4531-88d9-ab04cb50effc-istio-data" (OuterVolumeSpecName: "istio-data") pod "f9b6e770-8c4d-4531-88d9-ab04cb50effc" (UID: "f9b6e770-8c4d-4531-88d9-ab04cb50effc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:14:30.706675 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:30.706508    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9b6e770-8c4d-4531-88d9-ab04cb50effc-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f9b6e770-8c4d-4531-88d9-ab04cb50effc" (UID: "f9b6e770-8c4d-4531-88d9-ab04cb50effc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 19:14:30.706675 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:30.706587    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9b6e770-8c4d-4531-88d9-ab04cb50effc-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f9b6e770-8c4d-4531-88d9-ab04cb50effc" (UID: "f9b6e770-8c4d-4531-88d9-ab04cb50effc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:14:30.706771 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:30.706738    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9b6e770-8c4d-4531-88d9-ab04cb50effc-istio-data\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:14:30.706771 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:30.706758    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9b6e770-8c4d-4531-88d9-ab04cb50effc-credential-socket\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:14:30.706878 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:30.706772    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9b6e770-8c4d-4531-88d9-ab04cb50effc-workload-certs\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:14:30.706878 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:30.706786    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9b6e770-8c4d-4531-88d9-ab04cb50effc-workload-socket\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:14:30.706878 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:30.706800    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9b6e770-8c4d-4531-88d9-ab04cb50effc-istiod-ca-cert\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:14:30.708529 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:30.708507    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9b6e770-8c4d-4531-88d9-ab04cb50effc-istio-token" (OuterVolumeSpecName: "istio-token") pod "f9b6e770-8c4d-4531-88d9-ab04cb50effc" (UID: "f9b6e770-8c4d-4531-88d9-ab04cb50effc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:14:30.708645 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:30.708581    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f9b6e770-8c4d-4531-88d9-ab04cb50effc-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f9b6e770-8c4d-4531-88d9-ab04cb50effc" (UID: "f9b6e770-8c4d-4531-88d9-ab04cb50effc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 19:14:30.708840 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:30.708822    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9b6e770-8c4d-4531-88d9-ab04cb50effc-kube-api-access-hk67v" (OuterVolumeSpecName: "kube-api-access-hk67v") pod "f9b6e770-8c4d-4531-88d9-ab04cb50effc" (UID: "f9b6e770-8c4d-4531-88d9-ab04cb50effc"). InnerVolumeSpecName "kube-api-access-hk67v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:14:30.708896 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:30.708818    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9b6e770-8c4d-4531-88d9-ab04cb50effc-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f9b6e770-8c4d-4531-88d9-ab04cb50effc" (UID: "f9b6e770-8c4d-4531-88d9-ab04cb50effc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:14:30.807795 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:30.807751    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9b6e770-8c4d-4531-88d9-ab04cb50effc-istio-podinfo\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:14:30.807795 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:30.807785    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9b6e770-8c4d-4531-88d9-ab04cb50effc-istio-envoy\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:14:30.807795 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:30.807795    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hk67v\" (UniqueName: \"kubernetes.io/projected/f9b6e770-8c4d-4531-88d9-ab04cb50effc-kube-api-access-hk67v\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:14:30.807795 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:30.807805    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9b6e770-8c4d-4531-88d9-ab04cb50effc-istio-token\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:14:30.899369 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:30.899336    2578 generic.go:356] "Generic (PLEG): container finished" podID="f9b6e770-8c4d-4531-88d9-ab04cb50effc" containerID="6a30fd111fb853f849fa04f8a4479d325402f315685099084cac0e054f4d989b" exitCode=0
Sep 11 19:14:30.899856 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:30.899384    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c1aebc-openshift-default-5c87857866-4s5ww"
Sep 11 19:14:30.899856 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:30.899422    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c1aebc-openshift-default-5c87857866-4s5ww" event={"ID":"f9b6e770-8c4d-4531-88d9-ab04cb50effc","Type":"ContainerDied","Data":"6a30fd111fb853f849fa04f8a4479d325402f315685099084cac0e054f4d989b"}
Sep 11 19:14:30.899856 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:30.899485    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c1aebc-openshift-default-5c87857866-4s5ww" event={"ID":"f9b6e770-8c4d-4531-88d9-ab04cb50effc","Type":"ContainerDied","Data":"4ed77de082915928c36da19ec79b9940b800e51aeb27915cded683e36203a9af"}
Sep 11 19:14:30.899856 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:30.899521    2578 scope.go:117] "RemoveContainer" containerID="6a30fd111fb853f849fa04f8a4479d325402f315685099084cac0e054f4d989b"
Sep 11 19:14:30.901139 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:30.901118    2578 generic.go:356] "Generic (PLEG): container finished" podID="c8b21472-44e3-41bf-a7fd-bc06141fa13a" containerID="be6ce4f403a36a7dcb2efd8a002fc1efa6af513c1c5fbafbec0e7c4c0ad460ca" exitCode=0
Sep 11 19:14:30.901261 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:30.901151    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-19647e-n7l8w" event={"ID":"c8b21472-44e3-41bf-a7fd-bc06141fa13a","Type":"ContainerDied","Data":"be6ce4f403a36a7dcb2efd8a002fc1efa6af513c1c5fbafbec0e7c4c0ad460ca"}
Sep 11 19:14:30.910332 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:30.910315    2578 scope.go:117] "RemoveContainer" containerID="236544bd6a2ada481215866aedad1fb4c3cee2a3555d6ec1e253198f982252b5"
Sep 11 19:14:30.919825 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:30.919809    2578 scope.go:117] "RemoveContainer" containerID="6a30fd111fb853f849fa04f8a4479d325402f315685099084cac0e054f4d989b"
Sep 11 19:14:30.920126 ip-10-0-134-189 kubenswrapper[2578]: E0911 19:14:30.920107    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a30fd111fb853f849fa04f8a4479d325402f315685099084cac0e054f4d989b\": container with ID starting with 6a30fd111fb853f849fa04f8a4479d325402f315685099084cac0e054f4d989b not found: ID does not exist" containerID="6a30fd111fb853f849fa04f8a4479d325402f315685099084cac0e054f4d989b"
Sep 11 19:14:30.920197 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:30.920135    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a30fd111fb853f849fa04f8a4479d325402f315685099084cac0e054f4d989b"} err="failed to get container status \"6a30fd111fb853f849fa04f8a4479d325402f315685099084cac0e054f4d989b\": rpc error: code = NotFound desc = could not find container \"6a30fd111fb853f849fa04f8a4479d325402f315685099084cac0e054f4d989b\": container with ID starting with 6a30fd111fb853f849fa04f8a4479d325402f315685099084cac0e054f4d989b not found: ID does not exist"
Sep 11 19:14:30.920197 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:30.920152    2578 scope.go:117] "RemoveContainer" containerID="236544bd6a2ada481215866aedad1fb4c3cee2a3555d6ec1e253198f982252b5"
Sep 11 19:14:30.920404 ip-10-0-134-189 kubenswrapper[2578]: E0911 19:14:30.920389    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"236544bd6a2ada481215866aedad1fb4c3cee2a3555d6ec1e253198f982252b5\": container with ID starting with 236544bd6a2ada481215866aedad1fb4c3cee2a3555d6ec1e253198f982252b5 not found: ID does not exist" containerID="236544bd6a2ada481215866aedad1fb4c3cee2a3555d6ec1e253198f982252b5"
Sep 11 19:14:30.920449 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:30.920408    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"236544bd6a2ada481215866aedad1fb4c3cee2a3555d6ec1e253198f982252b5"} err="failed to get container status \"236544bd6a2ada481215866aedad1fb4c3cee2a3555d6ec1e253198f982252b5\": rpc error: code = NotFound desc = could not find container \"236544bd6a2ada481215866aedad1fb4c3cee2a3555d6ec1e253198f982252b5\": container with ID starting with 236544bd6a2ada481215866aedad1fb4c3cee2a3555d6ec1e253198f982252b5 not found: ID does not exist"
Sep 11 19:14:30.928144 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:30.928114    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c1aebc-openshift-default-5c87857866-4s5ww"]
Sep 11 19:14:30.935250 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:30.935176    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-c1aebc-openshift-default-5c87857866-4s5ww"]
Sep 11 19:14:32.044149 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:32.044117    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-19647e-n7l8w"
Sep 11 19:14:32.119862 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:32.119825    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8b21472-44e3-41bf-a7fd-bc06141fa13a-maas-api-service-ca\") pod \"c8b21472-44e3-41bf-a7fd-bc06141fa13a\" (UID: \"c8b21472-44e3-41bf-a7fd-bc06141fa13a\") "
Sep 11 19:14:32.120212 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:32.120187    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8b21472-44e3-41bf-a7fd-bc06141fa13a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c8b21472-44e3-41bf-a7fd-bc06141fa13a" (UID: "c8b21472-44e3-41bf-a7fd-bc06141fa13a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 19:14:32.220635 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:32.220569    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8b21472-44e3-41bf-a7fd-bc06141fa13a-maas-api-service-ca\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:14:32.397491 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:32.397460    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9b6e770-8c4d-4531-88d9-ab04cb50effc" path="/var/lib/kubelet/pods/f9b6e770-8c4d-4531-88d9-ab04cb50effc/volumes"
Sep 11 19:14:32.914885 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:32.914852    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-19647e-n7l8w" event={"ID":"c8b21472-44e3-41bf-a7fd-bc06141fa13a","Type":"ContainerDied","Data":"ca64a1aeeac34c26e76494b963828434172ef1fb8fa621ab2a7cde6af8e75ade"}
Sep 11 19:14:32.914885 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:32.914867    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-19647e-n7l8w"
Sep 11 19:14:32.915095 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:32.914902    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca64a1aeeac34c26e76494b963828434172ef1fb8fa621ab2a7cde6af8e75ade"
Sep 11 19:14:34.320278 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:34.320245    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3b7391/auth-test-model-3b7391-kserve-74db88fc4c-smtts"]
Sep 11 19:14:39.400471 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:39.400434    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-19647e-n7l8w"]
Sep 11 19:14:39.403101 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:39.403074    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-19647e-n7l8w"]
Sep 11 19:14:40.396710 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:40.396677    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8b21472-44e3-41bf-a7fd-bc06141fa13a" path="/var/lib/kubelet/pods/c8b21472-44e3-41bf-a7fd-bc06141fa13a/volumes"
Sep 11 19:14:45.919142 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:45.919105    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-8280eb/sub-test-model-8280eb-kserve-7d555d5589-hkn2n"]
Sep 11 19:14:45.919527 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:45.919449    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9b6e770-8c4d-4531-88d9-ab04cb50effc" containerName="istio-proxy"
Sep 11 19:14:45.919527 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:45.919460    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9b6e770-8c4d-4531-88d9-ab04cb50effc" containerName="istio-proxy"
Sep 11 19:14:45.919527 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:45.919467    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9b6e770-8c4d-4531-88d9-ab04cb50effc" containerName="istio-proxy"
Sep 11 19:14:45.919527 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:45.919473    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9b6e770-8c4d-4531-88d9-ab04cb50effc" containerName="istio-proxy"
Sep 11 19:14:45.919527 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:45.919482    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8b21472-44e3-41bf-a7fd-bc06141fa13a" containerName="revoke-keys"
Sep 11 19:14:45.919527 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:45.919488    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8b21472-44e3-41bf-a7fd-bc06141fa13a" containerName="revoke-keys"
Sep 11 19:14:45.919742 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:45.919545    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8b21472-44e3-41bf-a7fd-bc06141fa13a" containerName="revoke-keys"
Sep 11 19:14:45.919742 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:45.919556    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9b6e770-8c4d-4531-88d9-ab04cb50effc" containerName="istio-proxy"
Sep 11 19:14:45.919742 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:45.919563    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9b6e770-8c4d-4531-88d9-ab04cb50effc" containerName="istio-proxy"
Sep 11 19:14:45.924064 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:45.924041    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-8280eb/sub-test-model-8280eb-kserve-7d555d5589-hkn2n"
Sep 11 19:14:45.926754 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:45.926729    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-8280eb\"/\"openshift-service-ca.crt\""
Sep 11 19:14:45.926871 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:45.926754    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-8280eb\"/\"kube-root-ca.crt\""
Sep 11 19:14:45.926871 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:45.926735    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-8280eb\"/\"default-dockercfg-5k22v\""
Sep 11 19:14:45.926871 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:45.926735    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-8280eb\"/\"sub-test-model-8280eb-kserve-self-signed-certs\""
Sep 11 19:14:45.932543 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:45.932518    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-8280eb/sub-test-model-8280eb-kserve-7d555d5589-hkn2n"]
Sep 11 19:14:46.046912 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:46.046867    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9df66431-f3dc-4457-9dfe-2de4fe8f3a92-model-cache\") pod \"sub-test-model-8280eb-kserve-7d555d5589-hkn2n\" (UID: \"9df66431-f3dc-4457-9dfe-2de4fe8f3a92\") " pod="ai-tenant-e2e-shared-a-w3-8280eb/sub-test-model-8280eb-kserve-7d555d5589-hkn2n"
Sep 11 19:14:46.047074 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:46.046924    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9df66431-f3dc-4457-9dfe-2de4fe8f3a92-home\") pod \"sub-test-model-8280eb-kserve-7d555d5589-hkn2n\" (UID: \"9df66431-f3dc-4457-9dfe-2de4fe8f3a92\") " pod="ai-tenant-e2e-shared-a-w3-8280eb/sub-test-model-8280eb-kserve-7d555d5589-hkn2n"
Sep 11 19:14:46.047074 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:46.046952    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9df66431-f3dc-4457-9dfe-2de4fe8f3a92-tmp-dir\") pod \"sub-test-model-8280eb-kserve-7d555d5589-hkn2n\" (UID: \"9df66431-f3dc-4457-9dfe-2de4fe8f3a92\") " pod="ai-tenant-e2e-shared-a-w3-8280eb/sub-test-model-8280eb-kserve-7d555d5589-hkn2n"
Sep 11 19:14:46.047074 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:46.047012    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9df66431-f3dc-4457-9dfe-2de4fe8f3a92-tls-certs\") pod \"sub-test-model-8280eb-kserve-7d555d5589-hkn2n\" (UID: \"9df66431-f3dc-4457-9dfe-2de4fe8f3a92\") " pod="ai-tenant-e2e-shared-a-w3-8280eb/sub-test-model-8280eb-kserve-7d555d5589-hkn2n"
Sep 11 19:14:46.047074 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:46.047057    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6r2p\" (UniqueName: \"kubernetes.io/projected/9df66431-f3dc-4457-9dfe-2de4fe8f3a92-kube-api-access-q6r2p\") pod \"sub-test-model-8280eb-kserve-7d555d5589-hkn2n\" (UID: \"9df66431-f3dc-4457-9dfe-2de4fe8f3a92\") " pod="ai-tenant-e2e-shared-a-w3-8280eb/sub-test-model-8280eb-kserve-7d555d5589-hkn2n"
Sep 11 19:14:46.047235 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:46.047118    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9df66431-f3dc-4457-9dfe-2de4fe8f3a92-dshm\") pod \"sub-test-model-8280eb-kserve-7d555d5589-hkn2n\" (UID: \"9df66431-f3dc-4457-9dfe-2de4fe8f3a92\") " pod="ai-tenant-e2e-shared-a-w3-8280eb/sub-test-model-8280eb-kserve-7d555d5589-hkn2n"
Sep 11 19:14:46.148537 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:46.148498    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6r2p\" (UniqueName: \"kubernetes.io/projected/9df66431-f3dc-4457-9dfe-2de4fe8f3a92-kube-api-access-q6r2p\") pod \"sub-test-model-8280eb-kserve-7d555d5589-hkn2n\" (UID: \"9df66431-f3dc-4457-9dfe-2de4fe8f3a92\") " pod="ai-tenant-e2e-shared-a-w3-8280eb/sub-test-model-8280eb-kserve-7d555d5589-hkn2n"
Sep 11 19:14:46.148774 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:46.148558    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9df66431-f3dc-4457-9dfe-2de4fe8f3a92-dshm\") pod \"sub-test-model-8280eb-kserve-7d555d5589-hkn2n\" (UID: \"9df66431-f3dc-4457-9dfe-2de4fe8f3a92\") " pod="ai-tenant-e2e-shared-a-w3-8280eb/sub-test-model-8280eb-kserve-7d555d5589-hkn2n"
Sep 11 19:14:46.148774 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:46.148644    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9df66431-f3dc-4457-9dfe-2de4fe8f3a92-model-cache\") pod \"sub-test-model-8280eb-kserve-7d555d5589-hkn2n\" (UID: \"9df66431-f3dc-4457-9dfe-2de4fe8f3a92\") " pod="ai-tenant-e2e-shared-a-w3-8280eb/sub-test-model-8280eb-kserve-7d555d5589-hkn2n"
Sep 11 19:14:46.148774 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:46.148679    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9df66431-f3dc-4457-9dfe-2de4fe8f3a92-home\") pod \"sub-test-model-8280eb-kserve-7d555d5589-hkn2n\" (UID: \"9df66431-f3dc-4457-9dfe-2de4fe8f3a92\") " pod="ai-tenant-e2e-shared-a-w3-8280eb/sub-test-model-8280eb-kserve-7d555d5589-hkn2n"
Sep 11 19:14:46.148774 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:46.148699    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9df66431-f3dc-4457-9dfe-2de4fe8f3a92-tmp-dir\") pod \"sub-test-model-8280eb-kserve-7d555d5589-hkn2n\" (UID: \"9df66431-f3dc-4457-9dfe-2de4fe8f3a92\") " pod="ai-tenant-e2e-shared-a-w3-8280eb/sub-test-model-8280eb-kserve-7d555d5589-hkn2n"
Sep 11 19:14:46.148774 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:46.148719    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9df66431-f3dc-4457-9dfe-2de4fe8f3a92-tls-certs\") pod \"sub-test-model-8280eb-kserve-7d555d5589-hkn2n\" (UID: \"9df66431-f3dc-4457-9dfe-2de4fe8f3a92\") " pod="ai-tenant-e2e-shared-a-w3-8280eb/sub-test-model-8280eb-kserve-7d555d5589-hkn2n"
Sep 11 19:14:46.149067 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:46.149045    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9df66431-f3dc-4457-9dfe-2de4fe8f3a92-model-cache\") pod \"sub-test-model-8280eb-kserve-7d555d5589-hkn2n\" (UID: \"9df66431-f3dc-4457-9dfe-2de4fe8f3a92\") " pod="ai-tenant-e2e-shared-a-w3-8280eb/sub-test-model-8280eb-kserve-7d555d5589-hkn2n"
Sep 11 19:14:46.149126 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:46.149064    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9df66431-f3dc-4457-9dfe-2de4fe8f3a92-tmp-dir\") pod \"sub-test-model-8280eb-kserve-7d555d5589-hkn2n\" (UID: \"9df66431-f3dc-4457-9dfe-2de4fe8f3a92\") " pod="ai-tenant-e2e-shared-a-w3-8280eb/sub-test-model-8280eb-kserve-7d555d5589-hkn2n"
Sep 11 19:14:46.149184 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:46.149136    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9df66431-f3dc-4457-9dfe-2de4fe8f3a92-home\") pod \"sub-test-model-8280eb-kserve-7d555d5589-hkn2n\" (UID: \"9df66431-f3dc-4457-9dfe-2de4fe8f3a92\") " pod="ai-tenant-e2e-shared-a-w3-8280eb/sub-test-model-8280eb-kserve-7d555d5589-hkn2n"
Sep 11 19:14:46.151077 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:46.151055    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9df66431-f3dc-4457-9dfe-2de4fe8f3a92-dshm\") pod \"sub-test-model-8280eb-kserve-7d555d5589-hkn2n\" (UID: \"9df66431-f3dc-4457-9dfe-2de4fe8f3a92\") " pod="ai-tenant-e2e-shared-a-w3-8280eb/sub-test-model-8280eb-kserve-7d555d5589-hkn2n"
Sep 11 19:14:46.151319 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:46.151303    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9df66431-f3dc-4457-9dfe-2de4fe8f3a92-tls-certs\") pod \"sub-test-model-8280eb-kserve-7d555d5589-hkn2n\" (UID: \"9df66431-f3dc-4457-9dfe-2de4fe8f3a92\") " pod="ai-tenant-e2e-shared-a-w3-8280eb/sub-test-model-8280eb-kserve-7d555d5589-hkn2n"
Sep 11 19:14:46.155853 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:46.155828    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6r2p\" (UniqueName: \"kubernetes.io/projected/9df66431-f3dc-4457-9dfe-2de4fe8f3a92-kube-api-access-q6r2p\") pod \"sub-test-model-8280eb-kserve-7d555d5589-hkn2n\" (UID: \"9df66431-f3dc-4457-9dfe-2de4fe8f3a92\") " pod="ai-tenant-e2e-shared-a-w3-8280eb/sub-test-model-8280eb-kserve-7d555d5589-hkn2n"
Sep 11 19:14:46.234220 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:46.234174    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-8280eb/sub-test-model-8280eb-kserve-7d555d5589-hkn2n"
Sep 11 19:14:46.333010 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:46.332969    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-37ce96-2xmd4"]
Sep 11 19:14:46.337264 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:46.337246    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-37ce96-2xmd4"
Sep 11 19:14:46.339482 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:46.339448    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 19:14:46.339630 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:46.339483    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ln295\""
Sep 11 19:14:46.343066 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:46.343036    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-37ce96-2xmd4"]
Sep 11 19:14:46.452320 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:46.452277    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1193c80-969a-4fe6-b31a-f0d1f53e2a5f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-37ce96-2xmd4\" (UID: \"c1193c80-969a-4fe6-b31a-f0d1f53e2a5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-37ce96-2xmd4"
Sep 11 19:14:46.506813 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:46.506750    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-8280eb/sub-test-model-8280eb-kserve-7d555d5589-hkn2n"]
Sep 11 19:14:46.509309 ip-10-0-134-189 kubenswrapper[2578]: W0911 19:14:46.509282    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9df66431_f3dc_4457_9dfe_2de4fe8f3a92.slice/crio-07c00e9125541d5a934a97780a81cdd1e78a3ede7c6d5facf70dc5db672e7e22 WatchSource:0}: Error finding container 07c00e9125541d5a934a97780a81cdd1e78a3ede7c6d5facf70dc5db672e7e22: Status 404 returned error can't find the container with id 07c00e9125541d5a934a97780a81cdd1e78a3ede7c6d5facf70dc5db672e7e22
Sep 11 19:14:46.552940 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:46.552906    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1193c80-969a-4fe6-b31a-f0d1f53e2a5f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-37ce96-2xmd4\" (UID: \"c1193c80-969a-4fe6-b31a-f0d1f53e2a5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-37ce96-2xmd4"
Sep 11 19:14:46.553659 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:46.553637    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1193c80-969a-4fe6-b31a-f0d1f53e2a5f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-37ce96-2xmd4\" (UID: \"c1193c80-969a-4fe6-b31a-f0d1f53e2a5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-37ce96-2xmd4"
Sep 11 19:14:46.651614 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:46.651565    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-37ce96-2xmd4"
Sep 11 19:14:46.922744 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:46.922722    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-37ce96-2xmd4"]
Sep 11 19:14:46.924744 ip-10-0-134-189 kubenswrapper[2578]: W0911 19:14:46.924715    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1193c80_969a_4fe6_b31a_f0d1f53e2a5f.slice/crio-6667656ab87b595cba9390aeec3aba42a4b02a074c80650955f72370e0c7592f WatchSource:0}: Error finding container 6667656ab87b595cba9390aeec3aba42a4b02a074c80650955f72370e0c7592f: Status 404 returned error can't find the container with id 6667656ab87b595cba9390aeec3aba42a4b02a074c80650955f72370e0c7592f
Sep 11 19:14:46.988317 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:46.988267    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-37ce96-2xmd4" event={"ID":"c1193c80-969a-4fe6-b31a-f0d1f53e2a5f","Type":"ContainerStarted","Data":"6667656ab87b595cba9390aeec3aba42a4b02a074c80650955f72370e0c7592f"}
Sep 11 19:14:46.989825 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:46.989801    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-8280eb/sub-test-model-8280eb-kserve-7d555d5589-hkn2n" event={"ID":"9df66431-f3dc-4457-9dfe-2de4fe8f3a92","Type":"ContainerStarted","Data":"baf9217f3a54535243990dbc46eacc516a80decb9e5fa34bce6b572e4d8f26bf"}
Sep 11 19:14:46.989977 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:46.989836    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-8280eb/sub-test-model-8280eb-kserve-7d555d5589-hkn2n" event={"ID":"9df66431-f3dc-4457-9dfe-2de4fe8f3a92","Type":"ContainerStarted","Data":"07c00e9125541d5a934a97780a81cdd1e78a3ede7c6d5facf70dc5db672e7e22"}
Sep 11 19:14:47.005844 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:47.005794    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-8280eb/sub-test-model-8280eb-kserve-7d555d5589-hkn2n" podStartSLOduration=2.005777717 podStartE2EDuration="2.005777717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:14:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:14:47.004161741 +0000 UTC m=+1563.081752307" watchObservedRunningTime="2026-09-11 19:14:47.005777717 +0000 UTC m=+1563.083368318"
Sep 11 19:14:47.996910 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:47.996876    2578 generic.go:356] "Generic (PLEG): container finished" podID="c1193c80-969a-4fe6-b31a-f0d1f53e2a5f" containerID="fb5532e7b52fbc4a19fd2687359aaf32adea7e65f5d2d7d574c49cfe2b8e06b7" exitCode=0
Sep 11 19:14:47.997300 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:47.996946    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-37ce96-2xmd4" event={"ID":"c1193c80-969a-4fe6-b31a-f0d1f53e2a5f","Type":"ContainerDied","Data":"fb5532e7b52fbc4a19fd2687359aaf32adea7e65f5d2d7d574c49cfe2b8e06b7"}
Sep 11 19:14:49.138125 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:49.138097    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-37ce96-2xmd4"
Sep 11 19:14:49.280503 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:49.280413    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1193c80-969a-4fe6-b31a-f0d1f53e2a5f-maas-api-service-ca\") pod \"c1193c80-969a-4fe6-b31a-f0d1f53e2a5f\" (UID: \"c1193c80-969a-4fe6-b31a-f0d1f53e2a5f\") "
Sep 11 19:14:49.280855 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:49.280830    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1193c80-969a-4fe6-b31a-f0d1f53e2a5f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c1193c80-969a-4fe6-b31a-f0d1f53e2a5f" (UID: "c1193c80-969a-4fe6-b31a-f0d1f53e2a5f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 19:14:49.340173 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:49.340123    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-3b7391/auth-test-model-3b7391-kserve-74db88fc4c-smtts" podUID="3d0be388-987b-417e-85b1-0eaf6bb428b3" containerName="main" containerID="cri-o://415397f1a52cbc3d00c0a924afb5eec0f1095d6fb9c6739ab5d54cd54bdee625" gracePeriod=45
Sep 11 19:14:49.382028 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:49.381992    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1193c80-969a-4fe6-b31a-f0d1f53e2a5f-maas-api-service-ca\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:14:49.589578 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:49.589550    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3b7391/auth-test-model-3b7391-kserve-74db88fc4c-smtts"
Sep 11 19:14:49.684952 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:49.684913    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kzwsq\" (UniqueName: \"kubernetes.io/projected/3d0be388-987b-417e-85b1-0eaf6bb428b3-kube-api-access-kzwsq\") pod \"3d0be388-987b-417e-85b1-0eaf6bb428b3\" (UID: \"3d0be388-987b-417e-85b1-0eaf6bb428b3\") "
Sep 11 19:14:49.684952 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:49.684955    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d0be388-987b-417e-85b1-0eaf6bb428b3-model-cache\") pod \"3d0be388-987b-417e-85b1-0eaf6bb428b3\" (UID: \"3d0be388-987b-417e-85b1-0eaf6bb428b3\") "
Sep 11 19:14:49.685226 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:49.685001    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d0be388-987b-417e-85b1-0eaf6bb428b3-tmp-dir\") pod \"3d0be388-987b-417e-85b1-0eaf6bb428b3\" (UID: \"3d0be388-987b-417e-85b1-0eaf6bb428b3\") "
Sep 11 19:14:49.685226 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:49.685032    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d0be388-987b-417e-85b1-0eaf6bb428b3-home\") pod \"3d0be388-987b-417e-85b1-0eaf6bb428b3\" (UID: \"3d0be388-987b-417e-85b1-0eaf6bb428b3\") "
Sep 11 19:14:49.685226 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:49.685065    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d0be388-987b-417e-85b1-0eaf6bb428b3-tls-certs\") pod \"3d0be388-987b-417e-85b1-0eaf6bb428b3\" (UID: \"3d0be388-987b-417e-85b1-0eaf6bb428b3\") "
Sep 11 19:14:49.685226 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:49.685121    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d0be388-987b-417e-85b1-0eaf6bb428b3-dshm\") pod \"3d0be388-987b-417e-85b1-0eaf6bb428b3\" (UID: \"3d0be388-987b-417e-85b1-0eaf6bb428b3\") "
Sep 11 19:14:49.685432 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:49.685289    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d0be388-987b-417e-85b1-0eaf6bb428b3-model-cache" (OuterVolumeSpecName: "model-cache") pod "3d0be388-987b-417e-85b1-0eaf6bb428b3" (UID: "3d0be388-987b-417e-85b1-0eaf6bb428b3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:14:49.686153 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:49.685547    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d0be388-987b-417e-85b1-0eaf6bb428b3-model-cache\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:14:49.686153 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:49.685566    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d0be388-987b-417e-85b1-0eaf6bb428b3-home" (OuterVolumeSpecName: "home") pod "3d0be388-987b-417e-85b1-0eaf6bb428b3" (UID: "3d0be388-987b-417e-85b1-0eaf6bb428b3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:14:49.686153 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:49.685578    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d0be388-987b-417e-85b1-0eaf6bb428b3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3d0be388-987b-417e-85b1-0eaf6bb428b3" (UID: "3d0be388-987b-417e-85b1-0eaf6bb428b3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:14:49.687677 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:49.687652    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d0be388-987b-417e-85b1-0eaf6bb428b3-dshm" (OuterVolumeSpecName: "dshm") pod "3d0be388-987b-417e-85b1-0eaf6bb428b3" (UID: "3d0be388-987b-417e-85b1-0eaf6bb428b3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:14:49.687764 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:49.687693    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d0be388-987b-417e-85b1-0eaf6bb428b3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3d0be388-987b-417e-85b1-0eaf6bb428b3" (UID: "3d0be388-987b-417e-85b1-0eaf6bb428b3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 19:14:49.687764 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:49.687698    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d0be388-987b-417e-85b1-0eaf6bb428b3-kube-api-access-kzwsq" (OuterVolumeSpecName: "kube-api-access-kzwsq") pod "3d0be388-987b-417e-85b1-0eaf6bb428b3" (UID: "3d0be388-987b-417e-85b1-0eaf6bb428b3"). InnerVolumeSpecName "kube-api-access-kzwsq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:14:49.786548 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:49.786512    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzwsq\" (UniqueName: \"kubernetes.io/projected/3d0be388-987b-417e-85b1-0eaf6bb428b3-kube-api-access-kzwsq\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:14:49.786548 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:49.786540    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d0be388-987b-417e-85b1-0eaf6bb428b3-tmp-dir\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:14:49.786548 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:49.786551    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d0be388-987b-417e-85b1-0eaf6bb428b3-home\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:14:49.786548 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:49.786559    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d0be388-987b-417e-85b1-0eaf6bb428b3-tls-certs\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:14:49.786821 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:49.786567    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d0be388-987b-417e-85b1-0eaf6bb428b3-dshm\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:14:50.009030 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:50.008998    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-37ce96-2xmd4"
Sep 11 19:14:50.009030 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:50.009013    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-37ce96-2xmd4" event={"ID":"c1193c80-969a-4fe6-b31a-f0d1f53e2a5f","Type":"ContainerDied","Data":"6667656ab87b595cba9390aeec3aba42a4b02a074c80650955f72370e0c7592f"}
Sep 11 19:14:50.009293 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:50.009063    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6667656ab87b595cba9390aeec3aba42a4b02a074c80650955f72370e0c7592f"
Sep 11 19:14:50.010540 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:50.010516    2578 generic.go:356] "Generic (PLEG): container finished" podID="3d0be388-987b-417e-85b1-0eaf6bb428b3" containerID="415397f1a52cbc3d00c0a924afb5eec0f1095d6fb9c6739ab5d54cd54bdee625" exitCode=0
Sep 11 19:14:50.010699 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:50.010566    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3b7391/auth-test-model-3b7391-kserve-74db88fc4c-smtts"
Sep 11 19:14:50.010699 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:50.010585    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3b7391/auth-test-model-3b7391-kserve-74db88fc4c-smtts" event={"ID":"3d0be388-987b-417e-85b1-0eaf6bb428b3","Type":"ContainerDied","Data":"415397f1a52cbc3d00c0a924afb5eec0f1095d6fb9c6739ab5d54cd54bdee625"}
Sep 11 19:14:50.010699 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:50.010632    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3b7391/auth-test-model-3b7391-kserve-74db88fc4c-smtts" event={"ID":"3d0be388-987b-417e-85b1-0eaf6bb428b3","Type":"ContainerDied","Data":"dff0805a7e90e7f0f34de73d7d46b59d49c257e0700654ae36f8b6f370751849"}
Sep 11 19:14:50.010699 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:50.010650    2578 scope.go:117] "RemoveContainer" containerID="415397f1a52cbc3d00c0a924afb5eec0f1095d6fb9c6739ab5d54cd54bdee625"
Sep 11 19:14:50.019743 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:50.019644    2578 scope.go:117] "RemoveContainer" containerID="415397f1a52cbc3d00c0a924afb5eec0f1095d6fb9c6739ab5d54cd54bdee625"
Sep 11 19:14:50.020075 ip-10-0-134-189 kubenswrapper[2578]: E0911 19:14:50.020052    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"415397f1a52cbc3d00c0a924afb5eec0f1095d6fb9c6739ab5d54cd54bdee625\": container with ID starting with 415397f1a52cbc3d00c0a924afb5eec0f1095d6fb9c6739ab5d54cd54bdee625 not found: ID does not exist" containerID="415397f1a52cbc3d00c0a924afb5eec0f1095d6fb9c6739ab5d54cd54bdee625"
Sep 11 19:14:50.020149 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:50.020084    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"415397f1a52cbc3d00c0a924afb5eec0f1095d6fb9c6739ab5d54cd54bdee625"} err="failed to get container status \"415397f1a52cbc3d00c0a924afb5eec0f1095d6fb9c6739ab5d54cd54bdee625\": rpc error: code = NotFound desc = could not find container \"415397f1a52cbc3d00c0a924afb5eec0f1095d6fb9c6739ab5d54cd54bdee625\": container with ID starting with 415397f1a52cbc3d00c0a924afb5eec0f1095d6fb9c6739ab5d54cd54bdee625 not found: ID does not exist"
Sep 11 19:14:50.041889 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:50.037347    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3b7391/auth-test-model-3b7391-kserve-74db88fc4c-smtts"]
Sep 11 19:14:50.045068 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:50.045043    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3b7391/auth-test-model-3b7391-kserve-74db88fc4c-smtts"]
Sep 11 19:14:50.396663 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:50.396549    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d0be388-987b-417e-85b1-0eaf6bb428b3" path="/var/lib/kubelet/pods/3d0be388-987b-417e-85b1-0eaf6bb428b3/volumes"
Sep 11 19:14:56.234702 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:56.234653    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-8280eb/sub-test-model-8280eb-kserve-7d555d5589-hkn2n"
Sep 11 19:14:56.235113 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:56.234717    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-8280eb/sub-test-model-8280eb-kserve-7d555d5589-hkn2n"
Sep 11 19:14:56.247267 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:56.247239    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-8280eb/sub-test-model-8280eb-kserve-7d555d5589-hkn2n"
Sep 11 19:14:56.914661 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:56.914627    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-37ce96-85fb758d8f-wqfhk"]
Sep 11 19:14:56.914954 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:56.914905    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-37ce96-85fb758d8f-wqfhk" podUID="fb79a5c3-ada2-4aa7-bd25-fc07436a10a6" containerName="payload-processing" containerID="cri-o://c1c0a49d9db6eae59342d1595986bd82c29c377a095b0abb18e0214e6022aba8" gracePeriod=30
Sep 11 19:14:57.049157 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:57.049125    2578 generic.go:356] "Generic (PLEG): container finished" podID="fb79a5c3-ada2-4aa7-bd25-fc07436a10a6" containerID="c1c0a49d9db6eae59342d1595986bd82c29c377a095b0abb18e0214e6022aba8" exitCode=0
Sep 11 19:14:57.049343 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:57.049199    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-37ce96-85fb758d8f-wqfhk" event={"ID":"fb79a5c3-ada2-4aa7-bd25-fc07436a10a6","Type":"ContainerDied","Data":"c1c0a49d9db6eae59342d1595986bd82c29c377a095b0abb18e0214e6022aba8"}
Sep 11 19:14:57.061874 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:57.061840    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-8280eb/sub-test-model-8280eb-kserve-7d555d5589-hkn2n"
Sep 11 19:14:57.097313 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:57.097281    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-37ce96-2xmd4"]
Sep 11 19:14:57.103306 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:57.103273    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-37ce96-2xmd4"]
Sep 11 19:14:57.183943 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:57.183909    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-37ce96-85fb758d8f-wqfhk"
Sep 11 19:14:57.257222 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:57.257176    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tmj6f\" (UniqueName: \"kubernetes.io/projected/fb79a5c3-ada2-4aa7-bd25-fc07436a10a6-kube-api-access-tmj6f\") pod \"fb79a5c3-ada2-4aa7-bd25-fc07436a10a6\" (UID: \"fb79a5c3-ada2-4aa7-bd25-fc07436a10a6\") "
Sep 11 19:14:57.257724 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:57.257234    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb79a5c3-ada2-4aa7-bd25-fc07436a10a6-plugins-config-volume\") pod \"fb79a5c3-ada2-4aa7-bd25-fc07436a10a6\" (UID: \"fb79a5c3-ada2-4aa7-bd25-fc07436a10a6\") "
Sep 11 19:14:57.257724 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:57.257649    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb79a5c3-ada2-4aa7-bd25-fc07436a10a6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fb79a5c3-ada2-4aa7-bd25-fc07436a10a6" (UID: "fb79a5c3-ada2-4aa7-bd25-fc07436a10a6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 19:14:57.259514 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:57.259494    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb79a5c3-ada2-4aa7-bd25-fc07436a10a6-kube-api-access-tmj6f" (OuterVolumeSpecName: "kube-api-access-tmj6f") pod "fb79a5c3-ada2-4aa7-bd25-fc07436a10a6" (UID: "fb79a5c3-ada2-4aa7-bd25-fc07436a10a6"). InnerVolumeSpecName "kube-api-access-tmj6f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:14:57.358522 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:57.358485    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmj6f\" (UniqueName: \"kubernetes.io/projected/fb79a5c3-ada2-4aa7-bd25-fc07436a10a6-kube-api-access-tmj6f\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:14:57.358522 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:57.358513    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb79a5c3-ada2-4aa7-bd25-fc07436a10a6-plugins-config-volume\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:14:58.055777 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:58.055747    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-37ce96-85fb758d8f-wqfhk"
Sep 11 19:14:58.055995 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:58.055796    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-37ce96-85fb758d8f-wqfhk" event={"ID":"fb79a5c3-ada2-4aa7-bd25-fc07436a10a6","Type":"ContainerDied","Data":"468b48fb47f4de882937f0195a10604f10737712abbdd954657f4b565c282850"}
Sep 11 19:14:58.055995 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:58.055864    2578 scope.go:117] "RemoveContainer" containerID="c1c0a49d9db6eae59342d1595986bd82c29c377a095b0abb18e0214e6022aba8"
Sep 11 19:14:58.076858 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:58.076830    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-37ce96-85fb758d8f-wqfhk"]
Sep 11 19:14:58.080702 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:58.080679    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-37ce96-85fb758d8f-wqfhk"]
Sep 11 19:14:58.396264 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:58.396185    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1193c80-969a-4fe6-b31a-f0d1f53e2a5f" path="/var/lib/kubelet/pods/c1193c80-969a-4fe6-b31a-f0d1f53e2a5f/volumes"
Sep 11 19:14:58.396653 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:14:58.396529    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb79a5c3-ada2-4aa7-bd25-fc07436a10a6" path="/var/lib/kubelet/pods/fb79a5c3-ada2-4aa7-bd25-fc07436a10a6/volumes"
Sep 11 19:15:00.131391 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:15:00.131349    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819235-mdb4s"]
Sep 11 19:15:00.131905 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:15:00.131701    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d0be388-987b-417e-85b1-0eaf6bb428b3" containerName="main"
Sep 11 19:15:00.131905 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:15:00.131712    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d0be388-987b-417e-85b1-0eaf6bb428b3" containerName="main"
Sep 11 19:15:00.131905 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:15:00.131726    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb79a5c3-ada2-4aa7-bd25-fc07436a10a6" containerName="payload-processing"
Sep 11 19:15:00.131905 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:15:00.131732    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb79a5c3-ada2-4aa7-bd25-fc07436a10a6" containerName="payload-processing"
Sep 11 19:15:00.131905 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:15:00.131739    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1193c80-969a-4fe6-b31a-f0d1f53e2a5f" containerName="revoke-keys"
Sep 11 19:15:00.131905 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:15:00.131744    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1193c80-969a-4fe6-b31a-f0d1f53e2a5f" containerName="revoke-keys"
Sep 11 19:15:00.131905 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:15:00.131802    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb79a5c3-ada2-4aa7-bd25-fc07436a10a6" containerName="payload-processing"
Sep 11 19:15:00.131905 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:15:00.131810    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1193c80-969a-4fe6-b31a-f0d1f53e2a5f" containerName="revoke-keys"
Sep 11 19:15:00.131905 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:15:00.131816    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d0be388-987b-417e-85b1-0eaf6bb428b3" containerName="main"
Sep 11 19:15:00.135709 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:15:00.135690    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3b7391-29819235-4zg72"]
Sep 11 19:15:00.135833 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:15:00.135811    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819235-mdb4s"
Sep 11 19:15:00.137878 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:15:00.137853    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ln295\""
Sep 11 19:15:00.138795 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:15:00.138773    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3b7391-29819235-4zg72"
Sep 11 19:15:00.142852 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:15:00.142824    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819235-mdb4s"]
Sep 11 19:15:00.145168 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:15:00.145150    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3b7391-29819235-4zg72"]
Sep 11 19:15:00.148231 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:15:00.148206    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819235-mdb4s"
Sep 11 19:15:00.156452 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:15:00.156432    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3b7391-29819235-4zg72"
Sep 11 19:15:00.445457 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:15:00.445430    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3b7391-29819235-4zg72"]
Sep 11 19:15:00.448518 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:15:00.448360    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819235-mdb4s"]
Sep 11 19:15:00.448741 ip-10-0-134-189 kubenswrapper[2578]: W0911 19:15:00.448717    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd11c2973_70ae_461a_bfc4_08320c25c06d.slice/crio-f8010c409af217100f4f031246a0b53bef233a7672cf307931c211704eefd7f3 WatchSource:0}: Error finding container f8010c409af217100f4f031246a0b53bef233a7672cf307931c211704eefd7f3: Status 404 returned error can't find the container with id f8010c409af217100f4f031246a0b53bef233a7672cf307931c211704eefd7f3
Sep 11 19:15:00.450562 ip-10-0-134-189 kubenswrapper[2578]: W0911 19:15:00.450540    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e42c052_f443_4c3e_a254_aa6e376aa3aa.slice/crio-9fdd37fa5cbeea4307fc0ac41acb5eb062c1b18a7b1000c6e7dcda804f6b5674 WatchSource:0}: Error finding container 9fdd37fa5cbeea4307fc0ac41acb5eb062c1b18a7b1000c6e7dcda804f6b5674: Status 404 returned error can't find the container with id 9fdd37fa5cbeea4307fc0ac41acb5eb062c1b18a7b1000c6e7dcda804f6b5674
Sep 11 19:15:01.073568 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:15:01.073461    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3b7391-29819235-4zg72" event={"ID":"d11c2973-70ae-461a-bfc4-08320c25c06d","Type":"ContainerStarted","Data":"fdd8a894bc89693e8cf2119e9e5c0e69fc915017243c48f7ab2f69658b36e170"}
Sep 11 19:15:01.073568 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:15:01.073527    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3b7391-29819235-4zg72" event={"ID":"d11c2973-70ae-461a-bfc4-08320c25c06d","Type":"ContainerStarted","Data":"f8010c409af217100f4f031246a0b53bef233a7672cf307931c211704eefd7f3"}
Sep 11 19:15:01.075150 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:15:01.075122    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819235-mdb4s" event={"ID":"6e42c052-f443-4c3e-a254-aa6e376aa3aa","Type":"ContainerStarted","Data":"fb7036e762f187a5529ed1c4d3bf1d44e3e898b5eb62c2398b96dbb14a0c6d5e"}
Sep 11 19:15:01.075300 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:15:01.075166    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819235-mdb4s" event={"ID":"6e42c052-f443-4c3e-a254-aa6e376aa3aa","Type":"ContainerStarted","Data":"9fdd37fa5cbeea4307fc0ac41acb5eb062c1b18a7b1000c6e7dcda804f6b5674"}
Sep 11 19:15:01.087252 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:15:01.087201    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3b7391-29819235-4zg72" podStartSLOduration=1.087183765 podStartE2EDuration="1.087183765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:15:01.085885394 +0000 UTC m=+1577.163475964" watchObservedRunningTime="2026-09-11 19:15:01.087183765 +0000 UTC m=+1577.164774319"
Sep 11 19:15:01.099454 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:15:01.099406    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819235-mdb4s" podStartSLOduration=1.099392942 podStartE2EDuration="1.099392942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:15:01.098564485 +0000 UTC m=+1577.176155063" watchObservedRunningTime="2026-09-11 19:15:01.099392942 +0000 UTC m=+1577.176983496"
Sep 11 19:15:02.081690 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:15:02.081657    2578 generic.go:356] "Generic (PLEG): container finished" podID="d11c2973-70ae-461a-bfc4-08320c25c06d" containerID="fdd8a894bc89693e8cf2119e9e5c0e69fc915017243c48f7ab2f69658b36e170" exitCode=0
Sep 11 19:15:02.082141 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:15:02.081753    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3b7391-29819235-4zg72" event={"ID":"d11c2973-70ae-461a-bfc4-08320c25c06d","Type":"ContainerDied","Data":"fdd8a894bc89693e8cf2119e9e5c0e69fc915017243c48f7ab2f69658b36e170"}
Sep 11 19:15:02.083305 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:15:02.083279    2578 generic.go:356] "Generic (PLEG): container finished" podID="6e42c052-f443-4c3e-a254-aa6e376aa3aa" containerID="fb7036e762f187a5529ed1c4d3bf1d44e3e898b5eb62c2398b96dbb14a0c6d5e" exitCode=0
Sep 11 19:15:02.083434 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:15:02.083356    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819235-mdb4s" event={"ID":"6e42c052-f443-4c3e-a254-aa6e376aa3aa","Type":"ContainerDied","Data":"fb7036e762f187a5529ed1c4d3bf1d44e3e898b5eb62c2398b96dbb14a0c6d5e"}
Sep 11 19:15:03.228352 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:15:03.228322    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3b7391-29819235-4zg72"
Sep 11 19:15:03.231215 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:15:03.231195    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819235-mdb4s"
Sep 11 19:15:04.094416 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:15:04.094382    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3b7391-29819235-4zg72"
Sep 11 19:15:04.094665 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:15:04.094422    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3b7391-29819235-4zg72" event={"ID":"d11c2973-70ae-461a-bfc4-08320c25c06d","Type":"ContainerDied","Data":"f8010c409af217100f4f031246a0b53bef233a7672cf307931c211704eefd7f3"}
Sep 11 19:15:04.094665 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:15:04.094479    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f8010c409af217100f4f031246a0b53bef233a7672cf307931c211704eefd7f3"
Sep 11 19:15:04.096025 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:15:04.095988    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819235-mdb4s"
Sep 11 19:15:04.096142 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:15:04.095988    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819235-mdb4s" event={"ID":"6e42c052-f443-4c3e-a254-aa6e376aa3aa","Type":"ContainerDied","Data":"9fdd37fa5cbeea4307fc0ac41acb5eb062c1b18a7b1000c6e7dcda804f6b5674"}
Sep 11 19:15:04.096142 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:15:04.096120    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9fdd37fa5cbeea4307fc0ac41acb5eb062c1b18a7b1000c6e7dcda804f6b5674"
Sep 11 19:15:15.034062 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:15:15.034027    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-3b7391/sub-test-model-3b7391-kserve-7c7fcb8949-jm5r2"]
Sep 11 19:15:15.034504 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:15:15.034487    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e42c052-f443-4c3e-a254-aa6e376aa3aa" containerName="cleanup"
Sep 11 19:15:15.034557 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:15:15.034508    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e42c052-f443-4c3e-a254-aa6e376aa3aa" containerName="cleanup"
Sep 11 19:15:15.034557 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:15:15.034548    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d11c2973-70ae-461a-bfc4-08320c25c06d" containerName="cleanup"
Sep 11 19:15:15.034557 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:15:15.034555    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d11c2973-70ae-461a-bfc4-08320c25c06d" containerName="cleanup"
Sep 11 19:15:15.034711 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:15:15.034664    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d11c2973-70ae-461a-bfc4-08320c25c06d" containerName="cleanup"
Sep 11 19:15:15.034711 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:15:15.034682    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e42c052-f443-4c3e-a254-aa6e376aa3aa" containerName="cleanup"
Sep 11 19:15:15.037396 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:15:15.037378    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3b7391/sub-test-model-3b7391-kserve-7c7fcb8949-jm5r2"
Sep 11 19:15:15.039985 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:15:15.039954    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-3b7391\"/\"sub-test-model-3b7391-kserve-self-signed-certs\""
Sep 11 19:15:15.040127 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:15:15.040038    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-3b7391\"/\"default-dockercfg-296hb\""
Sep 11 19:15:15.040127 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:15:15.039976    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-3b7391\"/\"kube-root-ca.crt\""
Sep 11 19:15:15.040250 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:15:15.040042    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-3b7391\"/\"openshift-service-ca.crt\""
Sep 11 19:15:15.045186 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:15:15.045160    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3b7391/sub-test-model-3b7391-kserve-7c7fcb8949-jm5r2"]
Sep 11 19:15:15.124420 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:15:15.124380    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9aa9e368-4ae5-495f-a5ec-d2a9bbcc283b-dshm\") pod \"sub-test-model-3b7391-kserve-7c7fcb8949-jm5r2\" (UID: \"9aa9e368-4ae5-495f-a5ec-d2a9bbcc283b\") " pod="ai-tenant-e2e-shared-b-w3-3b7391/sub-test-model-3b7391-kserve-7c7fcb8949-jm5r2"
Sep 11 19:15:15.124644 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:15:15.124429    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9aa9e368-4ae5-495f-a5ec-d2a9bbcc283b-tls-certs\") pod \"sub-test-model-3b7391-kserve-7c7fcb8949-jm5r2\" (UID: \"9aa9e368-4ae5-495f-a5ec-d2a9bbcc283b\") " pod="ai-tenant-e2e-shared-b-w3-3b7391/sub-test-model-3b7391-kserve-7c7fcb8949-jm5r2"
Sep 11 19:15:15.124644 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:15:15.124448    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9aa9e368-4ae5-495f-a5ec-d2a9bbcc283b-home\") pod \"sub-test-model-3b7391-kserve-7c7fcb8949-jm5r2\" (UID: \"9aa9e368-4ae5-495f-a5ec-d2a9bbcc283b\") " pod="ai-tenant-e2e-shared-b-w3-3b7391/sub-test-model-3b7391-kserve-7c7fcb8949-jm5r2"
Sep 11 19:15:15.124644 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:15:15.124463    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9aa9e368-4ae5-495f-a5ec-d2a9bbcc283b-model-cache\") pod \"sub-test-model-3b7391-kserve-7c7fcb8949-jm5r2\" (UID: \"9aa9e368-4ae5-495f-a5ec-d2a9bbcc283b\") " pod="ai-tenant-e2e-shared-b-w3-3b7391/sub-test-model-3b7391-kserve-7c7fcb8949-jm5r2"
Sep 11 19:15:15.124644 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:15:15.124485    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9aa9e368-4ae5-495f-a5ec-d2a9bbcc283b-tmp-dir\") pod \"sub-test-model-3b7391-kserve-7c7fcb8949-jm5r2\" (UID: \"9aa9e368-4ae5-495f-a5ec-d2a9bbcc283b\") " pod="ai-tenant-e2e-shared-b-w3-3b7391/sub-test-model-3b7391-kserve-7c7fcb8949-jm5r2"
Sep 11 19:15:15.124644 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:15:15.124565    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxtpz\" (UniqueName: \"kubernetes.io/projected/9aa9e368-4ae5-495f-a5ec-d2a9bbcc283b-kube-api-access-bxtpz\") pod \"sub-test-model-3b7391-kserve-7c7fcb8949-jm5r2\" (UID: \"9aa9e368-4ae5-495f-a5ec-d2a9bbcc283b\") " pod="ai-tenant-e2e-shared-b-w3-3b7391/sub-test-model-3b7391-kserve-7c7fcb8949-jm5r2"
Sep 11 19:15:15.225839 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:15:15.225801    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9aa9e368-4ae5-495f-a5ec-d2a9bbcc283b-dshm\") pod \"sub-test-model-3b7391-kserve-7c7fcb8949-jm5r2\" (UID: \"9aa9e368-4ae5-495f-a5ec-d2a9bbcc283b\") " pod="ai-tenant-e2e-shared-b-w3-3b7391/sub-test-model-3b7391-kserve-7c7fcb8949-jm5r2"
Sep 11 19:15:15.225839 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:15:15.225855    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9aa9e368-4ae5-495f-a5ec-d2a9bbcc283b-tls-certs\") pod \"sub-test-model-3b7391-kserve-7c7fcb8949-jm5r2\" (UID: \"9aa9e368-4ae5-495f-a5ec-d2a9bbcc283b\") " pod="ai-tenant-e2e-shared-b-w3-3b7391/sub-test-model-3b7391-kserve-7c7fcb8949-jm5r2"
Sep 11 19:15:15.226063 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:15:15.225872    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9aa9e368-4ae5-495f-a5ec-d2a9bbcc283b-home\") pod \"sub-test-model-3b7391-kserve-7c7fcb8949-jm5r2\" (UID: \"9aa9e368-4ae5-495f-a5ec-d2a9bbcc283b\") " pod="ai-tenant-e2e-shared-b-w3-3b7391/sub-test-model-3b7391-kserve-7c7fcb8949-jm5r2"
Sep 11 19:15:15.226063 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:15:15.225886    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9aa9e368-4ae5-495f-a5ec-d2a9bbcc283b-model-cache\") pod \"sub-test-model-3b7391-kserve-7c7fcb8949-jm5r2\" (UID: \"9aa9e368-4ae5-495f-a5ec-d2a9bbcc283b\") " pod="ai-tenant-e2e-shared-b-w3-3b7391/sub-test-model-3b7391-kserve-7c7fcb8949-jm5r2"
Sep 11 19:15:15.226063 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:15:15.225910    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9aa9e368-4ae5-495f-a5ec-d2a9bbcc283b-tmp-dir\") pod \"sub-test-model-3b7391-kserve-7c7fcb8949-jm5r2\" (UID: \"9aa9e368-4ae5-495f-a5ec-d2a9bbcc283b\") " pod="ai-tenant-e2e-shared-b-w3-3b7391/sub-test-model-3b7391-kserve-7c7fcb8949-jm5r2"
Sep 11 19:15:15.226063 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:15:15.225946    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxtpz\" (UniqueName: \"kubernetes.io/projected/9aa9e368-4ae5-495f-a5ec-d2a9bbcc283b-kube-api-access-bxtpz\") pod \"sub-test-model-3b7391-kserve-7c7fcb8949-jm5r2\" (UID: \"9aa9e368-4ae5-495f-a5ec-d2a9bbcc283b\") " pod="ai-tenant-e2e-shared-b-w3-3b7391/sub-test-model-3b7391-kserve-7c7fcb8949-jm5r2"
Sep 11 19:15:15.226297 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:15:15.226277    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9aa9e368-4ae5-495f-a5ec-d2a9bbcc283b-home\") pod \"sub-test-model-3b7391-kserve-7c7fcb8949-jm5r2\" (UID: \"9aa9e368-4ae5-495f-a5ec-d2a9bbcc283b\") " pod="ai-tenant-e2e-shared-b-w3-3b7391/sub-test-model-3b7391-kserve-7c7fcb8949-jm5r2"
Sep 11 19:15:15.226366 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:15:15.226324    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9aa9e368-4ae5-495f-a5ec-d2a9bbcc283b-tmp-dir\") pod \"sub-test-model-3b7391-kserve-7c7fcb8949-jm5r2\" (UID: \"9aa9e368-4ae5-495f-a5ec-d2a9bbcc283b\") " pod="ai-tenant-e2e-shared-b-w3-3b7391/sub-test-model-3b7391-kserve-7c7fcb8949-jm5r2"
Sep 11 19:15:15.226366 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:15:15.226347    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9aa9e368-4ae5-495f-a5ec-d2a9bbcc283b-model-cache\") pod \"sub-test-model-3b7391-kserve-7c7fcb8949-jm5r2\" (UID: \"9aa9e368-4ae5-495f-a5ec-d2a9bbcc283b\") " pod="ai-tenant-e2e-shared-b-w3-3b7391/sub-test-model-3b7391-kserve-7c7fcb8949-jm5r2"
Sep 11 19:15:15.228357 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:15:15.228332    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9aa9e368-4ae5-495f-a5ec-d2a9bbcc283b-dshm\") pod \"sub-test-model-3b7391-kserve-7c7fcb8949-jm5r2\" (UID: \"9aa9e368-4ae5-495f-a5ec-d2a9bbcc283b\") " pod="ai-tenant-e2e-shared-b-w3-3b7391/sub-test-model-3b7391-kserve-7c7fcb8949-jm5r2"
Sep 11 19:15:15.228612 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:15:15.228575    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9aa9e368-4ae5-495f-a5ec-d2a9bbcc283b-tls-certs\") pod \"sub-test-model-3b7391-kserve-7c7fcb8949-jm5r2\" (UID: \"9aa9e368-4ae5-495f-a5ec-d2a9bbcc283b\") " pod="ai-tenant-e2e-shared-b-w3-3b7391/sub-test-model-3b7391-kserve-7c7fcb8949-jm5r2"
Sep 11 19:15:15.233330 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:15:15.233309    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxtpz\" (UniqueName: \"kubernetes.io/projected/9aa9e368-4ae5-495f-a5ec-d2a9bbcc283b-kube-api-access-bxtpz\") pod \"sub-test-model-3b7391-kserve-7c7fcb8949-jm5r2\" (UID: \"9aa9e368-4ae5-495f-a5ec-d2a9bbcc283b\") " pod="ai-tenant-e2e-shared-b-w3-3b7391/sub-test-model-3b7391-kserve-7c7fcb8949-jm5r2"
Sep 11 19:15:15.349161 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:15:15.349070    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3b7391/sub-test-model-3b7391-kserve-7c7fcb8949-jm5r2"
Sep 11 19:15:15.620665 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:15:15.620527    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3b7391/sub-test-model-3b7391-kserve-7c7fcb8949-jm5r2"]
Sep 11 19:15:15.622891 ip-10-0-134-189 kubenswrapper[2578]: W0911 19:15:15.622860    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9aa9e368_4ae5_495f_a5ec_d2a9bbcc283b.slice/crio-d3b9852d6e5760af7fb60e057161dee21ef4f44eb32622820d9d96af3ca9db41 WatchSource:0}: Error finding container d3b9852d6e5760af7fb60e057161dee21ef4f44eb32622820d9d96af3ca9db41: Status 404 returned error can't find the container with id d3b9852d6e5760af7fb60e057161dee21ef4f44eb32622820d9d96af3ca9db41
Sep 11 19:15:16.166989 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:15:16.166952    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3b7391/sub-test-model-3b7391-kserve-7c7fcb8949-jm5r2" event={"ID":"9aa9e368-4ae5-495f-a5ec-d2a9bbcc283b","Type":"ContainerStarted","Data":"ea8c7b694dd349013203b141e680b494b34e6e413ee7b64f55ca54f4960c335b"}
Sep 11 19:15:16.167419 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:15:16.166997    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3b7391/sub-test-model-3b7391-kserve-7c7fcb8949-jm5r2" event={"ID":"9aa9e368-4ae5-495f-a5ec-d2a9bbcc283b","Type":"ContainerStarted","Data":"d3b9852d6e5760af7fb60e057161dee21ef4f44eb32622820d9d96af3ca9db41"}
Sep 11 19:15:16.183801 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:15:16.183736    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-3b7391/sub-test-model-3b7391-kserve-7c7fcb8949-jm5r2" podStartSLOduration=1.183719234 podStartE2EDuration="1.183719234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:15:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:15:16.181558432 +0000 UTC m=+1592.259148987" watchObservedRunningTime="2026-09-11 19:15:16.183719234 +0000 UTC m=+1592.261309807"
Sep 11 19:15:25.350223 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:15:25.350164    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-3b7391/sub-test-model-3b7391-kserve-7c7fcb8949-jm5r2"
Sep 11 19:15:25.350223 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:15:25.350235    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-3b7391/sub-test-model-3b7391-kserve-7c7fcb8949-jm5r2"
Sep 11 19:15:25.362822 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:15:25.362795    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-3b7391/sub-test-model-3b7391-kserve-7c7fcb8949-jm5r2"
Sep 11 19:15:26.232257 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:15:26.232224    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-3b7391/sub-test-model-3b7391-kserve-7c7fcb8949-jm5r2"
Sep 11 19:15:44.656235 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:15:44.656137    2578 scope.go:117] "RemoveContainer" containerID="ac3ab7aa9b5c632ec4ea75ecaa8eb6f8eeab53507b6544a4ef8ad8738c6af40b"
Sep 11 19:16:06.820488 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:16:06.820455    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-8280eb/sub-test-model-8280eb-kserve-7d555d5589-hkn2n"]
Sep 11 19:16:07.822090 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:16:07.822053    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3b7391/sub-test-model-3b7391-kserve-7c7fcb8949-jm5r2"]
Sep 11 19:16:11.630875 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:16:11.630823    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-8280eb/rate-test-model-8280eb-kserve-7ffdbd9d9f-nb9pw"]
Sep 11 19:16:11.636288 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:16:11.636260    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-8280eb/rate-test-model-8280eb-kserve-7ffdbd9d9f-nb9pw"
Sep 11 19:16:11.639282 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:16:11.639080    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-8280eb\"/\"rate-test-model-8280eb-kserve-self-signed-certs\""
Sep 11 19:16:11.641665 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:16:11.641644    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-8280eb/rate-test-model-8280eb-kserve-7ffdbd9d9f-nb9pw"]
Sep 11 19:16:11.733409 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:16:11.733370    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57065b1c-bf8a-446e-8b09-fec2879f3398-model-cache\") pod \"rate-test-model-8280eb-kserve-7ffdbd9d9f-nb9pw\" (UID: \"57065b1c-bf8a-446e-8b09-fec2879f3398\") " pod="ai-tenant-e2e-shared-a-w3-8280eb/rate-test-model-8280eb-kserve-7ffdbd9d9f-nb9pw"
Sep 11 19:16:11.733589 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:16:11.733423    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57065b1c-bf8a-446e-8b09-fec2879f3398-tls-certs\") pod \"rate-test-model-8280eb-kserve-7ffdbd9d9f-nb9pw\" (UID: \"57065b1c-bf8a-446e-8b09-fec2879f3398\") " pod="ai-tenant-e2e-shared-a-w3-8280eb/rate-test-model-8280eb-kserve-7ffdbd9d9f-nb9pw"
Sep 11 19:16:11.733589 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:16:11.733540    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57065b1c-bf8a-446e-8b09-fec2879f3398-dshm\") pod \"rate-test-model-8280eb-kserve-7ffdbd9d9f-nb9pw\" (UID: \"57065b1c-bf8a-446e-8b09-fec2879f3398\") " pod="ai-tenant-e2e-shared-a-w3-8280eb/rate-test-model-8280eb-kserve-7ffdbd9d9f-nb9pw"
Sep 11 19:16:11.733589 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:16:11.733568    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57065b1c-bf8a-446e-8b09-fec2879f3398-tmp-dir\") pod \"rate-test-model-8280eb-kserve-7ffdbd9d9f-nb9pw\" (UID: \"57065b1c-bf8a-446e-8b09-fec2879f3398\") " pod="ai-tenant-e2e-shared-a-w3-8280eb/rate-test-model-8280eb-kserve-7ffdbd9d9f-nb9pw"
Sep 11 19:16:11.733755 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:16:11.733628    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57065b1c-bf8a-446e-8b09-fec2879f3398-home\") pod \"rate-test-model-8280eb-kserve-7ffdbd9d9f-nb9pw\" (UID: \"57065b1c-bf8a-446e-8b09-fec2879f3398\") " pod="ai-tenant-e2e-shared-a-w3-8280eb/rate-test-model-8280eb-kserve-7ffdbd9d9f-nb9pw"
Sep 11 19:16:11.733755 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:16:11.733659    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmsmh\" (UniqueName: \"kubernetes.io/projected/57065b1c-bf8a-446e-8b09-fec2879f3398-kube-api-access-hmsmh\") pod \"rate-test-model-8280eb-kserve-7ffdbd9d9f-nb9pw\" (UID: \"57065b1c-bf8a-446e-8b09-fec2879f3398\") " pod="ai-tenant-e2e-shared-a-w3-8280eb/rate-test-model-8280eb-kserve-7ffdbd9d9f-nb9pw"
Sep 11 19:16:11.834925 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:16:11.834881    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57065b1c-bf8a-446e-8b09-fec2879f3398-dshm\") pod \"rate-test-model-8280eb-kserve-7ffdbd9d9f-nb9pw\" (UID: \"57065b1c-bf8a-446e-8b09-fec2879f3398\") " pod="ai-tenant-e2e-shared-a-w3-8280eb/rate-test-model-8280eb-kserve-7ffdbd9d9f-nb9pw"
Sep 11 19:16:11.834925 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:16:11.834926    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57065b1c-bf8a-446e-8b09-fec2879f3398-tmp-dir\") pod \"rate-test-model-8280eb-kserve-7ffdbd9d9f-nb9pw\" (UID: \"57065b1c-bf8a-446e-8b09-fec2879f3398\") " pod="ai-tenant-e2e-shared-a-w3-8280eb/rate-test-model-8280eb-kserve-7ffdbd9d9f-nb9pw"
Sep 11 19:16:11.835195 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:16:11.834963    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57065b1c-bf8a-446e-8b09-fec2879f3398-home\") pod \"rate-test-model-8280eb-kserve-7ffdbd9d9f-nb9pw\" (UID: \"57065b1c-bf8a-446e-8b09-fec2879f3398\") " pod="ai-tenant-e2e-shared-a-w3-8280eb/rate-test-model-8280eb-kserve-7ffdbd9d9f-nb9pw"
Sep 11 19:16:11.835195 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:16:11.835013    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmsmh\" (UniqueName: \"kubernetes.io/projected/57065b1c-bf8a-446e-8b09-fec2879f3398-kube-api-access-hmsmh\") pod \"rate-test-model-8280eb-kserve-7ffdbd9d9f-nb9pw\" (UID: \"57065b1c-bf8a-446e-8b09-fec2879f3398\") " pod="ai-tenant-e2e-shared-a-w3-8280eb/rate-test-model-8280eb-kserve-7ffdbd9d9f-nb9pw"
Sep 11 19:16:11.835195 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:16:11.835041    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57065b1c-bf8a-446e-8b09-fec2879f3398-model-cache\") pod \"rate-test-model-8280eb-kserve-7ffdbd9d9f-nb9pw\" (UID: \"57065b1c-bf8a-446e-8b09-fec2879f3398\") " pod="ai-tenant-e2e-shared-a-w3-8280eb/rate-test-model-8280eb-kserve-7ffdbd9d9f-nb9pw"
Sep 11 19:16:11.835195 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:16:11.835090    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57065b1c-bf8a-446e-8b09-fec2879f3398-tls-certs\") pod \"rate-test-model-8280eb-kserve-7ffdbd9d9f-nb9pw\" (UID: \"57065b1c-bf8a-446e-8b09-fec2879f3398\") " pod="ai-tenant-e2e-shared-a-w3-8280eb/rate-test-model-8280eb-kserve-7ffdbd9d9f-nb9pw"
Sep 11 19:16:11.835474 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:16:11.835446    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57065b1c-bf8a-446e-8b09-fec2879f3398-home\") pod \"rate-test-model-8280eb-kserve-7ffdbd9d9f-nb9pw\" (UID: \"57065b1c-bf8a-446e-8b09-fec2879f3398\") " pod="ai-tenant-e2e-shared-a-w3-8280eb/rate-test-model-8280eb-kserve-7ffdbd9d9f-nb9pw"
Sep 11 19:16:11.835553 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:16:11.835461    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57065b1c-bf8a-446e-8b09-fec2879f3398-tmp-dir\") pod \"rate-test-model-8280eb-kserve-7ffdbd9d9f-nb9pw\" (UID: \"57065b1c-bf8a-446e-8b09-fec2879f3398\") " pod="ai-tenant-e2e-shared-a-w3-8280eb/rate-test-model-8280eb-kserve-7ffdbd9d9f-nb9pw"
Sep 11 19:16:11.835641 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:16:11.835553    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57065b1c-bf8a-446e-8b09-fec2879f3398-model-cache\") pod \"rate-test-model-8280eb-kserve-7ffdbd9d9f-nb9pw\" (UID: \"57065b1c-bf8a-446e-8b09-fec2879f3398\") " pod="ai-tenant-e2e-shared-a-w3-8280eb/rate-test-model-8280eb-kserve-7ffdbd9d9f-nb9pw"
Sep 11 19:16:11.837417 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:16:11.837399    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57065b1c-bf8a-446e-8b09-fec2879f3398-dshm\") pod \"rate-test-model-8280eb-kserve-7ffdbd9d9f-nb9pw\" (UID: \"57065b1c-bf8a-446e-8b09-fec2879f3398\") " pod="ai-tenant-e2e-shared-a-w3-8280eb/rate-test-model-8280eb-kserve-7ffdbd9d9f-nb9pw"
Sep 11 19:16:11.837717 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:16:11.837700    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57065b1c-bf8a-446e-8b09-fec2879f3398-tls-certs\") pod \"rate-test-model-8280eb-kserve-7ffdbd9d9f-nb9pw\" (UID: \"57065b1c-bf8a-446e-8b09-fec2879f3398\") " pod="ai-tenant-e2e-shared-a-w3-8280eb/rate-test-model-8280eb-kserve-7ffdbd9d9f-nb9pw"
Sep 11 19:16:11.842205 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:16:11.842182    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmsmh\" (UniqueName: \"kubernetes.io/projected/57065b1c-bf8a-446e-8b09-fec2879f3398-kube-api-access-hmsmh\") pod \"rate-test-model-8280eb-kserve-7ffdbd9d9f-nb9pw\" (UID: \"57065b1c-bf8a-446e-8b09-fec2879f3398\") " pod="ai-tenant-e2e-shared-a-w3-8280eb/rate-test-model-8280eb-kserve-7ffdbd9d9f-nb9pw"
Sep 11 19:16:11.947854 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:16:11.947745    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-8280eb/rate-test-model-8280eb-kserve-7ffdbd9d9f-nb9pw"
Sep 11 19:16:12.219383 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:16:12.219348    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-8280eb/rate-test-model-8280eb-kserve-7ffdbd9d9f-nb9pw"]
Sep 11 19:16:12.222296 ip-10-0-134-189 kubenswrapper[2578]: W0911 19:16:12.222268    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod57065b1c_bf8a_446e_8b09_fec2879f3398.slice/crio-a64abc02008bb8c3d287a653e3c3e8609c8d482e49594bbe60bf534768bd8fd8 WatchSource:0}: Error finding container a64abc02008bb8c3d287a653e3c3e8609c8d482e49594bbe60bf534768bd8fd8: Status 404 returned error can't find the container with id a64abc02008bb8c3d287a653e3c3e8609c8d482e49594bbe60bf534768bd8fd8
Sep 11 19:16:12.456096 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:16:12.456061    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-8280eb/rate-test-model-8280eb-kserve-7ffdbd9d9f-nb9pw" event={"ID":"57065b1c-bf8a-446e-8b09-fec2879f3398","Type":"ContainerStarted","Data":"5df2c2df46c4bbdc21e05664ffbd41234d79cad00c715751f1232b813ee70d64"}
Sep 11 19:16:12.456096 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:16:12.456107    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-8280eb/rate-test-model-8280eb-kserve-7ffdbd9d9f-nb9pw" event={"ID":"57065b1c-bf8a-446e-8b09-fec2879f3398","Type":"ContainerStarted","Data":"a64abc02008bb8c3d287a653e3c3e8609c8d482e49594bbe60bf534768bd8fd8"}
Sep 11 19:16:12.473668 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:16:12.473541    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-8280eb/rate-test-model-8280eb-kserve-7ffdbd9d9f-nb9pw" podStartSLOduration=1.473521404 podStartE2EDuration="1.473521404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:16:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:16:12.471210542 +0000 UTC m=+1648.548801119" watchObservedRunningTime="2026-09-11 19:16:12.473521404 +0000 UTC m=+1648.551111959"
Sep 11 19:16:21.840476 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:16:21.840415    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-8280eb/sub-test-model-8280eb-kserve-7d555d5589-hkn2n" podUID="9df66431-f3dc-4457-9dfe-2de4fe8f3a92" containerName="main" containerID="cri-o://baf9217f3a54535243990dbc46eacc516a80decb9e5fa34bce6b572e4d8f26bf" gracePeriod=45
Sep 11 19:16:21.947898 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:16:21.947857    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-8280eb/rate-test-model-8280eb-kserve-7ffdbd9d9f-nb9pw"
Sep 11 19:16:21.948061 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:16:21.947909    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-8280eb/rate-test-model-8280eb-kserve-7ffdbd9d9f-nb9pw"
Sep 11 19:16:21.960824 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:16:21.960792    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-8280eb/rate-test-model-8280eb-kserve-7ffdbd9d9f-nb9pw"
Sep 11 19:16:22.084948 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:16:22.084923    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-8280eb/sub-test-model-8280eb-kserve-7d555d5589-hkn2n"
Sep 11 19:16:22.128332 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:16:22.128245    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9df66431-f3dc-4457-9dfe-2de4fe8f3a92-tls-certs\") pod \"9df66431-f3dc-4457-9dfe-2de4fe8f3a92\" (UID: \"9df66431-f3dc-4457-9dfe-2de4fe8f3a92\") "
Sep 11 19:16:22.128332 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:16:22.128313    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9df66431-f3dc-4457-9dfe-2de4fe8f3a92-home\") pod \"9df66431-f3dc-4457-9dfe-2de4fe8f3a92\" (UID: \"9df66431-f3dc-4457-9dfe-2de4fe8f3a92\") "
Sep 11 19:16:22.128527 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:16:22.128337    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9df66431-f3dc-4457-9dfe-2de4fe8f3a92-dshm\") pod \"9df66431-f3dc-4457-9dfe-2de4fe8f3a92\" (UID: \"9df66431-f3dc-4457-9dfe-2de4fe8f3a92\") "
Sep 11 19:16:22.128527 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:16:22.128385    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9df66431-f3dc-4457-9dfe-2de4fe8f3a92-tmp-dir\") pod \"9df66431-f3dc-4457-9dfe-2de4fe8f3a92\" (UID: \"9df66431-f3dc-4457-9dfe-2de4fe8f3a92\") "
Sep 11 19:16:22.128527 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:16:22.128417    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q6r2p\" (UniqueName: \"kubernetes.io/projected/9df66431-f3dc-4457-9dfe-2de4fe8f3a92-kube-api-access-q6r2p\") pod \"9df66431-f3dc-4457-9dfe-2de4fe8f3a92\" (UID: \"9df66431-f3dc-4457-9dfe-2de4fe8f3a92\") "
Sep 11 19:16:22.128527 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:16:22.128466    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9df66431-f3dc-4457-9dfe-2de4fe8f3a92-model-cache\") pod \"9df66431-f3dc-4457-9dfe-2de4fe8f3a92\" (UID: \"9df66431-f3dc-4457-9dfe-2de4fe8f3a92\") "
Sep 11 19:16:22.128767 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:16:22.128646    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9df66431-f3dc-4457-9dfe-2de4fe8f3a92-home" (OuterVolumeSpecName: "home") pod "9df66431-f3dc-4457-9dfe-2de4fe8f3a92" (UID: "9df66431-f3dc-4457-9dfe-2de4fe8f3a92"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:16:22.128767 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:16:22.128676    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9df66431-f3dc-4457-9dfe-2de4fe8f3a92-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9df66431-f3dc-4457-9dfe-2de4fe8f3a92" (UID: "9df66431-f3dc-4457-9dfe-2de4fe8f3a92"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:16:22.128874 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:16:22.128783    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9df66431-f3dc-4457-9dfe-2de4fe8f3a92-home\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:16:22.128874 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:16:22.128800    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9df66431-f3dc-4457-9dfe-2de4fe8f3a92-model-cache" (OuterVolumeSpecName: "model-cache") pod "9df66431-f3dc-4457-9dfe-2de4fe8f3a92" (UID: "9df66431-f3dc-4457-9dfe-2de4fe8f3a92"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:16:22.130860 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:16:22.130818    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9df66431-f3dc-4457-9dfe-2de4fe8f3a92-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9df66431-f3dc-4457-9dfe-2de4fe8f3a92" (UID: "9df66431-f3dc-4457-9dfe-2de4fe8f3a92"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 19:16:22.130860 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:16:22.130850    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9df66431-f3dc-4457-9dfe-2de4fe8f3a92-dshm" (OuterVolumeSpecName: "dshm") pod "9df66431-f3dc-4457-9dfe-2de4fe8f3a92" (UID: "9df66431-f3dc-4457-9dfe-2de4fe8f3a92"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:16:22.131022 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:16:22.130889    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9df66431-f3dc-4457-9dfe-2de4fe8f3a92-kube-api-access-q6r2p" (OuterVolumeSpecName: "kube-api-access-q6r2p") pod "9df66431-f3dc-4457-9dfe-2de4fe8f3a92" (UID: "9df66431-f3dc-4457-9dfe-2de4fe8f3a92"). InnerVolumeSpecName "kube-api-access-q6r2p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:16:22.229854 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:16:22.229804    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9df66431-f3dc-4457-9dfe-2de4fe8f3a92-model-cache\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:16:22.229854 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:16:22.229847    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9df66431-f3dc-4457-9dfe-2de4fe8f3a92-tls-certs\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:16:22.229854 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:16:22.229855    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9df66431-f3dc-4457-9dfe-2de4fe8f3a92-dshm\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:16:22.229854 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:16:22.229862    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9df66431-f3dc-4457-9dfe-2de4fe8f3a92-tmp-dir\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:16:22.229854 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:16:22.229870    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6r2p\" (UniqueName: \"kubernetes.io/projected/9df66431-f3dc-4457-9dfe-2de4fe8f3a92-kube-api-access-q6r2p\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:16:22.506567 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:16:22.506532    2578 generic.go:356] "Generic (PLEG): container finished" podID="9df66431-f3dc-4457-9dfe-2de4fe8f3a92" containerID="baf9217f3a54535243990dbc46eacc516a80decb9e5fa34bce6b572e4d8f26bf" exitCode=0
Sep 11 19:16:22.506782 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:16:22.506574    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-8280eb/sub-test-model-8280eb-kserve-7d555d5589-hkn2n"
Sep 11 19:16:22.506782 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:16:22.506627    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-8280eb/sub-test-model-8280eb-kserve-7d555d5589-hkn2n" event={"ID":"9df66431-f3dc-4457-9dfe-2de4fe8f3a92","Type":"ContainerDied","Data":"baf9217f3a54535243990dbc46eacc516a80decb9e5fa34bce6b572e4d8f26bf"}
Sep 11 19:16:22.506782 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:16:22.506667    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-8280eb/sub-test-model-8280eb-kserve-7d555d5589-hkn2n" event={"ID":"9df66431-f3dc-4457-9dfe-2de4fe8f3a92","Type":"ContainerDied","Data":"07c00e9125541d5a934a97780a81cdd1e78a3ede7c6d5facf70dc5db672e7e22"}
Sep 11 19:16:22.506782 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:16:22.506687    2578 scope.go:117] "RemoveContainer" containerID="baf9217f3a54535243990dbc46eacc516a80decb9e5fa34bce6b572e4d8f26bf"
Sep 11 19:16:22.516870 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:16:22.516848    2578 scope.go:117] "RemoveContainer" containerID="baf9217f3a54535243990dbc46eacc516a80decb9e5fa34bce6b572e4d8f26bf"
Sep 11 19:16:22.517169 ip-10-0-134-189 kubenswrapper[2578]: E0911 19:16:22.517151    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"baf9217f3a54535243990dbc46eacc516a80decb9e5fa34bce6b572e4d8f26bf\": container with ID starting with baf9217f3a54535243990dbc46eacc516a80decb9e5fa34bce6b572e4d8f26bf not found: ID does not exist" containerID="baf9217f3a54535243990dbc46eacc516a80decb9e5fa34bce6b572e4d8f26bf"
Sep 11 19:16:22.517242 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:16:22.517175    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baf9217f3a54535243990dbc46eacc516a80decb9e5fa34bce6b572e4d8f26bf"} err="failed to get container status \"baf9217f3a54535243990dbc46eacc516a80decb9e5fa34bce6b572e4d8f26bf\": rpc error: code = NotFound desc = could not find container \"baf9217f3a54535243990dbc46eacc516a80decb9e5fa34bce6b572e4d8f26bf\": container with ID starting with baf9217f3a54535243990dbc46eacc516a80decb9e5fa34bce6b572e4d8f26bf not found: ID does not exist"
Sep 11 19:16:22.520353 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:16:22.520329    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-8280eb/rate-test-model-8280eb-kserve-7ffdbd9d9f-nb9pw"
Sep 11 19:16:22.522370 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:16:22.522345    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-8280eb/sub-test-model-8280eb-kserve-7d555d5589-hkn2n"]
Sep 11 19:16:22.524010 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:16:22.523990    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-8280eb/sub-test-model-8280eb-kserve-7d555d5589-hkn2n"]
Sep 11 19:16:22.842104 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:16:22.842062    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-3b7391/sub-test-model-3b7391-kserve-7c7fcb8949-jm5r2" podUID="9aa9e368-4ae5-495f-a5ec-d2a9bbcc283b" containerName="main" containerID="cri-o://ea8c7b694dd349013203b141e680b494b34e6e413ee7b64f55ca54f4960c335b" gracePeriod=45
Sep 11 19:16:23.090578 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:16:23.090553    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3b7391/sub-test-model-3b7391-kserve-7c7fcb8949-jm5r2"
Sep 11 19:16:23.138425 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:16:23.138326    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9aa9e368-4ae5-495f-a5ec-d2a9bbcc283b-home\") pod \"9aa9e368-4ae5-495f-a5ec-d2a9bbcc283b\" (UID: \"9aa9e368-4ae5-495f-a5ec-d2a9bbcc283b\") "
Sep 11 19:16:23.138425 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:16:23.138369    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bxtpz\" (UniqueName: \"kubernetes.io/projected/9aa9e368-4ae5-495f-a5ec-d2a9bbcc283b-kube-api-access-bxtpz\") pod \"9aa9e368-4ae5-495f-a5ec-d2a9bbcc283b\" (UID: \"9aa9e368-4ae5-495f-a5ec-d2a9bbcc283b\") "
Sep 11 19:16:23.138425 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:16:23.138408    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9aa9e368-4ae5-495f-a5ec-d2a9bbcc283b-tls-certs\") pod \"9aa9e368-4ae5-495f-a5ec-d2a9bbcc283b\" (UID: \"9aa9e368-4ae5-495f-a5ec-d2a9bbcc283b\") "
Sep 11 19:16:23.138787 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:16:23.138502    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9aa9e368-4ae5-495f-a5ec-d2a9bbcc283b-tmp-dir\") pod \"9aa9e368-4ae5-495f-a5ec-d2a9bbcc283b\" (UID: \"9aa9e368-4ae5-495f-a5ec-d2a9bbcc283b\") "
Sep 11 19:16:23.138787 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:16:23.138543    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9aa9e368-4ae5-495f-a5ec-d2a9bbcc283b-dshm\") pod \"9aa9e368-4ae5-495f-a5ec-d2a9bbcc283b\" (UID: \"9aa9e368-4ae5-495f-a5ec-d2a9bbcc283b\") "
Sep 11 19:16:23.138787 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:16:23.138581    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9aa9e368-4ae5-495f-a5ec-d2a9bbcc283b-model-cache\") pod \"9aa9e368-4ae5-495f-a5ec-d2a9bbcc283b\" (UID: \"9aa9e368-4ae5-495f-a5ec-d2a9bbcc283b\") "
Sep 11 19:16:23.138787 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:16:23.138693    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9aa9e368-4ae5-495f-a5ec-d2a9bbcc283b-home" (OuterVolumeSpecName: "home") pod "9aa9e368-4ae5-495f-a5ec-d2a9bbcc283b" (UID: "9aa9e368-4ae5-495f-a5ec-d2a9bbcc283b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:16:23.139022 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:16:23.138818    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9aa9e368-4ae5-495f-a5ec-d2a9bbcc283b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9aa9e368-4ae5-495f-a5ec-d2a9bbcc283b" (UID: "9aa9e368-4ae5-495f-a5ec-d2a9bbcc283b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:16:23.139022 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:16:23.138956    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9aa9e368-4ae5-495f-a5ec-d2a9bbcc283b-model-cache" (OuterVolumeSpecName: "model-cache") pod "9aa9e368-4ae5-495f-a5ec-d2a9bbcc283b" (UID: "9aa9e368-4ae5-495f-a5ec-d2a9bbcc283b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:16:23.139022 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:16:23.138987    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9aa9e368-4ae5-495f-a5ec-d2a9bbcc283b-tmp-dir\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:16:23.139022 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:16:23.139000    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9aa9e368-4ae5-495f-a5ec-d2a9bbcc283b-home\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:16:23.140823 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:16:23.140793    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9aa9e368-4ae5-495f-a5ec-d2a9bbcc283b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9aa9e368-4ae5-495f-a5ec-d2a9bbcc283b" (UID: "9aa9e368-4ae5-495f-a5ec-d2a9bbcc283b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 19:16:23.140955 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:16:23.140844    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9aa9e368-4ae5-495f-a5ec-d2a9bbcc283b-kube-api-access-bxtpz" (OuterVolumeSpecName: "kube-api-access-bxtpz") pod "9aa9e368-4ae5-495f-a5ec-d2a9bbcc283b" (UID: "9aa9e368-4ae5-495f-a5ec-d2a9bbcc283b"). InnerVolumeSpecName "kube-api-access-bxtpz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:16:23.140955 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:16:23.140860    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9aa9e368-4ae5-495f-a5ec-d2a9bbcc283b-dshm" (OuterVolumeSpecName: "dshm") pod "9aa9e368-4ae5-495f-a5ec-d2a9bbcc283b" (UID: "9aa9e368-4ae5-495f-a5ec-d2a9bbcc283b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:16:23.239646 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:16:23.239578    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxtpz\" (UniqueName: \"kubernetes.io/projected/9aa9e368-4ae5-495f-a5ec-d2a9bbcc283b-kube-api-access-bxtpz\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:16:23.239646 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:16:23.239641    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9aa9e368-4ae5-495f-a5ec-d2a9bbcc283b-tls-certs\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:16:23.239646 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:16:23.239653    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9aa9e368-4ae5-495f-a5ec-d2a9bbcc283b-dshm\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:16:23.239646 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:16:23.239661    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9aa9e368-4ae5-495f-a5ec-d2a9bbcc283b-model-cache\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:16:23.513363 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:16:23.513329    2578 generic.go:356] "Generic (PLEG): container finished" podID="9aa9e368-4ae5-495f-a5ec-d2a9bbcc283b" containerID="ea8c7b694dd349013203b141e680b494b34e6e413ee7b64f55ca54f4960c335b" exitCode=0
Sep 11 19:16:23.513569 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:16:23.513375    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3b7391/sub-test-model-3b7391-kserve-7c7fcb8949-jm5r2"
Sep 11 19:16:23.513569 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:16:23.513415    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3b7391/sub-test-model-3b7391-kserve-7c7fcb8949-jm5r2" event={"ID":"9aa9e368-4ae5-495f-a5ec-d2a9bbcc283b","Type":"ContainerDied","Data":"ea8c7b694dd349013203b141e680b494b34e6e413ee7b64f55ca54f4960c335b"}
Sep 11 19:16:23.513569 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:16:23.513462    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3b7391/sub-test-model-3b7391-kserve-7c7fcb8949-jm5r2" event={"ID":"9aa9e368-4ae5-495f-a5ec-d2a9bbcc283b","Type":"ContainerDied","Data":"d3b9852d6e5760af7fb60e057161dee21ef4f44eb32622820d9d96af3ca9db41"}
Sep 11 19:16:23.513569 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:16:23.513487    2578 scope.go:117] "RemoveContainer" containerID="ea8c7b694dd349013203b141e680b494b34e6e413ee7b64f55ca54f4960c335b"
Sep 11 19:16:23.522693 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:16:23.522673    2578 scope.go:117] "RemoveContainer" containerID="ea8c7b694dd349013203b141e680b494b34e6e413ee7b64f55ca54f4960c335b"
Sep 11 19:16:23.523006 ip-10-0-134-189 kubenswrapper[2578]: E0911 19:16:23.522987    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea8c7b694dd349013203b141e680b494b34e6e413ee7b64f55ca54f4960c335b\": container with ID starting with ea8c7b694dd349013203b141e680b494b34e6e413ee7b64f55ca54f4960c335b not found: ID does not exist" containerID="ea8c7b694dd349013203b141e680b494b34e6e413ee7b64f55ca54f4960c335b"
Sep 11 19:16:23.523064 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:16:23.523014    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea8c7b694dd349013203b141e680b494b34e6e413ee7b64f55ca54f4960c335b"} err="failed to get container status \"ea8c7b694dd349013203b141e680b494b34e6e413ee7b64f55ca54f4960c335b\": rpc error: code = NotFound desc = could not find container \"ea8c7b694dd349013203b141e680b494b34e6e413ee7b64f55ca54f4960c335b\": container with ID starting with ea8c7b694dd349013203b141e680b494b34e6e413ee7b64f55ca54f4960c335b not found: ID does not exist"
Sep 11 19:16:23.532447 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:16:23.532418    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3b7391/sub-test-model-3b7391-kserve-7c7fcb8949-jm5r2"]
Sep 11 19:16:23.536562 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:16:23.536539    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3b7391/sub-test-model-3b7391-kserve-7c7fcb8949-jm5r2"]
Sep 11 19:16:24.397474 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:16:24.397430    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9aa9e368-4ae5-495f-a5ec-d2a9bbcc283b" path="/var/lib/kubelet/pods/9aa9e368-4ae5-495f-a5ec-d2a9bbcc283b/volumes"
Sep 11 19:16:24.398039 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:16:24.398018    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9df66431-f3dc-4457-9dfe-2de4fe8f3a92" path="/var/lib/kubelet/pods/9df66431-f3dc-4457-9dfe-2de4fe8f3a92/volumes"
Sep 11 19:16:44.684281 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:16:44.684243    2578 scope.go:117] "RemoveContainer" containerID="127d82a9f24eb91b1d2f434140c4cf88836516909d426580e99fbfa9ff19ab6f"
Sep 11 19:17:11.234136 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:17:11.234051    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-3b7391/rate-test-model-3b7391-kserve-7dfbbd6cdc-8pcdk"]
Sep 11 19:17:11.234644 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:17:11.234585    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9aa9e368-4ae5-495f-a5ec-d2a9bbcc283b" containerName="main"
Sep 11 19:17:11.234644 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:17:11.234632    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9aa9e368-4ae5-495f-a5ec-d2a9bbcc283b" containerName="main"
Sep 11 19:17:11.234753 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:17:11.234650    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9df66431-f3dc-4457-9dfe-2de4fe8f3a92" containerName="main"
Sep 11 19:17:11.234753 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:17:11.234659    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9df66431-f3dc-4457-9dfe-2de4fe8f3a92" containerName="main"
Sep 11 19:17:11.234822 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:17:11.234768    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9df66431-f3dc-4457-9dfe-2de4fe8f3a92" containerName="main"
Sep 11 19:17:11.234822 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:17:11.234783    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9aa9e368-4ae5-495f-a5ec-d2a9bbcc283b" containerName="main"
Sep 11 19:17:11.238171 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:17:11.238151    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3b7391/rate-test-model-3b7391-kserve-7dfbbd6cdc-8pcdk"
Sep 11 19:17:11.241099 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:17:11.241075    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-3b7391\"/\"kube-root-ca.crt\""
Sep 11 19:17:11.241612 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:17:11.241571    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-3b7391\"/\"default-dockercfg-296hb\""
Sep 11 19:17:11.241878 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:17:11.241856    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-3b7391\"/\"rate-test-model-3b7391-kserve-self-signed-certs\""
Sep 11 19:17:11.242151 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:17:11.242134    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-3b7391\"/\"openshift-service-ca.crt\""
Sep 11 19:17:11.252553 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:17:11.252523    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3b7391/rate-test-model-3b7391-kserve-7dfbbd6cdc-8pcdk"]
Sep 11 19:17:11.383423 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:17:11.383387    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/180d6a07-9bf4-4876-ae17-3318f12b7d2d-home\") pod \"rate-test-model-3b7391-kserve-7dfbbd6cdc-8pcdk\" (UID: \"180d6a07-9bf4-4876-ae17-3318f12b7d2d\") " pod="ai-tenant-e2e-shared-b-w3-3b7391/rate-test-model-3b7391-kserve-7dfbbd6cdc-8pcdk"
Sep 11 19:17:11.383423 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:17:11.383426    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/180d6a07-9bf4-4876-ae17-3318f12b7d2d-dshm\") pod \"rate-test-model-3b7391-kserve-7dfbbd6cdc-8pcdk\" (UID: \"180d6a07-9bf4-4876-ae17-3318f12b7d2d\") " pod="ai-tenant-e2e-shared-b-w3-3b7391/rate-test-model-3b7391-kserve-7dfbbd6cdc-8pcdk"
Sep 11 19:17:11.383679 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:17:11.383509    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dx6p\" (UniqueName: \"kubernetes.io/projected/180d6a07-9bf4-4876-ae17-3318f12b7d2d-kube-api-access-8dx6p\") pod \"rate-test-model-3b7391-kserve-7dfbbd6cdc-8pcdk\" (UID: \"180d6a07-9bf4-4876-ae17-3318f12b7d2d\") " pod="ai-tenant-e2e-shared-b-w3-3b7391/rate-test-model-3b7391-kserve-7dfbbd6cdc-8pcdk"
Sep 11 19:17:11.383679 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:17:11.383586    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/180d6a07-9bf4-4876-ae17-3318f12b7d2d-model-cache\") pod \"rate-test-model-3b7391-kserve-7dfbbd6cdc-8pcdk\" (UID: \"180d6a07-9bf4-4876-ae17-3318f12b7d2d\") " pod="ai-tenant-e2e-shared-b-w3-3b7391/rate-test-model-3b7391-kserve-7dfbbd6cdc-8pcdk"
Sep 11 19:17:11.383679 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:17:11.383638    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/180d6a07-9bf4-4876-ae17-3318f12b7d2d-tls-certs\") pod \"rate-test-model-3b7391-kserve-7dfbbd6cdc-8pcdk\" (UID: \"180d6a07-9bf4-4876-ae17-3318f12b7d2d\") " pod="ai-tenant-e2e-shared-b-w3-3b7391/rate-test-model-3b7391-kserve-7dfbbd6cdc-8pcdk"
Sep 11 19:17:11.383679 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:17:11.383666    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/180d6a07-9bf4-4876-ae17-3318f12b7d2d-tmp-dir\") pod \"rate-test-model-3b7391-kserve-7dfbbd6cdc-8pcdk\" (UID: \"180d6a07-9bf4-4876-ae17-3318f12b7d2d\") " pod="ai-tenant-e2e-shared-b-w3-3b7391/rate-test-model-3b7391-kserve-7dfbbd6cdc-8pcdk"
Sep 11 19:17:11.484697 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:17:11.484579    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/180d6a07-9bf4-4876-ae17-3318f12b7d2d-model-cache\") pod \"rate-test-model-3b7391-kserve-7dfbbd6cdc-8pcdk\" (UID: \"180d6a07-9bf4-4876-ae17-3318f12b7d2d\") " pod="ai-tenant-e2e-shared-b-w3-3b7391/rate-test-model-3b7391-kserve-7dfbbd6cdc-8pcdk"
Sep 11 19:17:11.484697 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:17:11.484646    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/180d6a07-9bf4-4876-ae17-3318f12b7d2d-tls-certs\") pod \"rate-test-model-3b7391-kserve-7dfbbd6cdc-8pcdk\" (UID: \"180d6a07-9bf4-4876-ae17-3318f12b7d2d\") " pod="ai-tenant-e2e-shared-b-w3-3b7391/rate-test-model-3b7391-kserve-7dfbbd6cdc-8pcdk"
Sep 11 19:17:11.484697 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:17:11.484662    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/180d6a07-9bf4-4876-ae17-3318f12b7d2d-tmp-dir\") pod \"rate-test-model-3b7391-kserve-7dfbbd6cdc-8pcdk\" (UID: \"180d6a07-9bf4-4876-ae17-3318f12b7d2d\") " pod="ai-tenant-e2e-shared-b-w3-3b7391/rate-test-model-3b7391-kserve-7dfbbd6cdc-8pcdk"
Sep 11 19:17:11.484697 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:17:11.484699    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/180d6a07-9bf4-4876-ae17-3318f12b7d2d-home\") pod \"rate-test-model-3b7391-kserve-7dfbbd6cdc-8pcdk\" (UID: \"180d6a07-9bf4-4876-ae17-3318f12b7d2d\") " pod="ai-tenant-e2e-shared-b-w3-3b7391/rate-test-model-3b7391-kserve-7dfbbd6cdc-8pcdk"
Sep 11 19:17:11.485036 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:17:11.484722    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/180d6a07-9bf4-4876-ae17-3318f12b7d2d-dshm\") pod \"rate-test-model-3b7391-kserve-7dfbbd6cdc-8pcdk\" (UID: \"180d6a07-9bf4-4876-ae17-3318f12b7d2d\") " pod="ai-tenant-e2e-shared-b-w3-3b7391/rate-test-model-3b7391-kserve-7dfbbd6cdc-8pcdk"
Sep 11 19:17:11.485036 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:17:11.484800    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dx6p\" (UniqueName: \"kubernetes.io/projected/180d6a07-9bf4-4876-ae17-3318f12b7d2d-kube-api-access-8dx6p\") pod \"rate-test-model-3b7391-kserve-7dfbbd6cdc-8pcdk\" (UID: \"180d6a07-9bf4-4876-ae17-3318f12b7d2d\") " pod="ai-tenant-e2e-shared-b-w3-3b7391/rate-test-model-3b7391-kserve-7dfbbd6cdc-8pcdk"
Sep 11 19:17:11.485036 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:17:11.484993    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/180d6a07-9bf4-4876-ae17-3318f12b7d2d-model-cache\") pod \"rate-test-model-3b7391-kserve-7dfbbd6cdc-8pcdk\" (UID: \"180d6a07-9bf4-4876-ae17-3318f12b7d2d\") " pod="ai-tenant-e2e-shared-b-w3-3b7391/rate-test-model-3b7391-kserve-7dfbbd6cdc-8pcdk"
Sep 11 19:17:11.485204 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:17:11.485072    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/180d6a07-9bf4-4876-ae17-3318f12b7d2d-home\") pod \"rate-test-model-3b7391-kserve-7dfbbd6cdc-8pcdk\" (UID: \"180d6a07-9bf4-4876-ae17-3318f12b7d2d\") " pod="ai-tenant-e2e-shared-b-w3-3b7391/rate-test-model-3b7391-kserve-7dfbbd6cdc-8pcdk"
Sep 11 19:17:11.485204 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:17:11.485116    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/180d6a07-9bf4-4876-ae17-3318f12b7d2d-tmp-dir\") pod \"rate-test-model-3b7391-kserve-7dfbbd6cdc-8pcdk\" (UID: \"180d6a07-9bf4-4876-ae17-3318f12b7d2d\") " pod="ai-tenant-e2e-shared-b-w3-3b7391/rate-test-model-3b7391-kserve-7dfbbd6cdc-8pcdk"
Sep 11 19:17:11.487213 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:17:11.487185    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/180d6a07-9bf4-4876-ae17-3318f12b7d2d-dshm\") pod \"rate-test-model-3b7391-kserve-7dfbbd6cdc-8pcdk\" (UID: \"180d6a07-9bf4-4876-ae17-3318f12b7d2d\") " pod="ai-tenant-e2e-shared-b-w3-3b7391/rate-test-model-3b7391-kserve-7dfbbd6cdc-8pcdk"
Sep 11 19:17:11.487479 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:17:11.487452    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/180d6a07-9bf4-4876-ae17-3318f12b7d2d-tls-certs\") pod \"rate-test-model-3b7391-kserve-7dfbbd6cdc-8pcdk\" (UID: \"180d6a07-9bf4-4876-ae17-3318f12b7d2d\") " pod="ai-tenant-e2e-shared-b-w3-3b7391/rate-test-model-3b7391-kserve-7dfbbd6cdc-8pcdk"
Sep 11 19:17:11.495248 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:17:11.495225    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dx6p\" (UniqueName: \"kubernetes.io/projected/180d6a07-9bf4-4876-ae17-3318f12b7d2d-kube-api-access-8dx6p\") pod \"rate-test-model-3b7391-kserve-7dfbbd6cdc-8pcdk\" (UID: \"180d6a07-9bf4-4876-ae17-3318f12b7d2d\") " pod="ai-tenant-e2e-shared-b-w3-3b7391/rate-test-model-3b7391-kserve-7dfbbd6cdc-8pcdk"
Sep 11 19:17:11.552184 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:17:11.552148    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3b7391/rate-test-model-3b7391-kserve-7dfbbd6cdc-8pcdk"
Sep 11 19:17:11.828253 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:17:11.828226    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3b7391/rate-test-model-3b7391-kserve-7dfbbd6cdc-8pcdk"]
Sep 11 19:17:11.830980 ip-10-0-134-189 kubenswrapper[2578]: W0911 19:17:11.830950    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod180d6a07_9bf4_4876_ae17_3318f12b7d2d.slice/crio-759b1c21eb414492c3a164b41282a652c357b54fc06d8b2cf75d74ee05928da5 WatchSource:0}: Error finding container 759b1c21eb414492c3a164b41282a652c357b54fc06d8b2cf75d74ee05928da5: Status 404 returned error can't find the container with id 759b1c21eb414492c3a164b41282a652c357b54fc06d8b2cf75d74ee05928da5
Sep 11 19:17:12.762737 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:17:12.762703    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3b7391/rate-test-model-3b7391-kserve-7dfbbd6cdc-8pcdk" event={"ID":"180d6a07-9bf4-4876-ae17-3318f12b7d2d","Type":"ContainerStarted","Data":"9e33790015ff0d932caeeb076e2b832c5206b5f6fb8267cb46c7bc39394a8447"}
Sep 11 19:17:12.762737 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:17:12.762742    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3b7391/rate-test-model-3b7391-kserve-7dfbbd6cdc-8pcdk" event={"ID":"180d6a07-9bf4-4876-ae17-3318f12b7d2d","Type":"ContainerStarted","Data":"759b1c21eb414492c3a164b41282a652c357b54fc06d8b2cf75d74ee05928da5"}
Sep 11 19:17:12.778888 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:17:12.778835    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-3b7391/rate-test-model-3b7391-kserve-7dfbbd6cdc-8pcdk" podStartSLOduration=1.778808244 podStartE2EDuration="1.778808244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:17:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:17:12.777284409 +0000 UTC m=+1708.854874974" watchObservedRunningTime="2026-09-11 19:17:12.778808244 +0000 UTC m=+1708.856398839"
Sep 11 19:17:21.553576 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:17:21.553536    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-3b7391/rate-test-model-3b7391-kserve-7dfbbd6cdc-8pcdk"
Sep 11 19:17:21.554199 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:17:21.553618    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-3b7391/rate-test-model-3b7391-kserve-7dfbbd6cdc-8pcdk"
Sep 11 19:17:21.566988 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:17:21.566959    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-3b7391/rate-test-model-3b7391-kserve-7dfbbd6cdc-8pcdk"
Sep 11 19:17:21.819821 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:17:21.819735    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-3b7391/rate-test-model-3b7391-kserve-7dfbbd6cdc-8pcdk"
Sep 11 19:18:07.532657 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:07.532617    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-8280eb/rate-test-model-8280eb-kserve-7ffdbd9d9f-nb9pw"]
Sep 11 19:18:08.436244 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:08.436205    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3b7391/rate-test-model-3b7391-kserve-7dfbbd6cdc-8pcdk"]
Sep 11 19:18:14.455093 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:14.455049    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-021613-7ddc8f8c7d-94nhx"]
Sep 11 19:18:14.458726 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:14.458698    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-021613-7ddc8f8c7d-94nhx"
Sep 11 19:18:14.460775 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:14.460755    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-021613\""
Sep 11 19:18:14.469441 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:14.469418    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-021613-7ddc8f8c7d-94nhx"]
Sep 11 19:18:14.472203 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:14.472181    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7k4gs\" (UniqueName: \"kubernetes.io/projected/add780d4-d2a4-4224-9804-7468fedf25ee-kube-api-access-7k4gs\") pod \"payload-pre-processing-e2e-ipp-a-021613-7ddc8f8c7d-94nhx\" (UID: \"add780d4-d2a4-4224-9804-7468fedf25ee\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-021613-7ddc8f8c7d-94nhx"
Sep 11 19:18:14.472317 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:14.472235    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/add780d4-d2a4-4224-9804-7468fedf25ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-021613-7ddc8f8c7d-94nhx\" (UID: \"add780d4-d2a4-4224-9804-7468fedf25ee\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-021613-7ddc8f8c7d-94nhx"
Sep 11 19:18:14.573342 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:14.573290    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/add780d4-d2a4-4224-9804-7468fedf25ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-021613-7ddc8f8c7d-94nhx\" (UID: \"add780d4-d2a4-4224-9804-7468fedf25ee\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-021613-7ddc8f8c7d-94nhx"
Sep 11 19:18:14.573529 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:14.573443    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7k4gs\" (UniqueName: \"kubernetes.io/projected/add780d4-d2a4-4224-9804-7468fedf25ee-kube-api-access-7k4gs\") pod \"payload-pre-processing-e2e-ipp-a-021613-7ddc8f8c7d-94nhx\" (UID: \"add780d4-d2a4-4224-9804-7468fedf25ee\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-021613-7ddc8f8c7d-94nhx"
Sep 11 19:18:14.574251 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:14.574197    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/add780d4-d2a4-4224-9804-7468fedf25ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-021613-7ddc8f8c7d-94nhx\" (UID: \"add780d4-d2a4-4224-9804-7468fedf25ee\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-021613-7ddc8f8c7d-94nhx"
Sep 11 19:18:14.581174 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:14.581143    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7k4gs\" (UniqueName: \"kubernetes.io/projected/add780d4-d2a4-4224-9804-7468fedf25ee-kube-api-access-7k4gs\") pod \"payload-pre-processing-e2e-ipp-a-021613-7ddc8f8c7d-94nhx\" (UID: \"add780d4-d2a4-4224-9804-7468fedf25ee\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-021613-7ddc8f8c7d-94nhx"
Sep 11 19:18:14.770186 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:14.770074    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-021613-7ddc8f8c7d-94nhx"
Sep 11 19:18:15.047849 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:15.047823    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-021613-7ddc8f8c7d-94nhx"]
Sep 11 19:18:15.049907 ip-10-0-134-189 kubenswrapper[2578]: W0911 19:18:15.049882    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadd780d4_d2a4_4224_9804_7468fedf25ee.slice/crio-94e64825faecaf5830adc303949f2407af8bc65f4f4ec6b43be3121413b7d21c WatchSource:0}: Error finding container 94e64825faecaf5830adc303949f2407af8bc65f4f4ec6b43be3121413b7d21c: Status 404 returned error can't find the container with id 94e64825faecaf5830adc303949f2407af8bc65f4f4ec6b43be3121413b7d21c
Sep 11 19:18:15.079933 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:15.079904    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-021613-7ddc8f8c7d-94nhx" event={"ID":"add780d4-d2a4-4224-9804-7468fedf25ee","Type":"ContainerStarted","Data":"94e64825faecaf5830adc303949f2407af8bc65f4f4ec6b43be3121413b7d21c"}
Sep 11 19:18:16.086669 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:16.086631    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-021613-7ddc8f8c7d-94nhx" event={"ID":"add780d4-d2a4-4224-9804-7468fedf25ee","Type":"ContainerStarted","Data":"46c6b01797dee03843c15b1ee763659a36c0abcb8280909ad68285a6f5329999"}
Sep 11 19:18:16.087148 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:16.086698    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-021613-7ddc8f8c7d-94nhx"
Sep 11 19:18:16.102120 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:16.102067    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-021613-7ddc8f8c7d-94nhx" podStartSLOduration=2.102047944 podStartE2EDuration="2.102047944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:18:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:18:16.100729541 +0000 UTC m=+1772.178320125" watchObservedRunningTime="2026-09-11 19:18:16.102047944 +0000 UTC m=+1772.179638541"
Sep 11 19:18:22.553694 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:22.553531    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-8280eb/rate-test-model-8280eb-kserve-7ffdbd9d9f-nb9pw" podUID="57065b1c-bf8a-446e-8b09-fec2879f3398" containerName="main" containerID="cri-o://5df2c2df46c4bbdc21e05664ffbd41234d79cad00c715751f1232b813ee70d64" gracePeriod=45
Sep 11 19:18:22.807634 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:22.807608    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-8280eb/rate-test-model-8280eb-kserve-7ffdbd9d9f-nb9pw"
Sep 11 19:18:22.846698 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:22.846663    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57065b1c-bf8a-446e-8b09-fec2879f3398-home\") pod \"57065b1c-bf8a-446e-8b09-fec2879f3398\" (UID: \"57065b1c-bf8a-446e-8b09-fec2879f3398\") "
Sep 11 19:18:22.846905 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:22.846720    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57065b1c-bf8a-446e-8b09-fec2879f3398-tmp-dir\") pod \"57065b1c-bf8a-446e-8b09-fec2879f3398\" (UID: \"57065b1c-bf8a-446e-8b09-fec2879f3398\") "
Sep 11 19:18:22.846905 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:22.846780    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hmsmh\" (UniqueName: \"kubernetes.io/projected/57065b1c-bf8a-446e-8b09-fec2879f3398-kube-api-access-hmsmh\") pod \"57065b1c-bf8a-446e-8b09-fec2879f3398\" (UID: \"57065b1c-bf8a-446e-8b09-fec2879f3398\") "
Sep 11 19:18:22.846905 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:22.846806    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57065b1c-bf8a-446e-8b09-fec2879f3398-tls-certs\") pod \"57065b1c-bf8a-446e-8b09-fec2879f3398\" (UID: \"57065b1c-bf8a-446e-8b09-fec2879f3398\") "
Sep 11 19:18:22.846905 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:22.846832    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57065b1c-bf8a-446e-8b09-fec2879f3398-dshm\") pod \"57065b1c-bf8a-446e-8b09-fec2879f3398\" (UID: \"57065b1c-bf8a-446e-8b09-fec2879f3398\") "
Sep 11 19:18:22.846905 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:22.846852    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57065b1c-bf8a-446e-8b09-fec2879f3398-model-cache\") pod \"57065b1c-bf8a-446e-8b09-fec2879f3398\" (UID: \"57065b1c-bf8a-446e-8b09-fec2879f3398\") "
Sep 11 19:18:22.847181 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:22.846978    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57065b1c-bf8a-446e-8b09-fec2879f3398-home" (OuterVolumeSpecName: "home") pod "57065b1c-bf8a-446e-8b09-fec2879f3398" (UID: "57065b1c-bf8a-446e-8b09-fec2879f3398"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:18:22.847181 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:22.847079    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57065b1c-bf8a-446e-8b09-fec2879f3398-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "57065b1c-bf8a-446e-8b09-fec2879f3398" (UID: "57065b1c-bf8a-446e-8b09-fec2879f3398"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:18:22.847181 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:22.847137    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57065b1c-bf8a-446e-8b09-fec2879f3398-home\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:18:22.847362 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:22.847238    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57065b1c-bf8a-446e-8b09-fec2879f3398-model-cache" (OuterVolumeSpecName: "model-cache") pod "57065b1c-bf8a-446e-8b09-fec2879f3398" (UID: "57065b1c-bf8a-446e-8b09-fec2879f3398"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:18:22.849182 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:22.849154    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57065b1c-bf8a-446e-8b09-fec2879f3398-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "57065b1c-bf8a-446e-8b09-fec2879f3398" (UID: "57065b1c-bf8a-446e-8b09-fec2879f3398"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 19:18:22.849310 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:22.849275    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57065b1c-bf8a-446e-8b09-fec2879f3398-dshm" (OuterVolumeSpecName: "dshm") pod "57065b1c-bf8a-446e-8b09-fec2879f3398" (UID: "57065b1c-bf8a-446e-8b09-fec2879f3398"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:18:22.849389 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:22.849363    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57065b1c-bf8a-446e-8b09-fec2879f3398-kube-api-access-hmsmh" (OuterVolumeSpecName: "kube-api-access-hmsmh") pod "57065b1c-bf8a-446e-8b09-fec2879f3398" (UID: "57065b1c-bf8a-446e-8b09-fec2879f3398"). InnerVolumeSpecName "kube-api-access-hmsmh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:18:22.947738 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:22.947697    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57065b1c-bf8a-446e-8b09-fec2879f3398-tmp-dir\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:18:22.947738 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:22.947730    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmsmh\" (UniqueName: \"kubernetes.io/projected/57065b1c-bf8a-446e-8b09-fec2879f3398-kube-api-access-hmsmh\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:18:22.947738 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:22.947739    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57065b1c-bf8a-446e-8b09-fec2879f3398-tls-certs\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:18:22.947738 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:22.947746    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57065b1c-bf8a-446e-8b09-fec2879f3398-dshm\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:18:22.948003 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:22.947754    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57065b1c-bf8a-446e-8b09-fec2879f3398-model-cache\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:18:23.124078 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:23.123982    2578 generic.go:356] "Generic (PLEG): container finished" podID="57065b1c-bf8a-446e-8b09-fec2879f3398" containerID="5df2c2df46c4bbdc21e05664ffbd41234d79cad00c715751f1232b813ee70d64" exitCode=0
Sep 11 19:18:23.124078 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:23.124026    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-8280eb/rate-test-model-8280eb-kserve-7ffdbd9d9f-nb9pw"
Sep 11 19:18:23.124320 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:23.124071    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-8280eb/rate-test-model-8280eb-kserve-7ffdbd9d9f-nb9pw" event={"ID":"57065b1c-bf8a-446e-8b09-fec2879f3398","Type":"ContainerDied","Data":"5df2c2df46c4bbdc21e05664ffbd41234d79cad00c715751f1232b813ee70d64"}
Sep 11 19:18:23.124320 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:23.124117    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-8280eb/rate-test-model-8280eb-kserve-7ffdbd9d9f-nb9pw" event={"ID":"57065b1c-bf8a-446e-8b09-fec2879f3398","Type":"ContainerDied","Data":"a64abc02008bb8c3d287a653e3c3e8609c8d482e49594bbe60bf534768bd8fd8"}
Sep 11 19:18:23.124320 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:23.124139    2578 scope.go:117] "RemoveContainer" containerID="5df2c2df46c4bbdc21e05664ffbd41234d79cad00c715751f1232b813ee70d64"
Sep 11 19:18:23.134280 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:23.134262    2578 scope.go:117] "RemoveContainer" containerID="5df2c2df46c4bbdc21e05664ffbd41234d79cad00c715751f1232b813ee70d64"
Sep 11 19:18:23.134647 ip-10-0-134-189 kubenswrapper[2578]: E0911 19:18:23.134623    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5df2c2df46c4bbdc21e05664ffbd41234d79cad00c715751f1232b813ee70d64\": container with ID starting with 5df2c2df46c4bbdc21e05664ffbd41234d79cad00c715751f1232b813ee70d64 not found: ID does not exist" containerID="5df2c2df46c4bbdc21e05664ffbd41234d79cad00c715751f1232b813ee70d64"
Sep 11 19:18:23.134753 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:23.134660    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5df2c2df46c4bbdc21e05664ffbd41234d79cad00c715751f1232b813ee70d64"} err="failed to get container status \"5df2c2df46c4bbdc21e05664ffbd41234d79cad00c715751f1232b813ee70d64\": rpc error: code = NotFound desc = could not find container \"5df2c2df46c4bbdc21e05664ffbd41234d79cad00c715751f1232b813ee70d64\": container with ID starting with 5df2c2df46c4bbdc21e05664ffbd41234d79cad00c715751f1232b813ee70d64 not found: ID does not exist"
Sep 11 19:18:23.144350 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:23.144321    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-8280eb/rate-test-model-8280eb-kserve-7ffdbd9d9f-nb9pw"]
Sep 11 19:18:23.148053 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:23.148031    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-8280eb/rate-test-model-8280eb-kserve-7ffdbd9d9f-nb9pw"]
Sep 11 19:18:23.456161 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:23.456123    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-3b7391/rate-test-model-3b7391-kserve-7dfbbd6cdc-8pcdk" podUID="180d6a07-9bf4-4876-ae17-3318f12b7d2d" containerName="main" containerID="cri-o://9e33790015ff0d932caeeb076e2b832c5206b5f6fb8267cb46c7bc39394a8447" gracePeriod=45
Sep 11 19:18:23.701349 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:23.701323    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3b7391/rate-test-model-3b7391-kserve-7dfbbd6cdc-8pcdk"
Sep 11 19:18:23.755692 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:23.755585    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8dx6p\" (UniqueName: \"kubernetes.io/projected/180d6a07-9bf4-4876-ae17-3318f12b7d2d-kube-api-access-8dx6p\") pod \"180d6a07-9bf4-4876-ae17-3318f12b7d2d\" (UID: \"180d6a07-9bf4-4876-ae17-3318f12b7d2d\") "
Sep 11 19:18:23.755692 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:23.755666    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/180d6a07-9bf4-4876-ae17-3318f12b7d2d-home\") pod \"180d6a07-9bf4-4876-ae17-3318f12b7d2d\" (UID: \"180d6a07-9bf4-4876-ae17-3318f12b7d2d\") "
Sep 11 19:18:23.755919 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:23.755707    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/180d6a07-9bf4-4876-ae17-3318f12b7d2d-tmp-dir\") pod \"180d6a07-9bf4-4876-ae17-3318f12b7d2d\" (UID: \"180d6a07-9bf4-4876-ae17-3318f12b7d2d\") "
Sep 11 19:18:23.755919 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:23.755730    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/180d6a07-9bf4-4876-ae17-3318f12b7d2d-model-cache\") pod \"180d6a07-9bf4-4876-ae17-3318f12b7d2d\" (UID: \"180d6a07-9bf4-4876-ae17-3318f12b7d2d\") "
Sep 11 19:18:23.755919 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:23.755804    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/180d6a07-9bf4-4876-ae17-3318f12b7d2d-tls-certs\") pod \"180d6a07-9bf4-4876-ae17-3318f12b7d2d\" (UID: \"180d6a07-9bf4-4876-ae17-3318f12b7d2d\") "
Sep 11 19:18:23.755919 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:23.755838    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/180d6a07-9bf4-4876-ae17-3318f12b7d2d-dshm\") pod \"180d6a07-9bf4-4876-ae17-3318f12b7d2d\" (UID: \"180d6a07-9bf4-4876-ae17-3318f12b7d2d\") "
Sep 11 19:18:23.756131 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:23.755965    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/180d6a07-9bf4-4876-ae17-3318f12b7d2d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "180d6a07-9bf4-4876-ae17-3318f12b7d2d" (UID: "180d6a07-9bf4-4876-ae17-3318f12b7d2d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:18:23.756131 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:23.755980    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/180d6a07-9bf4-4876-ae17-3318f12b7d2d-home" (OuterVolumeSpecName: "home") pod "180d6a07-9bf4-4876-ae17-3318f12b7d2d" (UID: "180d6a07-9bf4-4876-ae17-3318f12b7d2d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:18:23.756131 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:23.756049    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/180d6a07-9bf4-4876-ae17-3318f12b7d2d-model-cache" (OuterVolumeSpecName: "model-cache") pod "180d6a07-9bf4-4876-ae17-3318f12b7d2d" (UID: "180d6a07-9bf4-4876-ae17-3318f12b7d2d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:18:23.756263 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:23.756141    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/180d6a07-9bf4-4876-ae17-3318f12b7d2d-home\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:18:23.756263 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:23.756159    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/180d6a07-9bf4-4876-ae17-3318f12b7d2d-tmp-dir\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:18:23.756263 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:23.756176    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/180d6a07-9bf4-4876-ae17-3318f12b7d2d-model-cache\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:18:23.757974 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:23.757949    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/180d6a07-9bf4-4876-ae17-3318f12b7d2d-dshm" (OuterVolumeSpecName: "dshm") pod "180d6a07-9bf4-4876-ae17-3318f12b7d2d" (UID: "180d6a07-9bf4-4876-ae17-3318f12b7d2d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:18:23.758094 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:23.758016    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/180d6a07-9bf4-4876-ae17-3318f12b7d2d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "180d6a07-9bf4-4876-ae17-3318f12b7d2d" (UID: "180d6a07-9bf4-4876-ae17-3318f12b7d2d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 19:18:23.758094 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:23.758065    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/180d6a07-9bf4-4876-ae17-3318f12b7d2d-kube-api-access-8dx6p" (OuterVolumeSpecName: "kube-api-access-8dx6p") pod "180d6a07-9bf4-4876-ae17-3318f12b7d2d" (UID: "180d6a07-9bf4-4876-ae17-3318f12b7d2d"). InnerVolumeSpecName "kube-api-access-8dx6p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:18:23.857587 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:23.857547    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/180d6a07-9bf4-4876-ae17-3318f12b7d2d-tls-certs\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:18:23.857587 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:23.857584    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/180d6a07-9bf4-4876-ae17-3318f12b7d2d-dshm\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:18:23.857835 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:23.857640    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dx6p\" (UniqueName: \"kubernetes.io/projected/180d6a07-9bf4-4876-ae17-3318f12b7d2d-kube-api-access-8dx6p\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:18:24.131178 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:24.131092    2578 generic.go:356] "Generic (PLEG): container finished" podID="180d6a07-9bf4-4876-ae17-3318f12b7d2d" containerID="9e33790015ff0d932caeeb076e2b832c5206b5f6fb8267cb46c7bc39394a8447" exitCode=0
Sep 11 19:18:24.131178 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:24.131143    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3b7391/rate-test-model-3b7391-kserve-7dfbbd6cdc-8pcdk"
Sep 11 19:18:24.131178 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:24.131167    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3b7391/rate-test-model-3b7391-kserve-7dfbbd6cdc-8pcdk" event={"ID":"180d6a07-9bf4-4876-ae17-3318f12b7d2d","Type":"ContainerDied","Data":"9e33790015ff0d932caeeb076e2b832c5206b5f6fb8267cb46c7bc39394a8447"}
Sep 11 19:18:24.131440 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:24.131203    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3b7391/rate-test-model-3b7391-kserve-7dfbbd6cdc-8pcdk" event={"ID":"180d6a07-9bf4-4876-ae17-3318f12b7d2d","Type":"ContainerDied","Data":"759b1c21eb414492c3a164b41282a652c357b54fc06d8b2cf75d74ee05928da5"}
Sep 11 19:18:24.131440 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:24.131222    2578 scope.go:117] "RemoveContainer" containerID="9e33790015ff0d932caeeb076e2b832c5206b5f6fb8267cb46c7bc39394a8447"
Sep 11 19:18:24.140690 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:24.140670    2578 scope.go:117] "RemoveContainer" containerID="9e33790015ff0d932caeeb076e2b832c5206b5f6fb8267cb46c7bc39394a8447"
Sep 11 19:18:24.140989 ip-10-0-134-189 kubenswrapper[2578]: E0911 19:18:24.140968    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e33790015ff0d932caeeb076e2b832c5206b5f6fb8267cb46c7bc39394a8447\": container with ID starting with 9e33790015ff0d932caeeb076e2b832c5206b5f6fb8267cb46c7bc39394a8447 not found: ID does not exist" containerID="9e33790015ff0d932caeeb076e2b832c5206b5f6fb8267cb46c7bc39394a8447"
Sep 11 19:18:24.141036 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:24.140999    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e33790015ff0d932caeeb076e2b832c5206b5f6fb8267cb46c7bc39394a8447"} err="failed to get container status \"9e33790015ff0d932caeeb076e2b832c5206b5f6fb8267cb46c7bc39394a8447\": rpc error: code = NotFound desc = could not find container \"9e33790015ff0d932caeeb076e2b832c5206b5f6fb8267cb46c7bc39394a8447\": container with ID starting with 9e33790015ff0d932caeeb076e2b832c5206b5f6fb8267cb46c7bc39394a8447 not found: ID does not exist"
Sep 11 19:18:24.151816 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:24.151791    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3b7391/rate-test-model-3b7391-kserve-7dfbbd6cdc-8pcdk"]
Sep 11 19:18:24.155609 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:24.155568    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3b7391/rate-test-model-3b7391-kserve-7dfbbd6cdc-8pcdk"]
Sep 11 19:18:24.396806 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:24.396722    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="180d6a07-9bf4-4876-ae17-3318f12b7d2d" path="/var/lib/kubelet/pods/180d6a07-9bf4-4876-ae17-3318f12b7d2d/volumes"
Sep 11 19:18:24.397139 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:24.397125    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57065b1c-bf8a-446e-8b09-fec2879f3398" path="/var/lib/kubelet/pods/57065b1c-bf8a-446e-8b09-fec2879f3398/volumes"
Sep 11 19:18:27.095504 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:27.095467    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-021613-7ddc8f8c7d-94nhx"
Sep 11 19:18:37.886323 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:37.886232    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-afd85c-openshift-default-7d69487dfd-hvsgv"]
Sep 11 19:18:37.886840 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:37.886793    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57065b1c-bf8a-446e-8b09-fec2879f3398" containerName="main"
Sep 11 19:18:37.886840 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:37.886812    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="57065b1c-bf8a-446e-8b09-fec2879f3398" containerName="main"
Sep 11 19:18:37.886840 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:37.886832    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="180d6a07-9bf4-4876-ae17-3318f12b7d2d" containerName="main"
Sep 11 19:18:37.886840 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:37.886840    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="180d6a07-9bf4-4876-ae17-3318f12b7d2d" containerName="main"
Sep 11 19:18:37.887057 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:37.886936    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="57065b1c-bf8a-446e-8b09-fec2879f3398" containerName="main"
Sep 11 19:18:37.887057 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:37.886948    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="180d6a07-9bf4-4876-ae17-3318f12b7d2d" containerName="main"
Sep 11 19:18:37.891087 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:37.891063    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-afd85c-openshift-default-7d69487dfd-hvsgv"
Sep 11 19:18:37.904767 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:37.904738    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-afd85c-openshift-default-7d69487dfd-hvsgv"]
Sep 11 19:18:37.908885 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:37.908857    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-afd85c-openshift-default-dockercfg-8xthh\""
Sep 11 19:18:37.991119 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:37.991072    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bfe0225-fb2e-4616-ba97-26f83d861f1e-credential-socket\") pod \"e2e-ipp-b-afd85c-openshift-default-7d69487dfd-hvsgv\" (UID: \"2bfe0225-fb2e-4616-ba97-26f83d861f1e\") " pod="openshift-ingress/e2e-ipp-b-afd85c-openshift-default-7d69487dfd-hvsgv"
Sep 11 19:18:37.991306 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:37.991149    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bfe0225-fb2e-4616-ba97-26f83d861f1e-istio-podinfo\") pod \"e2e-ipp-b-afd85c-openshift-default-7d69487dfd-hvsgv\" (UID: \"2bfe0225-fb2e-4616-ba97-26f83d861f1e\") " pod="openshift-ingress/e2e-ipp-b-afd85c-openshift-default-7d69487dfd-hvsgv"
Sep 11 19:18:37.991306 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:37.991181    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dx46\" (UniqueName: \"kubernetes.io/projected/2bfe0225-fb2e-4616-ba97-26f83d861f1e-kube-api-access-8dx46\") pod \"e2e-ipp-b-afd85c-openshift-default-7d69487dfd-hvsgv\" (UID: \"2bfe0225-fb2e-4616-ba97-26f83d861f1e\") " pod="openshift-ingress/e2e-ipp-b-afd85c-openshift-default-7d69487dfd-hvsgv"
Sep 11 19:18:37.991306 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:37.991229    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bfe0225-fb2e-4616-ba97-26f83d861f1e-istio-data\") pod \"e2e-ipp-b-afd85c-openshift-default-7d69487dfd-hvsgv\" (UID: \"2bfe0225-fb2e-4616-ba97-26f83d861f1e\") " pod="openshift-ingress/e2e-ipp-b-afd85c-openshift-default-7d69487dfd-hvsgv"
Sep 11 19:18:37.991306 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:37.991258    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bfe0225-fb2e-4616-ba97-26f83d861f1e-workload-socket\") pod \"e2e-ipp-b-afd85c-openshift-default-7d69487dfd-hvsgv\" (UID: \"2bfe0225-fb2e-4616-ba97-26f83d861f1e\") " pod="openshift-ingress/e2e-ipp-b-afd85c-openshift-default-7d69487dfd-hvsgv"
Sep 11 19:18:37.991306 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:37.991281    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bfe0225-fb2e-4616-ba97-26f83d861f1e-istiod-ca-cert\") pod \"e2e-ipp-b-afd85c-openshift-default-7d69487dfd-hvsgv\" (UID: \"2bfe0225-fb2e-4616-ba97-26f83d861f1e\") " pod="openshift-ingress/e2e-ipp-b-afd85c-openshift-default-7d69487dfd-hvsgv"
Sep 11 19:18:37.991514 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:37.991363    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bfe0225-fb2e-4616-ba97-26f83d861f1e-istio-envoy\") pod \"e2e-ipp-b-afd85c-openshift-default-7d69487dfd-hvsgv\" (UID: \"2bfe0225-fb2e-4616-ba97-26f83d861f1e\") " pod="openshift-ingress/e2e-ipp-b-afd85c-openshift-default-7d69487dfd-hvsgv"
Sep 11 19:18:37.991514 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:37.991392    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bfe0225-fb2e-4616-ba97-26f83d861f1e-workload-certs\") pod \"e2e-ipp-b-afd85c-openshift-default-7d69487dfd-hvsgv\" (UID: \"2bfe0225-fb2e-4616-ba97-26f83d861f1e\") " pod="openshift-ingress/e2e-ipp-b-afd85c-openshift-default-7d69487dfd-hvsgv"
Sep 11 19:18:37.991514 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:37.991433    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bfe0225-fb2e-4616-ba97-26f83d861f1e-istio-token\") pod \"e2e-ipp-b-afd85c-openshift-default-7d69487dfd-hvsgv\" (UID: \"2bfe0225-fb2e-4616-ba97-26f83d861f1e\") " pod="openshift-ingress/e2e-ipp-b-afd85c-openshift-default-7d69487dfd-hvsgv"
Sep 11 19:18:38.092086 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:38.092043    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bfe0225-fb2e-4616-ba97-26f83d861f1e-istio-envoy\") pod \"e2e-ipp-b-afd85c-openshift-default-7d69487dfd-hvsgv\" (UID: \"2bfe0225-fb2e-4616-ba97-26f83d861f1e\") " pod="openshift-ingress/e2e-ipp-b-afd85c-openshift-default-7d69487dfd-hvsgv"
Sep 11 19:18:38.092086 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:38.092085    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bfe0225-fb2e-4616-ba97-26f83d861f1e-workload-certs\") pod \"e2e-ipp-b-afd85c-openshift-default-7d69487dfd-hvsgv\" (UID: \"2bfe0225-fb2e-4616-ba97-26f83d861f1e\") " pod="openshift-ingress/e2e-ipp-b-afd85c-openshift-default-7d69487dfd-hvsgv"
Sep 11 19:18:38.092352 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:38.092108    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bfe0225-fb2e-4616-ba97-26f83d861f1e-istio-token\") pod \"e2e-ipp-b-afd85c-openshift-default-7d69487dfd-hvsgv\" (UID: \"2bfe0225-fb2e-4616-ba97-26f83d861f1e\") " pod="openshift-ingress/e2e-ipp-b-afd85c-openshift-default-7d69487dfd-hvsgv"
Sep 11 19:18:38.092352 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:38.092241    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bfe0225-fb2e-4616-ba97-26f83d861f1e-credential-socket\") pod \"e2e-ipp-b-afd85c-openshift-default-7d69487dfd-hvsgv\" (UID: \"2bfe0225-fb2e-4616-ba97-26f83d861f1e\") " pod="openshift-ingress/e2e-ipp-b-afd85c-openshift-default-7d69487dfd-hvsgv"
Sep 11 19:18:38.092352 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:38.092293    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bfe0225-fb2e-4616-ba97-26f83d861f1e-istio-podinfo\") pod \"e2e-ipp-b-afd85c-openshift-default-7d69487dfd-hvsgv\" (UID: \"2bfe0225-fb2e-4616-ba97-26f83d861f1e\") " pod="openshift-ingress/e2e-ipp-b-afd85c-openshift-default-7d69487dfd-hvsgv"
Sep 11 19:18:38.092352 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:38.092326    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dx46\" (UniqueName: \"kubernetes.io/projected/2bfe0225-fb2e-4616-ba97-26f83d861f1e-kube-api-access-8dx46\") pod \"e2e-ipp-b-afd85c-openshift-default-7d69487dfd-hvsgv\" (UID: \"2bfe0225-fb2e-4616-ba97-26f83d861f1e\") " pod="openshift-ingress/e2e-ipp-b-afd85c-openshift-default-7d69487dfd-hvsgv"
Sep 11 19:18:38.092568 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:38.092378    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bfe0225-fb2e-4616-ba97-26f83d861f1e-istio-data\") pod \"e2e-ipp-b-afd85c-openshift-default-7d69487dfd-hvsgv\" (UID: \"2bfe0225-fb2e-4616-ba97-26f83d861f1e\") " pod="openshift-ingress/e2e-ipp-b-afd85c-openshift-default-7d69487dfd-hvsgv"
Sep 11 19:18:38.092568 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:38.092426    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bfe0225-fb2e-4616-ba97-26f83d861f1e-workload-socket\") pod \"e2e-ipp-b-afd85c-openshift-default-7d69487dfd-hvsgv\" (UID: \"2bfe0225-fb2e-4616-ba97-26f83d861f1e\") " pod="openshift-ingress/e2e-ipp-b-afd85c-openshift-default-7d69487dfd-hvsgv"
Sep 11 19:18:38.092568 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:38.092458    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bfe0225-fb2e-4616-ba97-26f83d861f1e-istiod-ca-cert\") pod \"e2e-ipp-b-afd85c-openshift-default-7d69487dfd-hvsgv\" (UID: \"2bfe0225-fb2e-4616-ba97-26f83d861f1e\") " pod="openshift-ingress/e2e-ipp-b-afd85c-openshift-default-7d69487dfd-hvsgv"
Sep 11 19:18:38.092568 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:38.092513    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bfe0225-fb2e-4616-ba97-26f83d861f1e-workload-certs\") pod \"e2e-ipp-b-afd85c-openshift-default-7d69487dfd-hvsgv\" (UID: \"2bfe0225-fb2e-4616-ba97-26f83d861f1e\") " pod="openshift-ingress/e2e-ipp-b-afd85c-openshift-default-7d69487dfd-hvsgv"
Sep 11 19:18:38.092813 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:38.092631    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bfe0225-fb2e-4616-ba97-26f83d861f1e-credential-socket\") pod \"e2e-ipp-b-afd85c-openshift-default-7d69487dfd-hvsgv\" (UID: \"2bfe0225-fb2e-4616-ba97-26f83d861f1e\") " pod="openshift-ingress/e2e-ipp-b-afd85c-openshift-default-7d69487dfd-hvsgv"
Sep 11 19:18:38.092813 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:38.092785    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bfe0225-fb2e-4616-ba97-26f83d861f1e-workload-socket\") pod \"e2e-ipp-b-afd85c-openshift-default-7d69487dfd-hvsgv\" (UID: \"2bfe0225-fb2e-4616-ba97-26f83d861f1e\") " pod="openshift-ingress/e2e-ipp-b-afd85c-openshift-default-7d69487dfd-hvsgv"
Sep 11 19:18:38.092934 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:38.092916    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bfe0225-fb2e-4616-ba97-26f83d861f1e-istio-data\") pod \"e2e-ipp-b-afd85c-openshift-default-7d69487dfd-hvsgv\" (UID: \"2bfe0225-fb2e-4616-ba97-26f83d861f1e\") " pod="openshift-ingress/e2e-ipp-b-afd85c-openshift-default-7d69487dfd-hvsgv"
Sep 11 19:18:38.093365 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:38.093344    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bfe0225-fb2e-4616-ba97-26f83d861f1e-istiod-ca-cert\") pod \"e2e-ipp-b-afd85c-openshift-default-7d69487dfd-hvsgv\" (UID: \"2bfe0225-fb2e-4616-ba97-26f83d861f1e\") " pod="openshift-ingress/e2e-ipp-b-afd85c-openshift-default-7d69487dfd-hvsgv"
Sep 11 19:18:38.094762 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:38.094737    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bfe0225-fb2e-4616-ba97-26f83d861f1e-istio-envoy\") pod \"e2e-ipp-b-afd85c-openshift-default-7d69487dfd-hvsgv\" (UID: \"2bfe0225-fb2e-4616-ba97-26f83d861f1e\") " pod="openshift-ingress/e2e-ipp-b-afd85c-openshift-default-7d69487dfd-hvsgv"
Sep 11 19:18:38.094952 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:38.094932    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bfe0225-fb2e-4616-ba97-26f83d861f1e-istio-podinfo\") pod \"e2e-ipp-b-afd85c-openshift-default-7d69487dfd-hvsgv\" (UID: \"2bfe0225-fb2e-4616-ba97-26f83d861f1e\") " pod="openshift-ingress/e2e-ipp-b-afd85c-openshift-default-7d69487dfd-hvsgv"
Sep 11 19:18:38.100297 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:38.100268    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bfe0225-fb2e-4616-ba97-26f83d861f1e-istio-token\") pod \"e2e-ipp-b-afd85c-openshift-default-7d69487dfd-hvsgv\" (UID: \"2bfe0225-fb2e-4616-ba97-26f83d861f1e\") " pod="openshift-ingress/e2e-ipp-b-afd85c-openshift-default-7d69487dfd-hvsgv"
Sep 11 19:18:38.100384 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:38.100356    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dx46\" (UniqueName: \"kubernetes.io/projected/2bfe0225-fb2e-4616-ba97-26f83d861f1e-kube-api-access-8dx46\") pod \"e2e-ipp-b-afd85c-openshift-default-7d69487dfd-hvsgv\" (UID: \"2bfe0225-fb2e-4616-ba97-26f83d861f1e\") " pod="openshift-ingress/e2e-ipp-b-afd85c-openshift-default-7d69487dfd-hvsgv"
Sep 11 19:18:38.201440 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:38.201410    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-afd85c-openshift-default-7d69487dfd-hvsgv"
Sep 11 19:18:38.480810 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:38.480705    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-afd85c-openshift-default-7d69487dfd-hvsgv"]
Sep 11 19:18:38.483275 ip-10-0-134-189 kubenswrapper[2578]: W0911 19:18:38.483240    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bfe0225_fb2e_4616_ba97_26f83d861f1e.slice/crio-497566b73090b419702df16930f291c74f3a0af87e51d5e649cbc9181aa972c0 WatchSource:0}: Error finding container 497566b73090b419702df16930f291c74f3a0af87e51d5e649cbc9181aa972c0: Status 404 returned error can't find the container with id 497566b73090b419702df16930f291c74f3a0af87e51d5e649cbc9181aa972c0
Sep 11 19:18:38.485562 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:38.485530    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 19:18:38.485724 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:38.485616    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 19:18:38.485724 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:38.485648    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 19:18:39.209856 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:39.209819    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-afd85c-openshift-default-7d69487dfd-hvsgv" event={"ID":"2bfe0225-fb2e-4616-ba97-26f83d861f1e","Type":"ContainerStarted","Data":"5dfe84568a7a65e4b15aaa36e17306820cd07f47df78c65e6965a0103aadf2b6"}
Sep 11 19:18:39.210300 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:39.209879    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-afd85c-openshift-default-7d69487dfd-hvsgv" event={"ID":"2bfe0225-fb2e-4616-ba97-26f83d861f1e","Type":"ContainerStarted","Data":"497566b73090b419702df16930f291c74f3a0af87e51d5e649cbc9181aa972c0"}
Sep 11 19:18:39.230474 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:39.230411    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-afd85c-openshift-default-7d69487dfd-hvsgv" podStartSLOduration=2.230392189 podStartE2EDuration="2.230392189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:18:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:18:39.22801226 +0000 UTC m=+1795.305602816" watchObservedRunningTime="2026-09-11 19:18:39.230392189 +0000 UTC m=+1795.307982833"
Sep 11 19:18:40.201994 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:40.201950    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-afd85c-openshift-default-7d69487dfd-hvsgv"
Sep 11 19:18:40.207106 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:40.207077    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-afd85c-openshift-default-7d69487dfd-hvsgv"
Sep 11 19:18:40.215249 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:40.215224    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-afd85c-openshift-default-7d69487dfd-hvsgv"
Sep 11 19:18:40.216182 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:40.216162    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-afd85c-openshift-default-7d69487dfd-hvsgv"
Sep 11 19:18:44.191793 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:44.191758    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-afd85c-5c4cfc56c4-jw45m"]
Sep 11 19:18:44.195028 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:44.195012    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-afd85c-5c4cfc56c4-jw45m"
Sep 11 19:18:44.197072 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:44.197046    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-afd85c\""
Sep 11 19:18:44.197206 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:44.197098    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-afd85c-dockercfg-hzhrt\""
Sep 11 19:18:44.204670 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:44.204642    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-afd85c-5c4cfc56c4-jw45m"]
Sep 11 19:18:44.351268 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:44.351223    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17e21aa6-abdb-4fe1-a304-4e6c3bf03b2e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-afd85c-5c4cfc56c4-jw45m\" (UID: \"17e21aa6-abdb-4fe1-a304-4e6c3bf03b2e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-afd85c-5c4cfc56c4-jw45m"
Sep 11 19:18:44.351497 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:44.351338    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cww5v\" (UniqueName: \"kubernetes.io/projected/17e21aa6-abdb-4fe1-a304-4e6c3bf03b2e-kube-api-access-cww5v\") pod \"payload-pre-processing-e2e-ipp-b-afd85c-5c4cfc56c4-jw45m\" (UID: \"17e21aa6-abdb-4fe1-a304-4e6c3bf03b2e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-afd85c-5c4cfc56c4-jw45m"
Sep 11 19:18:44.432009 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:44.431978    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-hsbsf_d0880abe-ec36-48e4-a511-ec663873fffa/console-operator/1.log"
Sep 11 19:18:44.438669 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:44.438644    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-hsbsf_d0880abe-ec36-48e4-a511-ec663873fffa/console-operator/1.log"
Sep 11 19:18:44.452531 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:44.452508    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cww5v\" (UniqueName: \"kubernetes.io/projected/17e21aa6-abdb-4fe1-a304-4e6c3bf03b2e-kube-api-access-cww5v\") pod \"payload-pre-processing-e2e-ipp-b-afd85c-5c4cfc56c4-jw45m\" (UID: \"17e21aa6-abdb-4fe1-a304-4e6c3bf03b2e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-afd85c-5c4cfc56c4-jw45m"
Sep 11 19:18:44.452722 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:44.452552    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17e21aa6-abdb-4fe1-a304-4e6c3bf03b2e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-afd85c-5c4cfc56c4-jw45m\" (UID: \"17e21aa6-abdb-4fe1-a304-4e6c3bf03b2e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-afd85c-5c4cfc56c4-jw45m"
Sep 11 19:18:44.454505 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:44.454488    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-afd85c\""
Sep 11 19:18:44.460014 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:44.459987    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cww5v\" (UniqueName: \"kubernetes.io/projected/17e21aa6-abdb-4fe1-a304-4e6c3bf03b2e-kube-api-access-cww5v\") pod \"payload-pre-processing-e2e-ipp-b-afd85c-5c4cfc56c4-jw45m\" (UID: \"17e21aa6-abdb-4fe1-a304-4e6c3bf03b2e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-afd85c-5c4cfc56c4-jw45m"
Sep 11 19:18:44.464216 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:44.464191    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17e21aa6-abdb-4fe1-a304-4e6c3bf03b2e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-afd85c-5c4cfc56c4-jw45m\" (UID: \"17e21aa6-abdb-4fe1-a304-4e6c3bf03b2e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-afd85c-5c4cfc56c4-jw45m"
Sep 11 19:18:44.507919 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:44.507885    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-afd85c-dockercfg-hzhrt\""
Sep 11 19:18:44.516749 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:44.516724    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-afd85c-5c4cfc56c4-jw45m"
Sep 11 19:18:44.785644 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:44.785614    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-afd85c-5c4cfc56c4-jw45m"]
Sep 11 19:18:44.788552 ip-10-0-134-189 kubenswrapper[2578]: W0911 19:18:44.788520    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17e21aa6_abdb_4fe1_a304_4e6c3bf03b2e.slice/crio-74fdca47166c1854560f28ca5c4073a347c8f5648ecde9ca0f4cc196be97b389 WatchSource:0}: Error finding container 74fdca47166c1854560f28ca5c4073a347c8f5648ecde9ca0f4cc196be97b389: Status 404 returned error can't find the container with id 74fdca47166c1854560f28ca5c4073a347c8f5648ecde9ca0f4cc196be97b389
Sep 11 19:18:45.241869 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:45.241831    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-afd85c-5c4cfc56c4-jw45m" event={"ID":"17e21aa6-abdb-4fe1-a304-4e6c3bf03b2e","Type":"ContainerStarted","Data":"f11c085c5252519e7d89211e8fe2871d000c1ffa15dd019817dbf772777435a3"}
Sep 11 19:18:45.242291 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:45.241891    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-afd85c-5c4cfc56c4-jw45m" event={"ID":"17e21aa6-abdb-4fe1-a304-4e6c3bf03b2e","Type":"ContainerStarted","Data":"74fdca47166c1854560f28ca5c4073a347c8f5648ecde9ca0f4cc196be97b389"}
Sep 11 19:18:45.242291 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:45.241942    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-afd85c-5c4cfc56c4-jw45m"
Sep 11 19:18:45.257298 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:45.257240    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-afd85c-5c4cfc56c4-jw45m" podStartSLOduration=1.257222523 podStartE2EDuration="1.257222523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:18:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:18:45.256324479 +0000 UTC m=+1801.333915054" watchObservedRunningTime="2026-09-11 19:18:45.257222523 +0000 UTC m=+1801.334813077"
Sep 11 19:18:56.249708 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:56.249673    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-afd85c-5c4cfc56c4-jw45m"
Sep 11 19:18:59.323466 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:59.323434    2578 generic.go:356] "Generic (PLEG): container finished" podID="2bfe0225-fb2e-4616-ba97-26f83d861f1e" containerID="5dfe84568a7a65e4b15aaa36e17306820cd07f47df78c65e6965a0103aadf2b6" exitCode=0
Sep 11 19:18:59.323914 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:59.323515    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-afd85c-openshift-default-7d69487dfd-hvsgv" event={"ID":"2bfe0225-fb2e-4616-ba97-26f83d861f1e","Type":"ContainerDied","Data":"5dfe84568a7a65e4b15aaa36e17306820cd07f47df78c65e6965a0103aadf2b6"}
Sep 11 19:18:59.324019 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:59.324001    2578 scope.go:117] "RemoveContainer" containerID="5dfe84568a7a65e4b15aaa36e17306820cd07f47df78c65e6965a0103aadf2b6"
Sep 11 19:18:59.325085 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:59.325058    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 19:18:59.325138 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:59.325129    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 19:18:59.325174 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:18:59.325159    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 19:19:00.331072 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:19:00.331038    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-afd85c-openshift-default-7d69487dfd-hvsgv" event={"ID":"2bfe0225-fb2e-4616-ba97-26f83d861f1e","Type":"ContainerStarted","Data":"fdb611210da3b260d6c75d7a2bc1a2aa8045d40d4f827e25e9caf19240c67775"}
Sep 11 19:19:01.202154 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:19:01.202109    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-afd85c-openshift-default-7d69487dfd-hvsgv"
Sep 11 19:19:01.202357 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:19:01.202316    2578 patch_prober.go:28] interesting pod/e2e-ipp-b-afd85c-openshift-default-7d69487dfd-hvsgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Sep 11 19:19:01.202407 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:19:01.202365    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-afd85c-openshift-default-7d69487dfd-hvsgv" podUID="2bfe0225-fb2e-4616-ba97-26f83d861f1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Sep 11 19:19:02.202517 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:19:02.202479    2578 patch_prober.go:28] interesting pod/e2e-ipp-b-afd85c-openshift-default-7d69487dfd-hvsgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Sep 11 19:19:02.202960 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:19:02.202539    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-afd85c-openshift-default-7d69487dfd-hvsgv" podUID="2bfe0225-fb2e-4616-ba97-26f83d861f1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Sep 11 19:19:03.202764 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:19:03.202720    2578 patch_prober.go:28] interesting pod/e2e-ipp-b-afd85c-openshift-default-7d69487dfd-hvsgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Sep 11 19:19:03.203245 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:19:03.202788    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-afd85c-openshift-default-7d69487dfd-hvsgv" podUID="2bfe0225-fb2e-4616-ba97-26f83d861f1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Sep 11 19:19:04.202588 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:19:04.202545    2578 patch_prober.go:28] interesting pod/e2e-ipp-b-afd85c-openshift-default-7d69487dfd-hvsgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Sep 11 19:19:04.202852 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:19:04.202640    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-afd85c-openshift-default-7d69487dfd-hvsgv" podUID="2bfe0225-fb2e-4616-ba97-26f83d861f1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Sep 11 19:19:05.202168 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:19:05.202122    2578 patch_prober.go:28] interesting pod/e2e-ipp-b-afd85c-openshift-default-7d69487dfd-hvsgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Sep 11 19:19:05.202380 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:19:05.202189    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-afd85c-openshift-default-7d69487dfd-hvsgv" podUID="2bfe0225-fb2e-4616-ba97-26f83d861f1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Sep 11 19:19:06.206343 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:19:06.206314    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-afd85c-openshift-default-7d69487dfd-hvsgv"
Sep 11 19:19:06.206757 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:19:06.206619    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-afd85c-openshift-default-7d69487dfd-hvsgv"
Sep 11 19:19:06.207254 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:19:06.207237    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-afd85c-openshift-default-7d69487dfd-hvsgv"
Sep 11 19:19:11.063369 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:19:11.063336    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-pm7kk_1edf23d3-5888-4b32-9809-066efe3155c6/payload-processing/0.log"
Sep 11 19:19:24.031785 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:19:24.031745    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-021613/ipp-route-021613-kserve-d76b9655-qtpvx"]
Sep 11 19:19:24.034285 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:19:24.034263    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-021613/ipp-route-021613-kserve-d76b9655-qtpvx"
Sep 11 19:19:24.037449 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:19:24.037427    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-021613\"/\"openshift-service-ca.crt\""
Sep 11 19:19:24.037646 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:19:24.037441    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-021613\"/\"kube-root-ca.crt\""
Sep 11 19:19:24.037784 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:19:24.037450    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-021613\"/\"ipp-route-021613-kserve-self-signed-certs\""
Sep 11 19:19:24.037784 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:19:24.037477    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-021613\"/\"default-dockercfg-9jl6c\""
Sep 11 19:19:24.044420 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:19:24.044393    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-021613/ipp-route-021613-kserve-d76b9655-qtpvx"]
Sep 11 19:19:24.126331 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:19:24.126296    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afeba088-3a12-4810-9f54-c531f3cefdfe-dshm\") pod \"ipp-route-021613-kserve-d76b9655-qtpvx\" (UID: \"afeba088-3a12-4810-9f54-c531f3cefdfe\") " pod="ai-tenant-e2e-ipp-a-021613/ipp-route-021613-kserve-d76b9655-qtpvx"
Sep 11 19:19:24.126526 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:19:24.126338    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afeba088-3a12-4810-9f54-c531f3cefdfe-home\") pod \"ipp-route-021613-kserve-d76b9655-qtpvx\" (UID: \"afeba088-3a12-4810-9f54-c531f3cefdfe\") " pod="ai-tenant-e2e-ipp-a-021613/ipp-route-021613-kserve-d76b9655-qtpvx"
Sep 11 19:19:24.126526 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:19:24.126437    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afeba088-3a12-4810-9f54-c531f3cefdfe-tmp-dir\") pod \"ipp-route-021613-kserve-d76b9655-qtpvx\" (UID: \"afeba088-3a12-4810-9f54-c531f3cefdfe\") " pod="ai-tenant-e2e-ipp-a-021613/ipp-route-021613-kserve-d76b9655-qtpvx"
Sep 11 19:19:24.126526 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:19:24.126474    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afeba088-3a12-4810-9f54-c531f3cefdfe-tls-certs\") pod \"ipp-route-021613-kserve-d76b9655-qtpvx\" (UID: \"afeba088-3a12-4810-9f54-c531f3cefdfe\") " pod="ai-tenant-e2e-ipp-a-021613/ipp-route-021613-kserve-d76b9655-qtpvx"
Sep 11 19:19:24.126720 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:19:24.126554    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksfsh\" (UniqueName: \"kubernetes.io/projected/afeba088-3a12-4810-9f54-c531f3cefdfe-kube-api-access-ksfsh\") pod \"ipp-route-021613-kserve-d76b9655-qtpvx\" (UID: \"afeba088-3a12-4810-9f54-c531f3cefdfe\") " pod="ai-tenant-e2e-ipp-a-021613/ipp-route-021613-kserve-d76b9655-qtpvx"
Sep 11 19:19:24.126720 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:19:24.126639    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afeba088-3a12-4810-9f54-c531f3cefdfe-model-cache\") pod \"ipp-route-021613-kserve-d76b9655-qtpvx\" (UID: \"afeba088-3a12-4810-9f54-c531f3cefdfe\") " pod="ai-tenant-e2e-ipp-a-021613/ipp-route-021613-kserve-d76b9655-qtpvx"
Sep 11 19:19:24.227969 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:19:24.227925    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afeba088-3a12-4810-9f54-c531f3cefdfe-model-cache\") pod \"ipp-route-021613-kserve-d76b9655-qtpvx\" (UID: \"afeba088-3a12-4810-9f54-c531f3cefdfe\") " pod="ai-tenant-e2e-ipp-a-021613/ipp-route-021613-kserve-d76b9655-qtpvx"
Sep 11 19:19:24.228188 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:19:24.227982    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afeba088-3a12-4810-9f54-c531f3cefdfe-dshm\") pod \"ipp-route-021613-kserve-d76b9655-qtpvx\" (UID: \"afeba088-3a12-4810-9f54-c531f3cefdfe\") " pod="ai-tenant-e2e-ipp-a-021613/ipp-route-021613-kserve-d76b9655-qtpvx"
Sep 11 19:19:24.228188 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:19:24.228003    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afeba088-3a12-4810-9f54-c531f3cefdfe-home\") pod \"ipp-route-021613-kserve-d76b9655-qtpvx\" (UID: \"afeba088-3a12-4810-9f54-c531f3cefdfe\") " pod="ai-tenant-e2e-ipp-a-021613/ipp-route-021613-kserve-d76b9655-qtpvx"
Sep 11 19:19:24.228188 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:19:24.228063    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afeba088-3a12-4810-9f54-c531f3cefdfe-tmp-dir\") pod \"ipp-route-021613-kserve-d76b9655-qtpvx\" (UID: \"afeba088-3a12-4810-9f54-c531f3cefdfe\") " pod="ai-tenant-e2e-ipp-a-021613/ipp-route-021613-kserve-d76b9655-qtpvx"
Sep 11 19:19:24.228188 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:19:24.228090    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afeba088-3a12-4810-9f54-c531f3cefdfe-tls-certs\") pod \"ipp-route-021613-kserve-d76b9655-qtpvx\" (UID: \"afeba088-3a12-4810-9f54-c531f3cefdfe\") " pod="ai-tenant-e2e-ipp-a-021613/ipp-route-021613-kserve-d76b9655-qtpvx"
Sep 11 19:19:24.228188 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:19:24.228138    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksfsh\" (UniqueName: \"kubernetes.io/projected/afeba088-3a12-4810-9f54-c531f3cefdfe-kube-api-access-ksfsh\") pod \"ipp-route-021613-kserve-d76b9655-qtpvx\" (UID: \"afeba088-3a12-4810-9f54-c531f3cefdfe\") " pod="ai-tenant-e2e-ipp-a-021613/ipp-route-021613-kserve-d76b9655-qtpvx"
Sep 11 19:19:24.228466 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:19:24.228432    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afeba088-3a12-4810-9f54-c531f3cefdfe-model-cache\") pod \"ipp-route-021613-kserve-d76b9655-qtpvx\" (UID: \"afeba088-3a12-4810-9f54-c531f3cefdfe\") " pod="ai-tenant-e2e-ipp-a-021613/ipp-route-021613-kserve-d76b9655-qtpvx"
Sep 11 19:19:24.228512 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:19:24.228464    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afeba088-3a12-4810-9f54-c531f3cefdfe-home\") pod \"ipp-route-021613-kserve-d76b9655-qtpvx\" (UID: \"afeba088-3a12-4810-9f54-c531f3cefdfe\") " pod="ai-tenant-e2e-ipp-a-021613/ipp-route-021613-kserve-d76b9655-qtpvx"
Sep 11 19:19:24.228512 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:19:24.228472    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afeba088-3a12-4810-9f54-c531f3cefdfe-tmp-dir\") pod \"ipp-route-021613-kserve-d76b9655-qtpvx\" (UID: \"afeba088-3a12-4810-9f54-c531f3cefdfe\") " pod="ai-tenant-e2e-ipp-a-021613/ipp-route-021613-kserve-d76b9655-qtpvx"
Sep 11 19:19:24.230708 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:19:24.230685    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afeba088-3a12-4810-9f54-c531f3cefdfe-dshm\") pod \"ipp-route-021613-kserve-d76b9655-qtpvx\" (UID: \"afeba088-3a12-4810-9f54-c531f3cefdfe\") " pod="ai-tenant-e2e-ipp-a-021613/ipp-route-021613-kserve-d76b9655-qtpvx"
Sep 11 19:19:24.231018 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:19:24.230994    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afeba088-3a12-4810-9f54-c531f3cefdfe-tls-certs\") pod \"ipp-route-021613-kserve-d76b9655-qtpvx\" (UID: \"afeba088-3a12-4810-9f54-c531f3cefdfe\") " pod="ai-tenant-e2e-ipp-a-021613/ipp-route-021613-kserve-d76b9655-qtpvx"
Sep 11 19:19:24.235413 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:19:24.235383    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksfsh\" (UniqueName: \"kubernetes.io/projected/afeba088-3a12-4810-9f54-c531f3cefdfe-kube-api-access-ksfsh\") pod \"ipp-route-021613-kserve-d76b9655-qtpvx\" (UID: \"afeba088-3a12-4810-9f54-c531f3cefdfe\") " pod="ai-tenant-e2e-ipp-a-021613/ipp-route-021613-kserve-d76b9655-qtpvx"
Sep 11 19:19:24.344661 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:19:24.344539    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-021613/ipp-route-021613-kserve-d76b9655-qtpvx"
Sep 11 19:19:24.617073 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:19:24.617045    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-021613/ipp-route-021613-kserve-d76b9655-qtpvx"]
Sep 11 19:19:24.619444 ip-10-0-134-189 kubenswrapper[2578]: W0911 19:19:24.619416    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podafeba088_3a12_4810_9f54_c531f3cefdfe.slice/crio-fc14cdfaa2033eccf1f315adfbc6c2acefec6f7f83a03c887e2af68faa62a007 WatchSource:0}: Error finding container fc14cdfaa2033eccf1f315adfbc6c2acefec6f7f83a03c887e2af68faa62a007: Status 404 returned error can't find the container with id fc14cdfaa2033eccf1f315adfbc6c2acefec6f7f83a03c887e2af68faa62a007
Sep 11 19:19:25.462235 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:19:25.462198    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-021613/ipp-route-021613-kserve-d76b9655-qtpvx" event={"ID":"afeba088-3a12-4810-9f54-c531f3cefdfe","Type":"ContainerStarted","Data":"aa7ba998f8fc183c0fdc53d11e47eb00ba30814685455e9897622e57e5752693"}
Sep 11 19:19:25.462235 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:19:25.462243    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-021613/ipp-route-021613-kserve-d76b9655-qtpvx" event={"ID":"afeba088-3a12-4810-9f54-c531f3cefdfe","Type":"ContainerStarted","Data":"fc14cdfaa2033eccf1f315adfbc6c2acefec6f7f83a03c887e2af68faa62a007"}
Sep 11 19:19:25.478997 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:19:25.478932    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-021613/ipp-route-021613-kserve-d76b9655-qtpvx" podStartSLOduration=1.478908125 podStartE2EDuration="1.478908125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:19:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:19:25.477202789 +0000 UTC m=+1841.554793373" watchObservedRunningTime="2026-09-11 19:19:25.478908125 +0000 UTC m=+1841.556498681"
Sep 11 19:19:34.345131 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:19:34.345090    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-021613/ipp-route-021613-kserve-d76b9655-qtpvx"
Sep 11 19:19:34.345765 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:19:34.345168    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-021613/ipp-route-021613-kserve-d76b9655-qtpvx"
Sep 11 19:19:34.357878 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:19:34.357853    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-021613/ipp-route-021613-kserve-d76b9655-qtpvx"
Sep 11 19:19:34.524762 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:19:34.524718    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-021613/ipp-route-021613-kserve-d76b9655-qtpvx"
Sep 11 19:19:44.733288 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:19:44.733254    2578 scope.go:117] "RemoveContainer" containerID="75f0ef66ac0d888aff1c36d5ecedbedd976887df2752f8694e17698a6f78690b"
Sep 11 19:19:47.098726 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:19:47.098688    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-pm7kk_1edf23d3-5888-4b32-9809-066efe3155c6/payload-processing/0.log"
Sep 11 19:19:55.091618 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:19:55.091556    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-75b1ed-65b646cd54-q6zkf"]
Sep 11 19:19:55.094458 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:19:55.094440    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-75b1ed-65b646cd54-q6zkf"
Sep 11 19:19:55.096884 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:19:55.096860    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-75b1ed\""
Sep 11 19:19:55.097021 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:19:55.096967    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-75b1ed-dockercfg-952f8\""
Sep 11 19:19:55.108119 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:19:55.108088    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-75b1ed-57b8fff9f-fj4fl"]
Sep 11 19:19:55.111528 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:19:55.111505    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-75b1ed-65b646cd54-q6zkf"]
Sep 11 19:19:55.111649 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:19:55.111632    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-75b1ed-57b8fff9f-fj4fl"
Sep 11 19:19:55.125904 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:19:55.125870    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-75b1ed-57b8fff9f-fj4fl"]
Sep 11 19:19:55.237225 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:19:55.237183    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55fe8bc7-80cf-4448-9722-0f0e3114dfc4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-75b1ed-65b646cd54-q6zkf\" (UID: \"55fe8bc7-80cf-4448-9722-0f0e3114dfc4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-75b1ed-65b646cd54-q6zkf"
Sep 11 19:19:55.237409 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:19:55.237251    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/823effa9-68b5-4cea-986a-356c73787df4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-75b1ed-57b8fff9f-fj4fl\" (UID: \"823effa9-68b5-4cea-986a-356c73787df4\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-75b1ed-57b8fff9f-fj4fl"
Sep 11 19:19:55.237409 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:19:55.237272    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p58lp\" (UniqueName: \"kubernetes.io/projected/55fe8bc7-80cf-4448-9722-0f0e3114dfc4-kube-api-access-p58lp\") pod \"payload-pre-processing-e2e-ipp-cleanup-75b1ed-65b646cd54-q6zkf\" (UID: \"55fe8bc7-80cf-4448-9722-0f0e3114dfc4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-75b1ed-65b646cd54-q6zkf"
Sep 11 19:19:55.237409 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:19:55.237292    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4plz2\" (UniqueName: \"kubernetes.io/projected/823effa9-68b5-4cea-986a-356c73787df4-kube-api-access-4plz2\") pod \"payload-processing-e2e-ipp-cleanup-75b1ed-57b8fff9f-fj4fl\" (UID: \"823effa9-68b5-4cea-986a-356c73787df4\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-75b1ed-57b8fff9f-fj4fl"
Sep 11 19:19:55.337834 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:19:55.337780    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55fe8bc7-80cf-4448-9722-0f0e3114dfc4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-75b1ed-65b646cd54-q6zkf\" (UID: \"55fe8bc7-80cf-4448-9722-0f0e3114dfc4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-75b1ed-65b646cd54-q6zkf"
Sep 11 19:19:55.338027 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:19:55.337878    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/823effa9-68b5-4cea-986a-356c73787df4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-75b1ed-57b8fff9f-fj4fl\" (UID: \"823effa9-68b5-4cea-986a-356c73787df4\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-75b1ed-57b8fff9f-fj4fl"
Sep 11 19:19:55.338027 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:19:55.337900    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p58lp\" (UniqueName: \"kubernetes.io/projected/55fe8bc7-80cf-4448-9722-0f0e3114dfc4-kube-api-access-p58lp\") pod \"payload-pre-processing-e2e-ipp-cleanup-75b1ed-65b646cd54-q6zkf\" (UID: \"55fe8bc7-80cf-4448-9722-0f0e3114dfc4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-75b1ed-65b646cd54-q6zkf"
Sep 11 19:19:55.338027 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:19:55.337921    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4plz2\" (UniqueName: \"kubernetes.io/projected/823effa9-68b5-4cea-986a-356c73787df4-kube-api-access-4plz2\") pod \"payload-processing-e2e-ipp-cleanup-75b1ed-57b8fff9f-fj4fl\" (UID: \"823effa9-68b5-4cea-986a-356c73787df4\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-75b1ed-57b8fff9f-fj4fl"
Sep 11 19:19:55.338639 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:19:55.338582    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/823effa9-68b5-4cea-986a-356c73787df4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-75b1ed-57b8fff9f-fj4fl\" (UID: \"823effa9-68b5-4cea-986a-356c73787df4\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-75b1ed-57b8fff9f-fj4fl"
Sep 11 19:19:55.338694 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:19:55.338646    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55fe8bc7-80cf-4448-9722-0f0e3114dfc4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-75b1ed-65b646cd54-q6zkf\" (UID: \"55fe8bc7-80cf-4448-9722-0f0e3114dfc4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-75b1ed-65b646cd54-q6zkf"
Sep 11 19:19:55.346545 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:19:55.346476    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p58lp\" (UniqueName: \"kubernetes.io/projected/55fe8bc7-80cf-4448-9722-0f0e3114dfc4-kube-api-access-p58lp\") pod \"payload-pre-processing-e2e-ipp-cleanup-75b1ed-65b646cd54-q6zkf\" (UID: \"55fe8bc7-80cf-4448-9722-0f0e3114dfc4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-75b1ed-65b646cd54-q6zkf"
Sep 11 19:19:55.346706 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:19:55.346556    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4plz2\" (UniqueName: \"kubernetes.io/projected/823effa9-68b5-4cea-986a-356c73787df4-kube-api-access-4plz2\") pod \"payload-processing-e2e-ipp-cleanup-75b1ed-57b8fff9f-fj4fl\" (UID: \"823effa9-68b5-4cea-986a-356c73787df4\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-75b1ed-57b8fff9f-fj4fl"
Sep 11 19:19:55.405171 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:19:55.405130    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-75b1ed-65b646cd54-q6zkf"
Sep 11 19:19:55.422023 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:19:55.421990    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-75b1ed-57b8fff9f-fj4fl"
Sep 11 19:19:55.684214 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:19:55.684169    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-75b1ed-65b646cd54-q6zkf"]
Sep 11 19:19:55.686389 ip-10-0-134-189 kubenswrapper[2578]: W0911 19:19:55.686354    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55fe8bc7_80cf_4448_9722_0f0e3114dfc4.slice/crio-e0e454f682493e703bbd5f50a78937fa961c8ec1528d7c6005fda75a61247add WatchSource:0}: Error finding container e0e454f682493e703bbd5f50a78937fa961c8ec1528d7c6005fda75a61247add: Status 404 returned error can't find the container with id e0e454f682493e703bbd5f50a78937fa961c8ec1528d7c6005fda75a61247add
Sep 11 19:19:55.713576 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:19:55.713553    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-75b1ed-57b8fff9f-fj4fl"]
Sep 11 19:19:55.720703 ip-10-0-134-189 kubenswrapper[2578]: W0911 19:19:55.720679    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod823effa9_68b5_4cea_986a_356c73787df4.slice/crio-dcd8ece744671881905ddfe9774dfa5595e7729d1499b10cd6b5ff0bc48329cb WatchSource:0}: Error finding container dcd8ece744671881905ddfe9774dfa5595e7729d1499b10cd6b5ff0bc48329cb: Status 404 returned error can't find the container with id dcd8ece744671881905ddfe9774dfa5595e7729d1499b10cd6b5ff0bc48329cb
Sep 11 19:19:56.620367 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:19:56.620326    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-75b1ed-57b8fff9f-fj4fl" event={"ID":"823effa9-68b5-4cea-986a-356c73787df4","Type":"ContainerStarted","Data":"59d0bbb307116273ac45014cdf67956013e8e62e086a37701999cb46534121d2"}
Sep 11 19:19:56.620848 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:19:56.620395    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-75b1ed-57b8fff9f-fj4fl" event={"ID":"823effa9-68b5-4cea-986a-356c73787df4","Type":"ContainerStarted","Data":"dcd8ece744671881905ddfe9774dfa5595e7729d1499b10cd6b5ff0bc48329cb"}
Sep 11 19:19:56.620848 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:19:56.620427    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-75b1ed-57b8fff9f-fj4fl"
Sep 11 19:19:56.622076 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:19:56.622052    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-75b1ed-65b646cd54-q6zkf" event={"ID":"55fe8bc7-80cf-4448-9722-0f0e3114dfc4","Type":"ContainerStarted","Data":"4742c8f5b43f26ccd3958b7da58009d18eadcee2b949ed0ca8dc8b0f9f9ca3dd"}
Sep 11 19:19:56.622214 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:19:56.622097    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-75b1ed-65b646cd54-q6zkf" event={"ID":"55fe8bc7-80cf-4448-9722-0f0e3114dfc4","Type":"ContainerStarted","Data":"e0e454f682493e703bbd5f50a78937fa961c8ec1528d7c6005fda75a61247add"}
Sep 11 19:19:56.622214 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:19:56.622168    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-75b1ed-65b646cd54-q6zkf"
Sep 11 19:19:56.649022 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:19:56.648972    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-75b1ed-57b8fff9f-fj4fl" podStartSLOduration=1.6489535690000001 podStartE2EDuration="1.648953569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:19:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:19:56.647518685 +0000 UTC m=+1872.725109262" watchObservedRunningTime="2026-09-11 19:19:56.648953569 +0000 UTC m=+1872.726544173"
Sep 11 19:19:56.666487 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:19:56.666430    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-75b1ed-65b646cd54-q6zkf" podStartSLOduration=1.666404933 podStartE2EDuration="1.666404933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:19:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:19:56.666276218 +0000 UTC m=+1872.743866795" watchObservedRunningTime="2026-09-11 19:19:56.666404933 +0000 UTC m=+1872.743995491"
Sep 11 19:20:07.630321 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:07.630240    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-75b1ed-57b8fff9f-fj4fl"
Sep 11 19:20:07.630795 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:07.630322    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-75b1ed-65b646cd54-q6zkf"
Sep 11 19:20:21.404506 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:21.404469    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-75b1ed-bbzrs"]
Sep 11 19:20:21.407825 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:21.407807    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-75b1ed-bbzrs"
Sep 11 19:20:21.410050 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:21.410022    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 19:20:21.410491 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:21.410474    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ln295\""
Sep 11 19:20:21.417717 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:21.417687    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-75b1ed-bbzrs"]
Sep 11 19:20:21.483635 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:21.483559    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/628cdb4a-8717-4323-9c0b-9c8f58c05f6c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-75b1ed-bbzrs\" (UID: \"628cdb4a-8717-4323-9c0b-9c8f58c05f6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-75b1ed-bbzrs"
Sep 11 19:20:21.584857 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:21.584816    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/628cdb4a-8717-4323-9c0b-9c8f58c05f6c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-75b1ed-bbzrs\" (UID: \"628cdb4a-8717-4323-9c0b-9c8f58c05f6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-75b1ed-bbzrs"
Sep 11 19:20:21.585505 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:21.585483    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/628cdb4a-8717-4323-9c0b-9c8f58c05f6c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-75b1ed-bbzrs\" (UID: \"628cdb4a-8717-4323-9c0b-9c8f58c05f6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-75b1ed-bbzrs"
Sep 11 19:20:21.720455 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:21.720418    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-75b1ed-bbzrs"
Sep 11 19:20:22.001408 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:22.001384    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-75b1ed-bbzrs"]
Sep 11 19:20:22.003562 ip-10-0-134-189 kubenswrapper[2578]: W0911 19:20:22.003534    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod628cdb4a_8717_4323_9c0b_9c8f58c05f6c.slice/crio-4feaa175211ae12817408dfcb0e937dbcf5529e8edc1b43b3da80edb17cd2c34 WatchSource:0}: Error finding container 4feaa175211ae12817408dfcb0e937dbcf5529e8edc1b43b3da80edb17cd2c34: Status 404 returned error can't find the container with id 4feaa175211ae12817408dfcb0e937dbcf5529e8edc1b43b3da80edb17cd2c34
Sep 11 19:20:22.764371 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:22.764286    2578 generic.go:356] "Generic (PLEG): container finished" podID="628cdb4a-8717-4323-9c0b-9c8f58c05f6c" containerID="88882d904f2de2effb2af1e6cc53146357d1b29005d84c3fd25673b76e8125e8" exitCode=0
Sep 11 19:20:22.764772 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:22.764381    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-75b1ed-bbzrs" event={"ID":"628cdb4a-8717-4323-9c0b-9c8f58c05f6c","Type":"ContainerDied","Data":"88882d904f2de2effb2af1e6cc53146357d1b29005d84c3fd25673b76e8125e8"}
Sep 11 19:20:22.764772 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:22.764420    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-75b1ed-bbzrs" event={"ID":"628cdb4a-8717-4323-9c0b-9c8f58c05f6c","Type":"ContainerStarted","Data":"4feaa175211ae12817408dfcb0e937dbcf5529e8edc1b43b3da80edb17cd2c34"}
Sep 11 19:20:23.910506 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:23.910479    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-75b1ed-bbzrs"
Sep 11 19:20:24.008199 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:24.008159    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/628cdb4a-8717-4323-9c0b-9c8f58c05f6c-maas-api-service-ca\") pod \"628cdb4a-8717-4323-9c0b-9c8f58c05f6c\" (UID: \"628cdb4a-8717-4323-9c0b-9c8f58c05f6c\") "
Sep 11 19:20:24.008546 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:24.008517    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/628cdb4a-8717-4323-9c0b-9c8f58c05f6c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "628cdb4a-8717-4323-9c0b-9c8f58c05f6c" (UID: "628cdb4a-8717-4323-9c0b-9c8f58c05f6c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 19:20:24.109040 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:24.108962    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/628cdb4a-8717-4323-9c0b-9c8f58c05f6c-maas-api-service-ca\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:20:24.777353 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:24.777323    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-75b1ed-bbzrs"
Sep 11 19:20:24.777541 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:24.777360    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-75b1ed-bbzrs" event={"ID":"628cdb4a-8717-4323-9c0b-9c8f58c05f6c","Type":"ContainerDied","Data":"4feaa175211ae12817408dfcb0e937dbcf5529e8edc1b43b3da80edb17cd2c34"}
Sep 11 19:20:24.777541 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:24.777406    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4feaa175211ae12817408dfcb0e937dbcf5529e8edc1b43b3da80edb17cd2c34"
Sep 11 19:20:32.582619 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:32.582555    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-75b1ed-57b8fff9f-fj4fl"]
Sep 11 19:20:32.583238 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:32.582881    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-75b1ed-57b8fff9f-fj4fl" podUID="823effa9-68b5-4cea-986a-356c73787df4" containerName="payload-processing" containerID="cri-o://59d0bbb307116273ac45014cdf67956013e8e62e086a37701999cb46534121d2" gracePeriod=30
Sep 11 19:20:32.585864 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:32.585831    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-75b1ed-65b646cd54-q6zkf"]
Sep 11 19:20:32.586122 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:32.586077    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-75b1ed-65b646cd54-q6zkf" podUID="55fe8bc7-80cf-4448-9722-0f0e3114dfc4" containerName="payload-pre-processing" containerID="cri-o://4742c8f5b43f26ccd3958b7da58009d18eadcee2b949ed0ca8dc8b0f9f9ca3dd" gracePeriod=30
Sep 11 19:20:32.762232 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:32.762199    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-75b1ed-bbzrs"]
Sep 11 19:20:32.766298 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:32.766270    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-75b1ed-bbzrs"]
Sep 11 19:20:32.820482 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:32.820447    2578 generic.go:356] "Generic (PLEG): container finished" podID="55fe8bc7-80cf-4448-9722-0f0e3114dfc4" containerID="4742c8f5b43f26ccd3958b7da58009d18eadcee2b949ed0ca8dc8b0f9f9ca3dd" exitCode=0
Sep 11 19:20:32.820698 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:32.820516    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-75b1ed-65b646cd54-q6zkf" event={"ID":"55fe8bc7-80cf-4448-9722-0f0e3114dfc4","Type":"ContainerDied","Data":"4742c8f5b43f26ccd3958b7da58009d18eadcee2b949ed0ca8dc8b0f9f9ca3dd"}
Sep 11 19:20:32.822251 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:32.822223    2578 generic.go:356] "Generic (PLEG): container finished" podID="823effa9-68b5-4cea-986a-356c73787df4" containerID="59d0bbb307116273ac45014cdf67956013e8e62e086a37701999cb46534121d2" exitCode=0
Sep 11 19:20:32.822396 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:32.822295    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-75b1ed-57b8fff9f-fj4fl" event={"ID":"823effa9-68b5-4cea-986a-356c73787df4","Type":"ContainerDied","Data":"59d0bbb307116273ac45014cdf67956013e8e62e086a37701999cb46534121d2"}
Sep 11 19:20:32.864391 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:32.864362    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-75b1ed-57b8fff9f-fj4fl"
Sep 11 19:20:32.867511 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:32.867489    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-75b1ed-65b646cd54-q6zkf"
Sep 11 19:20:33.000629 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:33.000562    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4plz2\" (UniqueName: \"kubernetes.io/projected/823effa9-68b5-4cea-986a-356c73787df4-kube-api-access-4plz2\") pod \"823effa9-68b5-4cea-986a-356c73787df4\" (UID: \"823effa9-68b5-4cea-986a-356c73787df4\") "
Sep 11 19:20:33.000629 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:33.000634    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55fe8bc7-80cf-4448-9722-0f0e3114dfc4-plugins-config-volume\") pod \"55fe8bc7-80cf-4448-9722-0f0e3114dfc4\" (UID: \"55fe8bc7-80cf-4448-9722-0f0e3114dfc4\") "
Sep 11 19:20:33.000908 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:33.000735    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/823effa9-68b5-4cea-986a-356c73787df4-plugins-config-volume\") pod \"823effa9-68b5-4cea-986a-356c73787df4\" (UID: \"823effa9-68b5-4cea-986a-356c73787df4\") "
Sep 11 19:20:33.000908 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:33.000786    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p58lp\" (UniqueName: \"kubernetes.io/projected/55fe8bc7-80cf-4448-9722-0f0e3114dfc4-kube-api-access-p58lp\") pod \"55fe8bc7-80cf-4448-9722-0f0e3114dfc4\" (UID: \"55fe8bc7-80cf-4448-9722-0f0e3114dfc4\") "
Sep 11 19:20:33.001078 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:33.001050    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55fe8bc7-80cf-4448-9722-0f0e3114dfc4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "55fe8bc7-80cf-4448-9722-0f0e3114dfc4" (UID: "55fe8bc7-80cf-4448-9722-0f0e3114dfc4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 19:20:33.001169 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:33.001144    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/823effa9-68b5-4cea-986a-356c73787df4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "823effa9-68b5-4cea-986a-356c73787df4" (UID: "823effa9-68b5-4cea-986a-356c73787df4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 19:20:33.002986 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:33.002962    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/823effa9-68b5-4cea-986a-356c73787df4-kube-api-access-4plz2" (OuterVolumeSpecName: "kube-api-access-4plz2") pod "823effa9-68b5-4cea-986a-356c73787df4" (UID: "823effa9-68b5-4cea-986a-356c73787df4"). InnerVolumeSpecName "kube-api-access-4plz2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:20:33.003082 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:33.003067    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55fe8bc7-80cf-4448-9722-0f0e3114dfc4-kube-api-access-p58lp" (OuterVolumeSpecName: "kube-api-access-p58lp") pod "55fe8bc7-80cf-4448-9722-0f0e3114dfc4" (UID: "55fe8bc7-80cf-4448-9722-0f0e3114dfc4"). InnerVolumeSpecName "kube-api-access-p58lp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:20:33.102096 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:33.101997    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/823effa9-68b5-4cea-986a-356c73787df4-plugins-config-volume\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:20:33.102096 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:33.102031    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p58lp\" (UniqueName: \"kubernetes.io/projected/55fe8bc7-80cf-4448-9722-0f0e3114dfc4-kube-api-access-p58lp\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:20:33.102096 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:33.102044    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4plz2\" (UniqueName: \"kubernetes.io/projected/823effa9-68b5-4cea-986a-356c73787df4-kube-api-access-4plz2\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:20:33.102096 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:33.102057    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55fe8bc7-80cf-4448-9722-0f0e3114dfc4-plugins-config-volume\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:20:33.828675 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:33.828639    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-75b1ed-65b646cd54-q6zkf"
Sep 11 19:20:33.828675 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:33.828659    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-75b1ed-65b646cd54-q6zkf" event={"ID":"55fe8bc7-80cf-4448-9722-0f0e3114dfc4","Type":"ContainerDied","Data":"e0e454f682493e703bbd5f50a78937fa961c8ec1528d7c6005fda75a61247add"}
Sep 11 19:20:33.829236 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:33.828723    2578 scope.go:117] "RemoveContainer" containerID="4742c8f5b43f26ccd3958b7da58009d18eadcee2b949ed0ca8dc8b0f9f9ca3dd"
Sep 11 19:20:33.830055 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:33.830037    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-75b1ed-57b8fff9f-fj4fl" event={"ID":"823effa9-68b5-4cea-986a-356c73787df4","Type":"ContainerDied","Data":"dcd8ece744671881905ddfe9774dfa5595e7729d1499b10cd6b5ff0bc48329cb"}
Sep 11 19:20:33.830124 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:33.830047    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-75b1ed-57b8fff9f-fj4fl"
Sep 11 19:20:33.838588 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:33.838571    2578 scope.go:117] "RemoveContainer" containerID="59d0bbb307116273ac45014cdf67956013e8e62e086a37701999cb46534121d2"
Sep 11 19:20:33.853239 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:33.853177    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-75b1ed-65b646cd54-q6zkf"]
Sep 11 19:20:33.856542 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:33.856511    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-75b1ed-65b646cd54-q6zkf"]
Sep 11 19:20:33.865230 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:33.865205    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-75b1ed-57b8fff9f-fj4fl"]
Sep 11 19:20:33.869160 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:33.869136    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-75b1ed-57b8fff9f-fj4fl"]
Sep 11 19:20:34.397159 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:34.397123    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55fe8bc7-80cf-4448-9722-0f0e3114dfc4" path="/var/lib/kubelet/pods/55fe8bc7-80cf-4448-9722-0f0e3114dfc4/volumes"
Sep 11 19:20:34.397549 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:34.397529    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="628cdb4a-8717-4323-9c0b-9c8f58c05f6c" path="/var/lib/kubelet/pods/628cdb4a-8717-4323-9c0b-9c8f58c05f6c/volumes"
Sep 11 19:20:34.397952 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:34.397933    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="823effa9-68b5-4cea-986a-356c73787df4" path="/var/lib/kubelet/pods/823effa9-68b5-4cea-986a-356c73787df4/volumes"
Sep 11 19:20:42.477173 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:42.477137    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-021613-485bh"]
Sep 11 19:20:42.477792 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:42.477658    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="628cdb4a-8717-4323-9c0b-9c8f58c05f6c" containerName="revoke-keys"
Sep 11 19:20:42.477792 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:42.477678    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="628cdb4a-8717-4323-9c0b-9c8f58c05f6c" containerName="revoke-keys"
Sep 11 19:20:42.477792 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:42.477695    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="823effa9-68b5-4cea-986a-356c73787df4" containerName="payload-processing"
Sep 11 19:20:42.477792 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:42.477705    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="823effa9-68b5-4cea-986a-356c73787df4" containerName="payload-processing"
Sep 11 19:20:42.477792 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:42.477729    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55fe8bc7-80cf-4448-9722-0f0e3114dfc4" containerName="payload-pre-processing"
Sep 11 19:20:42.477792 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:42.477737    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="55fe8bc7-80cf-4448-9722-0f0e3114dfc4" containerName="payload-pre-processing"
Sep 11 19:20:42.478112 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:42.477833    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="55fe8bc7-80cf-4448-9722-0f0e3114dfc4" containerName="payload-pre-processing"
Sep 11 19:20:42.478112 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:42.477848    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="628cdb4a-8717-4323-9c0b-9c8f58c05f6c" containerName="revoke-keys"
Sep 11 19:20:42.478112 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:42.477858    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="823effa9-68b5-4cea-986a-356c73787df4" containerName="payload-processing"
Sep 11 19:20:42.482171 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:42.482146    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-021613-485bh"
Sep 11 19:20:42.484764 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:42.484699    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 19:20:42.485029 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:42.484982    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ln295\""
Sep 11 19:20:42.486566 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:42.486539    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-021613-485bh"]
Sep 11 19:20:42.590737 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:42.590693    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c716675-04b5-4447-9b2b-5f05874c168a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-021613-485bh\" (UID: \"8c716675-04b5-4447-9b2b-5f05874c168a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-021613-485bh"
Sep 11 19:20:42.692019 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:42.691981    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c716675-04b5-4447-9b2b-5f05874c168a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-021613-485bh\" (UID: \"8c716675-04b5-4447-9b2b-5f05874c168a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-021613-485bh"
Sep 11 19:20:42.692645 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:42.692622    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c716675-04b5-4447-9b2b-5f05874c168a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-021613-485bh\" (UID: \"8c716675-04b5-4447-9b2b-5f05874c168a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-021613-485bh"
Sep 11 19:20:42.794203 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:42.794109    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-021613-485bh"
Sep 11 19:20:43.063402 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:43.063337    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-021613-485bh"]
Sep 11 19:20:43.065747 ip-10-0-134-189 kubenswrapper[2578]: W0911 19:20:43.065717    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c716675_04b5_4447_9b2b_5f05874c168a.slice/crio-29eb27bbdcafc79c4e273d19fda89465b063d1f75d03bf33b733cac98af8a5a4 WatchSource:0}: Error finding container 29eb27bbdcafc79c4e273d19fda89465b063d1f75d03bf33b733cac98af8a5a4: Status 404 returned error can't find the container with id 29eb27bbdcafc79c4e273d19fda89465b063d1f75d03bf33b733cac98af8a5a4
Sep 11 19:20:43.884992 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:43.884959    2578 generic.go:356] "Generic (PLEG): container finished" podID="8c716675-04b5-4447-9b2b-5f05874c168a" containerID="b2722492336c270deadf4fa9dc5c1a1c6368e87019d4e900ebbbc135a72973aa" exitCode=0
Sep 11 19:20:43.885412 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:43.885055    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-021613-485bh" event={"ID":"8c716675-04b5-4447-9b2b-5f05874c168a","Type":"ContainerDied","Data":"b2722492336c270deadf4fa9dc5c1a1c6368e87019d4e900ebbbc135a72973aa"}
Sep 11 19:20:43.885412 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:43.885121    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-021613-485bh" event={"ID":"8c716675-04b5-4447-9b2b-5f05874c168a","Type":"ContainerStarted","Data":"29eb27bbdcafc79c4e273d19fda89465b063d1f75d03bf33b733cac98af8a5a4"}
Sep 11 19:20:44.752623 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:44.752561    2578 scope.go:117] "RemoveContainer" containerID="1e014b2897b963f4424a7682fd26436bcbdd145318e57dcae72b38453810a5ab"
Sep 11 19:20:44.761297 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:44.761271    2578 scope.go:117] "RemoveContainer" containerID="dc688307e065dcb247ad0d164e842205d0a4b2e456733f9dd30cb2bed2af93ac"
Sep 11 19:20:44.769436 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:44.769415    2578 scope.go:117] "RemoveContainer" containerID="be6ce4f403a36a7dcb2efd8a002fc1efa6af513c1c5fbafbec0e7c4c0ad460ca"
Sep 11 19:20:45.031174 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:45.031140    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-021613-485bh"
Sep 11 19:20:45.216503 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:45.216467    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c716675-04b5-4447-9b2b-5f05874c168a-maas-api-service-ca\") pod \"8c716675-04b5-4447-9b2b-5f05874c168a\" (UID: \"8c716675-04b5-4447-9b2b-5f05874c168a\") "
Sep 11 19:20:45.216910 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:45.216885    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c716675-04b5-4447-9b2b-5f05874c168a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8c716675-04b5-4447-9b2b-5f05874c168a" (UID: "8c716675-04b5-4447-9b2b-5f05874c168a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 19:20:45.317267 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:45.317170    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c716675-04b5-4447-9b2b-5f05874c168a-maas-api-service-ca\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:20:45.897943 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:45.897906    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-021613-485bh" event={"ID":"8c716675-04b5-4447-9b2b-5f05874c168a","Type":"ContainerDied","Data":"29eb27bbdcafc79c4e273d19fda89465b063d1f75d03bf33b733cac98af8a5a4"}
Sep 11 19:20:45.897943 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:45.897933    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-021613-485bh"
Sep 11 19:20:45.898178 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:45.897958    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29eb27bbdcafc79c4e273d19fda89465b063d1f75d03bf33b733cac98af8a5a4"
Sep 11 19:20:58.199741 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:58.199690    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-021613-7ddc8f8c7d-94nhx"]
Sep 11 19:20:58.200182 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:58.199955    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-021613-7ddc8f8c7d-94nhx" podUID="add780d4-d2a4-4224-9804-7468fedf25ee" containerName="payload-pre-processing" containerID="cri-o://46c6b01797dee03843c15b1ee763659a36c0abcb8280909ad68285a6f5329999" gracePeriod=30
Sep 11 19:20:58.368707 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:58.368674    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-021613-485bh"]
Sep 11 19:20:58.372051 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:58.371987    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-021613-485bh"]
Sep 11 19:20:58.399009 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:58.398970    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c716675-04b5-4447-9b2b-5f05874c168a" path="/var/lib/kubelet/pods/8c716675-04b5-4447-9b2b-5f05874c168a/volumes"
Sep 11 19:20:58.455288 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:58.455218    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-021613-7ddc8f8c7d-94nhx"
Sep 11 19:20:58.535860 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:58.535826    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/add780d4-d2a4-4224-9804-7468fedf25ee-plugins-config-volume\") pod \"add780d4-d2a4-4224-9804-7468fedf25ee\" (UID: \"add780d4-d2a4-4224-9804-7468fedf25ee\") "
Sep 11 19:20:58.536093 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:58.535934    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7k4gs\" (UniqueName: \"kubernetes.io/projected/add780d4-d2a4-4224-9804-7468fedf25ee-kube-api-access-7k4gs\") pod \"add780d4-d2a4-4224-9804-7468fedf25ee\" (UID: \"add780d4-d2a4-4224-9804-7468fedf25ee\") "
Sep 11 19:20:58.536402 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:58.536372    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/add780d4-d2a4-4224-9804-7468fedf25ee-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "add780d4-d2a4-4224-9804-7468fedf25ee" (UID: "add780d4-d2a4-4224-9804-7468fedf25ee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 19:20:58.539000 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:58.538961    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/add780d4-d2a4-4224-9804-7468fedf25ee-kube-api-access-7k4gs" (OuterVolumeSpecName: "kube-api-access-7k4gs") pod "add780d4-d2a4-4224-9804-7468fedf25ee" (UID: "add780d4-d2a4-4224-9804-7468fedf25ee"). InnerVolumeSpecName "kube-api-access-7k4gs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:20:58.637550 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:58.637507    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/add780d4-d2a4-4224-9804-7468fedf25ee-plugins-config-volume\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:20:58.637550 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:58.637538    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7k4gs\" (UniqueName: \"kubernetes.io/projected/add780d4-d2a4-4224-9804-7468fedf25ee-kube-api-access-7k4gs\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:20:58.962842 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:58.962797    2578 generic.go:356] "Generic (PLEG): container finished" podID="add780d4-d2a4-4224-9804-7468fedf25ee" containerID="46c6b01797dee03843c15b1ee763659a36c0abcb8280909ad68285a6f5329999" exitCode=0
Sep 11 19:20:58.963018 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:58.962850    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-021613-7ddc8f8c7d-94nhx"
Sep 11 19:20:58.963018 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:58.962864    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-021613-7ddc8f8c7d-94nhx" event={"ID":"add780d4-d2a4-4224-9804-7468fedf25ee","Type":"ContainerDied","Data":"46c6b01797dee03843c15b1ee763659a36c0abcb8280909ad68285a6f5329999"}
Sep 11 19:20:58.963018 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:58.962924    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-021613-7ddc8f8c7d-94nhx" event={"ID":"add780d4-d2a4-4224-9804-7468fedf25ee","Type":"ContainerDied","Data":"94e64825faecaf5830adc303949f2407af8bc65f4f4ec6b43be3121413b7d21c"}
Sep 11 19:20:58.963018 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:58.962946    2578 scope.go:117] "RemoveContainer" containerID="46c6b01797dee03843c15b1ee763659a36c0abcb8280909ad68285a6f5329999"
Sep 11 19:20:58.972294 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:58.972270    2578 scope.go:117] "RemoveContainer" containerID="46c6b01797dee03843c15b1ee763659a36c0abcb8280909ad68285a6f5329999"
Sep 11 19:20:58.972560 ip-10-0-134-189 kubenswrapper[2578]: E0911 19:20:58.972539    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46c6b01797dee03843c15b1ee763659a36c0abcb8280909ad68285a6f5329999\": container with ID starting with 46c6b01797dee03843c15b1ee763659a36c0abcb8280909ad68285a6f5329999 not found: ID does not exist" containerID="46c6b01797dee03843c15b1ee763659a36c0abcb8280909ad68285a6f5329999"
Sep 11 19:20:58.972679 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:58.972573    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46c6b01797dee03843c15b1ee763659a36c0abcb8280909ad68285a6f5329999"} err="failed to get container status \"46c6b01797dee03843c15b1ee763659a36c0abcb8280909ad68285a6f5329999\": rpc error: code = NotFound desc = could not find container \"46c6b01797dee03843c15b1ee763659a36c0abcb8280909ad68285a6f5329999\": container with ID starting with 46c6b01797dee03843c15b1ee763659a36c0abcb8280909ad68285a6f5329999 not found: ID does not exist"
Sep 11 19:20:58.984234 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:58.984204    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-021613-7ddc8f8c7d-94nhx"]
Sep 11 19:20:58.990704 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:20:58.990675    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-021613-7ddc8f8c7d-94nhx"]
Sep 11 19:21:00.397401 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:21:00.397364    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="add780d4-d2a4-4224-9804-7468fedf25ee" path="/var/lib/kubelet/pods/add780d4-d2a4-4224-9804-7468fedf25ee/volumes"
Sep 11 19:21:03.705634 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:21:03.705583    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-021613/ipp-route-021613-kserve-d76b9655-qtpvx"]
Sep 11 19:21:18.725316 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:21:18.725266    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-021613/ipp-route-021613-kserve-d76b9655-qtpvx" podUID="afeba088-3a12-4810-9f54-c531f3cefdfe" containerName="main" containerID="cri-o://aa7ba998f8fc183c0fdc53d11e47eb00ba30814685455e9897622e57e5752693" gracePeriod=45
Sep 11 19:21:18.973336 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:21:18.973309    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-021613/ipp-route-021613-kserve-d76b9655-qtpvx"
Sep 11 19:21:19.063044 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:21:19.062951    2578 generic.go:356] "Generic (PLEG): container finished" podID="afeba088-3a12-4810-9f54-c531f3cefdfe" containerID="aa7ba998f8fc183c0fdc53d11e47eb00ba30814685455e9897622e57e5752693" exitCode=0
Sep 11 19:21:19.063044 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:21:19.062995    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-021613/ipp-route-021613-kserve-d76b9655-qtpvx"
Sep 11 19:21:19.063044 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:21:19.063034    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-021613/ipp-route-021613-kserve-d76b9655-qtpvx" event={"ID":"afeba088-3a12-4810-9f54-c531f3cefdfe","Type":"ContainerDied","Data":"aa7ba998f8fc183c0fdc53d11e47eb00ba30814685455e9897622e57e5752693"}
Sep 11 19:21:19.063307 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:21:19.063079    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-021613/ipp-route-021613-kserve-d76b9655-qtpvx" event={"ID":"afeba088-3a12-4810-9f54-c531f3cefdfe","Type":"ContainerDied","Data":"fc14cdfaa2033eccf1f315adfbc6c2acefec6f7f83a03c887e2af68faa62a007"}
Sep 11 19:21:19.063307 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:21:19.063098    2578 scope.go:117] "RemoveContainer" containerID="aa7ba998f8fc183c0fdc53d11e47eb00ba30814685455e9897622e57e5752693"
Sep 11 19:21:19.071614 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:21:19.071577    2578 scope.go:117] "RemoveContainer" containerID="aa7ba998f8fc183c0fdc53d11e47eb00ba30814685455e9897622e57e5752693"
Sep 11 19:21:19.071989 ip-10-0-134-189 kubenswrapper[2578]: E0911 19:21:19.071963    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa7ba998f8fc183c0fdc53d11e47eb00ba30814685455e9897622e57e5752693\": container with ID starting with aa7ba998f8fc183c0fdc53d11e47eb00ba30814685455e9897622e57e5752693 not found: ID does not exist" containerID="aa7ba998f8fc183c0fdc53d11e47eb00ba30814685455e9897622e57e5752693"
Sep 11 19:21:19.072104 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:21:19.071996    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa7ba998f8fc183c0fdc53d11e47eb00ba30814685455e9897622e57e5752693"} err="failed to get container status \"aa7ba998f8fc183c0fdc53d11e47eb00ba30814685455e9897622e57e5752693\": rpc error: code = NotFound desc = could not find container \"aa7ba998f8fc183c0fdc53d11e47eb00ba30814685455e9897622e57e5752693\": container with ID starting with aa7ba998f8fc183c0fdc53d11e47eb00ba30814685455e9897622e57e5752693 not found: ID does not exist"
Sep 11 19:21:19.126378 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:21:19.126337    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afeba088-3a12-4810-9f54-c531f3cefdfe-tmp-dir\") pod \"afeba088-3a12-4810-9f54-c531f3cefdfe\" (UID: \"afeba088-3a12-4810-9f54-c531f3cefdfe\") "
Sep 11 19:21:19.126568 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:21:19.126391    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afeba088-3a12-4810-9f54-c531f3cefdfe-dshm\") pod \"afeba088-3a12-4810-9f54-c531f3cefdfe\" (UID: \"afeba088-3a12-4810-9f54-c531f3cefdfe\") "
Sep 11 19:21:19.126568 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:21:19.126509    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afeba088-3a12-4810-9f54-c531f3cefdfe-tls-certs\") pod \"afeba088-3a12-4810-9f54-c531f3cefdfe\" (UID: \"afeba088-3a12-4810-9f54-c531f3cefdfe\") "
Sep 11 19:21:19.126568 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:21:19.126550    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ksfsh\" (UniqueName: \"kubernetes.io/projected/afeba088-3a12-4810-9f54-c531f3cefdfe-kube-api-access-ksfsh\") pod \"afeba088-3a12-4810-9f54-c531f3cefdfe\" (UID: \"afeba088-3a12-4810-9f54-c531f3cefdfe\") "
Sep 11 19:21:19.126791 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:21:19.126579    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afeba088-3a12-4810-9f54-c531f3cefdfe-model-cache\") pod \"afeba088-3a12-4810-9f54-c531f3cefdfe\" (UID: \"afeba088-3a12-4810-9f54-c531f3cefdfe\") "
Sep 11 19:21:19.126791 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:21:19.126640    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afeba088-3a12-4810-9f54-c531f3cefdfe-home\") pod \"afeba088-3a12-4810-9f54-c531f3cefdfe\" (UID: \"afeba088-3a12-4810-9f54-c531f3cefdfe\") "
Sep 11 19:21:19.126791 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:21:19.126704    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afeba088-3a12-4810-9f54-c531f3cefdfe-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "afeba088-3a12-4810-9f54-c531f3cefdfe" (UID: "afeba088-3a12-4810-9f54-c531f3cefdfe"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:21:19.126967 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:21:19.126902    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afeba088-3a12-4810-9f54-c531f3cefdfe-tmp-dir\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:21:19.126967 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:21:19.126915    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afeba088-3a12-4810-9f54-c531f3cefdfe-model-cache" (OuterVolumeSpecName: "model-cache") pod "afeba088-3a12-4810-9f54-c531f3cefdfe" (UID: "afeba088-3a12-4810-9f54-c531f3cefdfe"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:21:19.127044 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:21:19.126958    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afeba088-3a12-4810-9f54-c531f3cefdfe-home" (OuterVolumeSpecName: "home") pod "afeba088-3a12-4810-9f54-c531f3cefdfe" (UID: "afeba088-3a12-4810-9f54-c531f3cefdfe"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:21:19.128898 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:21:19.128871    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afeba088-3a12-4810-9f54-c531f3cefdfe-dshm" (OuterVolumeSpecName: "dshm") pod "afeba088-3a12-4810-9f54-c531f3cefdfe" (UID: "afeba088-3a12-4810-9f54-c531f3cefdfe"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:21:19.129006 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:21:19.128908    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afeba088-3a12-4810-9f54-c531f3cefdfe-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "afeba088-3a12-4810-9f54-c531f3cefdfe" (UID: "afeba088-3a12-4810-9f54-c531f3cefdfe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 19:21:19.129006 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:21:19.128994    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afeba088-3a12-4810-9f54-c531f3cefdfe-kube-api-access-ksfsh" (OuterVolumeSpecName: "kube-api-access-ksfsh") pod "afeba088-3a12-4810-9f54-c531f3cefdfe" (UID: "afeba088-3a12-4810-9f54-c531f3cefdfe"). InnerVolumeSpecName "kube-api-access-ksfsh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:21:19.227355 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:21:19.227318    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afeba088-3a12-4810-9f54-c531f3cefdfe-dshm\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:21:19.227355 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:21:19.227348    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afeba088-3a12-4810-9f54-c531f3cefdfe-tls-certs\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:21:19.227355 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:21:19.227359    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksfsh\" (UniqueName: \"kubernetes.io/projected/afeba088-3a12-4810-9f54-c531f3cefdfe-kube-api-access-ksfsh\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:21:19.227355 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:21:19.227368    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afeba088-3a12-4810-9f54-c531f3cefdfe-model-cache\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:21:19.227668 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:21:19.227377    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afeba088-3a12-4810-9f54-c531f3cefdfe-home\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:21:19.383291 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:21:19.383234    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-021613/ipp-route-021613-kserve-d76b9655-qtpvx"]
Sep 11 19:21:19.387678 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:21:19.387653    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-021613/ipp-route-021613-kserve-d76b9655-qtpvx"]
Sep 11 19:21:20.396966 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:21:20.396935    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="afeba088-3a12-4810-9f54-c531f3cefdfe" path="/var/lib/kubelet/pods/afeba088-3a12-4810-9f54-c531f3cefdfe/volumes"
Sep 11 19:21:44.803246 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:21:44.803167    2578 scope.go:117] "RemoveContainer" containerID="fb5532e7b52fbc4a19fd2687359aaf32adea7e65f5d2d7d574c49cfe2b8e06b7"
Sep 11 19:21:46.998879 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:21:46.998844    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-afd85c-kvkjl"]
Sep 11 19:21:46.999255 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:21:46.999178    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="add780d4-d2a4-4224-9804-7468fedf25ee" containerName="payload-pre-processing"
Sep 11 19:21:46.999255 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:21:46.999189    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="add780d4-d2a4-4224-9804-7468fedf25ee" containerName="payload-pre-processing"
Sep 11 19:21:46.999255 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:21:46.999214    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afeba088-3a12-4810-9f54-c531f3cefdfe" containerName="main"
Sep 11 19:21:46.999255 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:21:46.999219    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="afeba088-3a12-4810-9f54-c531f3cefdfe" containerName="main"
Sep 11 19:21:46.999255 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:21:46.999226    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c716675-04b5-4447-9b2b-5f05874c168a" containerName="revoke-keys"
Sep 11 19:21:46.999255 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:21:46.999232    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c716675-04b5-4447-9b2b-5f05874c168a" containerName="revoke-keys"
Sep 11 19:21:46.999453 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:21:46.999287    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c716675-04b5-4447-9b2b-5f05874c168a" containerName="revoke-keys"
Sep 11 19:21:46.999453 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:21:46.999296    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="add780d4-d2a4-4224-9804-7468fedf25ee" containerName="payload-pre-processing"
Sep 11 19:21:46.999453 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:21:46.999301    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="afeba088-3a12-4810-9f54-c531f3cefdfe" containerName="main"
Sep 11 19:21:47.001992 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:21:47.001973    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-afd85c-kvkjl"
Sep 11 19:21:47.003959 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:21:47.003937    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 19:21:47.004089 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:21:47.003944    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ln295\""
Sep 11 19:21:47.012417 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:21:47.012395    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-afd85c-kvkjl"]
Sep 11 19:21:47.086367 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:21:47.086327    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0582a783-0126-4433-a491-0fe21b501184-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-afd85c-kvkjl\" (UID: \"0582a783-0126-4433-a491-0fe21b501184\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-afd85c-kvkjl"
Sep 11 19:21:47.187498 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:21:47.187464    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0582a783-0126-4433-a491-0fe21b501184-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-afd85c-kvkjl\" (UID: \"0582a783-0126-4433-a491-0fe21b501184\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-afd85c-kvkjl"
Sep 11 19:21:47.188182 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:21:47.188161    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0582a783-0126-4433-a491-0fe21b501184-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-afd85c-kvkjl\" (UID: \"0582a783-0126-4433-a491-0fe21b501184\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-afd85c-kvkjl"
Sep 11 19:21:47.312788 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:21:47.312693    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-afd85c-kvkjl"
Sep 11 19:21:47.585187 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:21:47.585161    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-afd85c-kvkjl"]
Sep 11 19:21:47.587317 ip-10-0-134-189 kubenswrapper[2578]: W0911 19:21:47.587287    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0582a783_0126_4433_a491_0fe21b501184.slice/crio-cbd417a9d16384ecd86cff66ef72c21ecc9f70683c870f25151f5d86eac2cfe0 WatchSource:0}: Error finding container cbd417a9d16384ecd86cff66ef72c21ecc9f70683c870f25151f5d86eac2cfe0: Status 404 returned error can't find the container with id cbd417a9d16384ecd86cff66ef72c21ecc9f70683c870f25151f5d86eac2cfe0
Sep 11 19:21:48.205890 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:21:48.205853    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-afd85c-kvkjl" event={"ID":"0582a783-0126-4433-a491-0fe21b501184","Type":"ContainerStarted","Data":"5bca4c15f362c3454655b27f8a6a5e3e03af078343a6d1954184d5c63f1401c8"}
Sep 11 19:21:48.206295 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:21:48.205940    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-afd85c-kvkjl" event={"ID":"0582a783-0126-4433-a491-0fe21b501184","Type":"ContainerStarted","Data":"cbd417a9d16384ecd86cff66ef72c21ecc9f70683c870f25151f5d86eac2cfe0"}
Sep 11 19:21:48.221138 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:21:48.221081    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-afd85c-kvkjl" podStartSLOduration=2.221064595 podStartE2EDuration="2.221064595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:21:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:21:48.21886393 +0000 UTC m=+1984.296454481" watchObservedRunningTime="2026-09-11 19:21:48.221064595 +0000 UTC m=+1984.298655151"
Sep 11 19:21:49.211712 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:21:49.211678    2578 generic.go:356] "Generic (PLEG): container finished" podID="0582a783-0126-4433-a491-0fe21b501184" containerID="5bca4c15f362c3454655b27f8a6a5e3e03af078343a6d1954184d5c63f1401c8" exitCode=0
Sep 11 19:21:49.212122 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:21:49.211771    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-afd85c-kvkjl" event={"ID":"0582a783-0126-4433-a491-0fe21b501184","Type":"ContainerDied","Data":"5bca4c15f362c3454655b27f8a6a5e3e03af078343a6d1954184d5c63f1401c8"}
Sep 11 19:21:50.354431 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:21:50.354405    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-afd85c-kvkjl"
Sep 11 19:21:50.416248 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:21:50.416208    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0582a783-0126-4433-a491-0fe21b501184-maas-api-service-ca\") pod \"0582a783-0126-4433-a491-0fe21b501184\" (UID: \"0582a783-0126-4433-a491-0fe21b501184\") "
Sep 11 19:21:50.416619 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:21:50.416566    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0582a783-0126-4433-a491-0fe21b501184-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0582a783-0126-4433-a491-0fe21b501184" (UID: "0582a783-0126-4433-a491-0fe21b501184"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 19:21:50.517515 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:21:50.517425    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0582a783-0126-4433-a491-0fe21b501184-maas-api-service-ca\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:21:51.223818 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:21:51.223791    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-afd85c-kvkjl"
Sep 11 19:21:51.223818 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:21:51.223798    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-afd85c-kvkjl" event={"ID":"0582a783-0126-4433-a491-0fe21b501184","Type":"ContainerDied","Data":"cbd417a9d16384ecd86cff66ef72c21ecc9f70683c870f25151f5d86eac2cfe0"}
Sep 11 19:21:51.224041 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:21:51.223843    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cbd417a9d16384ecd86cff66ef72c21ecc9f70683c870f25151f5d86eac2cfe0"
Sep 11 19:21:57.543076 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:21:57.543030    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-afd85c-5c4cfc56c4-jw45m"]
Sep 11 19:21:57.543583 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:21:57.543238    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-afd85c-5c4cfc56c4-jw45m" podUID="17e21aa6-abdb-4fe1-a304-4e6c3bf03b2e" containerName="payload-pre-processing" containerID="cri-o://f11c085c5252519e7d89211e8fe2871d000c1ffa15dd019817dbf772777435a3" gracePeriod=30
Sep 11 19:21:57.723460 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:21:57.723423    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-afd85c-kvkjl"]
Sep 11 19:21:57.726078 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:21:57.726048    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-afd85c-kvkjl"]
Sep 11 19:21:57.791239 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:21:57.791212    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-afd85c-5c4cfc56c4-jw45m"
Sep 11 19:21:57.886448 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:21:57.886341    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17e21aa6-abdb-4fe1-a304-4e6c3bf03b2e-plugins-config-volume\") pod \"17e21aa6-abdb-4fe1-a304-4e6c3bf03b2e\" (UID: \"17e21aa6-abdb-4fe1-a304-4e6c3bf03b2e\") "
Sep 11 19:21:57.886448 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:21:57.886415    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cww5v\" (UniqueName: \"kubernetes.io/projected/17e21aa6-abdb-4fe1-a304-4e6c3bf03b2e-kube-api-access-cww5v\") pod \"17e21aa6-abdb-4fe1-a304-4e6c3bf03b2e\" (UID: \"17e21aa6-abdb-4fe1-a304-4e6c3bf03b2e\") "
Sep 11 19:21:57.891021 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:21:57.890974    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17e21aa6-abdb-4fe1-a304-4e6c3bf03b2e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "17e21aa6-abdb-4fe1-a304-4e6c3bf03b2e" (UID: "17e21aa6-abdb-4fe1-a304-4e6c3bf03b2e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 19:21:57.901787 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:21:57.901744    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17e21aa6-abdb-4fe1-a304-4e6c3bf03b2e-kube-api-access-cww5v" (OuterVolumeSpecName: "kube-api-access-cww5v") pod "17e21aa6-abdb-4fe1-a304-4e6c3bf03b2e" (UID: "17e21aa6-abdb-4fe1-a304-4e6c3bf03b2e"). InnerVolumeSpecName "kube-api-access-cww5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:21:57.987937 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:21:57.987895    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17e21aa6-abdb-4fe1-a304-4e6c3bf03b2e-plugins-config-volume\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:21:57.987937 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:21:57.987927    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cww5v\" (UniqueName: \"kubernetes.io/projected/17e21aa6-abdb-4fe1-a304-4e6c3bf03b2e-kube-api-access-cww5v\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:21:58.258022 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:21:58.257984    2578 generic.go:356] "Generic (PLEG): container finished" podID="17e21aa6-abdb-4fe1-a304-4e6c3bf03b2e" containerID="f11c085c5252519e7d89211e8fe2871d000c1ffa15dd019817dbf772777435a3" exitCode=0
Sep 11 19:21:58.258246 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:21:58.258047    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-afd85c-5c4cfc56c4-jw45m"
Sep 11 19:21:58.258246 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:21:58.258079    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-afd85c-5c4cfc56c4-jw45m" event={"ID":"17e21aa6-abdb-4fe1-a304-4e6c3bf03b2e","Type":"ContainerDied","Data":"f11c085c5252519e7d89211e8fe2871d000c1ffa15dd019817dbf772777435a3"}
Sep 11 19:21:58.258246 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:21:58.258177    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-afd85c-5c4cfc56c4-jw45m" event={"ID":"17e21aa6-abdb-4fe1-a304-4e6c3bf03b2e","Type":"ContainerDied","Data":"74fdca47166c1854560f28ca5c4073a347c8f5648ecde9ca0f4cc196be97b389"}
Sep 11 19:21:58.258246 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:21:58.258208    2578 scope.go:117] "RemoveContainer" containerID="f11c085c5252519e7d89211e8fe2871d000c1ffa15dd019817dbf772777435a3"
Sep 11 19:21:58.267283 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:21:58.267267    2578 scope.go:117] "RemoveContainer" containerID="f11c085c5252519e7d89211e8fe2871d000c1ffa15dd019817dbf772777435a3"
Sep 11 19:21:58.267611 ip-10-0-134-189 kubenswrapper[2578]: E0911 19:21:58.267575    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f11c085c5252519e7d89211e8fe2871d000c1ffa15dd019817dbf772777435a3\": container with ID starting with f11c085c5252519e7d89211e8fe2871d000c1ffa15dd019817dbf772777435a3 not found: ID does not exist" containerID="f11c085c5252519e7d89211e8fe2871d000c1ffa15dd019817dbf772777435a3"
Sep 11 19:21:58.267662 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:21:58.267629    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f11c085c5252519e7d89211e8fe2871d000c1ffa15dd019817dbf772777435a3"} err="failed to get container status \"f11c085c5252519e7d89211e8fe2871d000c1ffa15dd019817dbf772777435a3\": rpc error: code = NotFound desc = could not find container \"f11c085c5252519e7d89211e8fe2871d000c1ffa15dd019817dbf772777435a3\": container with ID starting with f11c085c5252519e7d89211e8fe2871d000c1ffa15dd019817dbf772777435a3 not found: ID does not exist"
Sep 11 19:21:58.284887 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:21:58.284857    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-afd85c-5c4cfc56c4-jw45m"]
Sep 11 19:21:58.290952 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:21:58.290918    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-afd85c-5c4cfc56c4-jw45m"]
Sep 11 19:21:58.396788 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:21:58.396756    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0582a783-0126-4433-a491-0fe21b501184" path="/var/lib/kubelet/pods/0582a783-0126-4433-a491-0fe21b501184/volumes"
Sep 11 19:21:58.397137 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:21:58.397124    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17e21aa6-abdb-4fe1-a304-4e6c3bf03b2e" path="/var/lib/kubelet/pods/17e21aa6-abdb-4fe1-a304-4e6c3bf03b2e/volumes"
Sep 11 19:22:04.170183 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:04.170150    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-afd85c-openshift-default-7d69487dfd-hvsgv"]
Sep 11 19:22:04.170691 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:04.170395    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-afd85c-openshift-default-7d69487dfd-hvsgv" podUID="2bfe0225-fb2e-4616-ba97-26f83d861f1e" containerName="istio-proxy" containerID="cri-o://fdb611210da3b260d6c75d7a2bc1a2aa8045d40d4f827e25e9caf19240c67775" gracePeriod=30
Sep 11 19:22:04.585187 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:04.585145    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8280eb-mfqgz"]
Sep 11 19:22:04.585498 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:04.585485    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0582a783-0126-4433-a491-0fe21b501184" containerName="revoke-keys"
Sep 11 19:22:04.585545 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:04.585500    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0582a783-0126-4433-a491-0fe21b501184" containerName="revoke-keys"
Sep 11 19:22:04.585545 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:04.585516    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17e21aa6-abdb-4fe1-a304-4e6c3bf03b2e" containerName="payload-pre-processing"
Sep 11 19:22:04.585545 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:04.585522    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="17e21aa6-abdb-4fe1-a304-4e6c3bf03b2e" containerName="payload-pre-processing"
Sep 11 19:22:04.585680 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:04.585571    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="0582a783-0126-4433-a491-0fe21b501184" containerName="revoke-keys"
Sep 11 19:22:04.585680 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:04.585581    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="17e21aa6-abdb-4fe1-a304-4e6c3bf03b2e" containerName="payload-pre-processing"
Sep 11 19:22:04.590331 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:04.590299    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8280eb-mfqgz"
Sep 11 19:22:04.590704 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:04.590673    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8280eb-mfqgz"]
Sep 11 19:22:04.592666 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:04.592638    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ln295\""
Sep 11 19:22:04.592803 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:04.592691    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 19:22:04.748896 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:04.748854    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b812694-cf9d-4ae3-9b86-f48bd716c4c9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-8280eb-mfqgz\" (UID: \"0b812694-cf9d-4ae3-9b86-f48bd716c4c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8280eb-mfqgz"
Sep 11 19:22:04.849468 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:04.849376    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b812694-cf9d-4ae3-9b86-f48bd716c4c9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-8280eb-mfqgz\" (UID: \"0b812694-cf9d-4ae3-9b86-f48bd716c4c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8280eb-mfqgz"
Sep 11 19:22:04.850119 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:04.850096    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b812694-cf9d-4ae3-9b86-f48bd716c4c9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-8280eb-mfqgz\" (UID: \"0b812694-cf9d-4ae3-9b86-f48bd716c4c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8280eb-mfqgz"
Sep 11 19:22:04.902406 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:04.902356    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8280eb-mfqgz"
Sep 11 19:22:05.175735 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:05.175709    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8280eb-mfqgz"]
Sep 11 19:22:05.178094 ip-10-0-134-189 kubenswrapper[2578]: W0911 19:22:05.178066    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b812694_cf9d_4ae3_9b86_f48bd716c4c9.slice/crio-1e7e09a396592dcb713652d2e7e893d112a05aa1e56166a250dd55a565111cfa WatchSource:0}: Error finding container 1e7e09a396592dcb713652d2e7e893d112a05aa1e56166a250dd55a565111cfa: Status 404 returned error can't find the container with id 1e7e09a396592dcb713652d2e7e893d112a05aa1e56166a250dd55a565111cfa
Sep 11 19:22:05.294045 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:05.294007    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8280eb-mfqgz" event={"ID":"0b812694-cf9d-4ae3-9b86-f48bd716c4c9","Type":"ContainerStarted","Data":"1e7e09a396592dcb713652d2e7e893d112a05aa1e56166a250dd55a565111cfa"}
Sep 11 19:22:06.209282 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:06.209241    2578 patch_prober.go:28] interesting pod/e2e-ipp-b-afd85c-openshift-default-7d69487dfd-hvsgv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 11 19:22:06.209728 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:06.209316    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-afd85c-openshift-default-7d69487dfd-hvsgv" podUID="2bfe0225-fb2e-4616-ba97-26f83d861f1e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 19:22:06.300948 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:06.300915    2578 generic.go:356] "Generic (PLEG): container finished" podID="0b812694-cf9d-4ae3-9b86-f48bd716c4c9" containerID="c6d16d226c3f3a7d389112e467b97c43c14f14e49abf7563134691e523e9b4a1" exitCode=0
Sep 11 19:22:06.301124 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:06.300998    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8280eb-mfqgz" event={"ID":"0b812694-cf9d-4ae3-9b86-f48bd716c4c9","Type":"ContainerDied","Data":"c6d16d226c3f3a7d389112e467b97c43c14f14e49abf7563134691e523e9b4a1"}
Sep 11 19:22:07.437714 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:07.437683    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8280eb-mfqgz"
Sep 11 19:22:07.575313 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:07.575228    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b812694-cf9d-4ae3-9b86-f48bd716c4c9-maas-api-service-ca\") pod \"0b812694-cf9d-4ae3-9b86-f48bd716c4c9\" (UID: \"0b812694-cf9d-4ae3-9b86-f48bd716c4c9\") "
Sep 11 19:22:07.575604 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:07.575566    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b812694-cf9d-4ae3-9b86-f48bd716c4c9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0b812694-cf9d-4ae3-9b86-f48bd716c4c9" (UID: "0b812694-cf9d-4ae3-9b86-f48bd716c4c9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 19:22:07.675957 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:07.675919    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b812694-cf9d-4ae3-9b86-f48bd716c4c9-maas-api-service-ca\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:22:08.312361 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:08.312318    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8280eb-mfqgz" event={"ID":"0b812694-cf9d-4ae3-9b86-f48bd716c4c9","Type":"ContainerDied","Data":"1e7e09a396592dcb713652d2e7e893d112a05aa1e56166a250dd55a565111cfa"}
Sep 11 19:22:08.312549 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:08.312406    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e7e09a396592dcb713652d2e7e893d112a05aa1e56166a250dd55a565111cfa"
Sep 11 19:22:08.312549 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:08.312340    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8280eb-mfqgz"
Sep 11 19:22:09.321198 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:09.321153    2578 generic.go:356] "Generic (PLEG): container finished" podID="2bfe0225-fb2e-4616-ba97-26f83d861f1e" containerID="fdb611210da3b260d6c75d7a2bc1a2aa8045d40d4f827e25e9caf19240c67775" exitCode=0
Sep 11 19:22:09.321694 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:09.321227    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-afd85c-openshift-default-7d69487dfd-hvsgv" event={"ID":"2bfe0225-fb2e-4616-ba97-26f83d861f1e","Type":"ContainerDied","Data":"fdb611210da3b260d6c75d7a2bc1a2aa8045d40d4f827e25e9caf19240c67775"}
Sep 11 19:22:09.321694 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:09.321306    2578 scope.go:117] "RemoveContainer" containerID="5dfe84568a7a65e4b15aaa36e17306820cd07f47df78c65e6965a0103aadf2b6"
Sep 11 19:22:09.443050 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:09.442983    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-afd85c-openshift-default-7d69487dfd-hvsgv"
Sep 11 19:22:09.595195 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:09.595155    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bfe0225-fb2e-4616-ba97-26f83d861f1e-workload-socket\") pod \"2bfe0225-fb2e-4616-ba97-26f83d861f1e\" (UID: \"2bfe0225-fb2e-4616-ba97-26f83d861f1e\") "
Sep 11 19:22:09.595195 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:09.595204    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bfe0225-fb2e-4616-ba97-26f83d861f1e-istio-envoy\") pod \"2bfe0225-fb2e-4616-ba97-26f83d861f1e\" (UID: \"2bfe0225-fb2e-4616-ba97-26f83d861f1e\") "
Sep 11 19:22:09.595442 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:09.595224    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bfe0225-fb2e-4616-ba97-26f83d861f1e-credential-socket\") pod \"2bfe0225-fb2e-4616-ba97-26f83d861f1e\" (UID: \"2bfe0225-fb2e-4616-ba97-26f83d861f1e\") "
Sep 11 19:22:09.595442 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:09.595245    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bfe0225-fb2e-4616-ba97-26f83d861f1e-istio-token\") pod \"2bfe0225-fb2e-4616-ba97-26f83d861f1e\" (UID: \"2bfe0225-fb2e-4616-ba97-26f83d861f1e\") "
Sep 11 19:22:09.595442 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:09.595282    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8dx46\" (UniqueName: \"kubernetes.io/projected/2bfe0225-fb2e-4616-ba97-26f83d861f1e-kube-api-access-8dx46\") pod \"2bfe0225-fb2e-4616-ba97-26f83d861f1e\" (UID: \"2bfe0225-fb2e-4616-ba97-26f83d861f1e\") "
Sep 11 19:22:09.595442 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:09.595309    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bfe0225-fb2e-4616-ba97-26f83d861f1e-istio-podinfo\") pod \"2bfe0225-fb2e-4616-ba97-26f83d861f1e\" (UID: \"2bfe0225-fb2e-4616-ba97-26f83d861f1e\") "
Sep 11 19:22:09.595442 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:09.595329    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bfe0225-fb2e-4616-ba97-26f83d861f1e-workload-certs\") pod \"2bfe0225-fb2e-4616-ba97-26f83d861f1e\" (UID: \"2bfe0225-fb2e-4616-ba97-26f83d861f1e\") "
Sep 11 19:22:09.595442 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:09.595366    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bfe0225-fb2e-4616-ba97-26f83d861f1e-istio-data\") pod \"2bfe0225-fb2e-4616-ba97-26f83d861f1e\" (UID: \"2bfe0225-fb2e-4616-ba97-26f83d861f1e\") "
Sep 11 19:22:09.595787 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:09.595443    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bfe0225-fb2e-4616-ba97-26f83d861f1e-istiod-ca-cert\") pod \"2bfe0225-fb2e-4616-ba97-26f83d861f1e\" (UID: \"2bfe0225-fb2e-4616-ba97-26f83d861f1e\") "
Sep 11 19:22:09.595787 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:09.595539    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bfe0225-fb2e-4616-ba97-26f83d861f1e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2bfe0225-fb2e-4616-ba97-26f83d861f1e" (UID: "2bfe0225-fb2e-4616-ba97-26f83d861f1e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:22:09.595787 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:09.595752    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bfe0225-fb2e-4616-ba97-26f83d861f1e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2bfe0225-fb2e-4616-ba97-26f83d861f1e" (UID: "2bfe0225-fb2e-4616-ba97-26f83d861f1e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:22:09.596008 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:09.595839    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bfe0225-fb2e-4616-ba97-26f83d861f1e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2bfe0225-fb2e-4616-ba97-26f83d861f1e" (UID: "2bfe0225-fb2e-4616-ba97-26f83d861f1e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:22:09.596008 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:09.595925    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bfe0225-fb2e-4616-ba97-26f83d861f1e-credential-socket\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:22:09.596008 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:09.595948    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bfe0225-fb2e-4616-ba97-26f83d861f1e-workload-certs\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:22:09.596008 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:09.595949    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bfe0225-fb2e-4616-ba97-26f83d861f1e-istio-data" (OuterVolumeSpecName: "istio-data") pod "2bfe0225-fb2e-4616-ba97-26f83d861f1e" (UID: "2bfe0225-fb2e-4616-ba97-26f83d861f1e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:22:09.596008 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:09.595953    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bfe0225-fb2e-4616-ba97-26f83d861f1e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2bfe0225-fb2e-4616-ba97-26f83d861f1e" (UID: "2bfe0225-fb2e-4616-ba97-26f83d861f1e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 19:22:09.597695 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:09.597666    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bfe0225-fb2e-4616-ba97-26f83d861f1e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2bfe0225-fb2e-4616-ba97-26f83d861f1e" (UID: "2bfe0225-fb2e-4616-ba97-26f83d861f1e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:22:09.597805 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:09.597701    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bfe0225-fb2e-4616-ba97-26f83d861f1e-istio-token" (OuterVolumeSpecName: "istio-token") pod "2bfe0225-fb2e-4616-ba97-26f83d861f1e" (UID: "2bfe0225-fb2e-4616-ba97-26f83d861f1e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:22:09.597805 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:09.597751    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bfe0225-fb2e-4616-ba97-26f83d861f1e-kube-api-access-8dx46" (OuterVolumeSpecName: "kube-api-access-8dx46") pod "2bfe0225-fb2e-4616-ba97-26f83d861f1e" (UID: "2bfe0225-fb2e-4616-ba97-26f83d861f1e"). InnerVolumeSpecName "kube-api-access-8dx46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:22:09.597878 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:09.597818    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2bfe0225-fb2e-4616-ba97-26f83d861f1e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2bfe0225-fb2e-4616-ba97-26f83d861f1e" (UID: "2bfe0225-fb2e-4616-ba97-26f83d861f1e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 19:22:09.697097 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:09.696984    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bfe0225-fb2e-4616-ba97-26f83d861f1e-istio-data\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:22:09.697097 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:09.697032    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bfe0225-fb2e-4616-ba97-26f83d861f1e-istiod-ca-cert\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:22:09.697097 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:09.697046    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bfe0225-fb2e-4616-ba97-26f83d861f1e-workload-socket\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:22:09.697097 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:09.697058    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bfe0225-fb2e-4616-ba97-26f83d861f1e-istio-envoy\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:22:09.697097 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:09.697070    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bfe0225-fb2e-4616-ba97-26f83d861f1e-istio-token\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:22:09.697097 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:09.697081    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dx46\" (UniqueName: \"kubernetes.io/projected/2bfe0225-fb2e-4616-ba97-26f83d861f1e-kube-api-access-8dx46\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:22:09.697097 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:09.697095    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bfe0225-fb2e-4616-ba97-26f83d861f1e-istio-podinfo\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:22:10.327605 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:10.327571    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-afd85c-openshift-default-7d69487dfd-hvsgv"
Sep 11 19:22:10.328061 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:10.327625    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-afd85c-openshift-default-7d69487dfd-hvsgv" event={"ID":"2bfe0225-fb2e-4616-ba97-26f83d861f1e","Type":"ContainerDied","Data":"497566b73090b419702df16930f291c74f3a0af87e51d5e649cbc9181aa972c0"}
Sep 11 19:22:10.328061 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:10.327686    2578 scope.go:117] "RemoveContainer" containerID="fdb611210da3b260d6c75d7a2bc1a2aa8045d40d4f827e25e9caf19240c67775"
Sep 11 19:22:10.347761 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:10.347727    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-afd85c-openshift-default-7d69487dfd-hvsgv"]
Sep 11 19:22:10.351320 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:10.351292    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-afd85c-openshift-default-7d69487dfd-hvsgv"]
Sep 11 19:22:10.396881 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:10.396850    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2bfe0225-fb2e-4616-ba97-26f83d861f1e" path="/var/lib/kubelet/pods/2bfe0225-fb2e-4616-ba97-26f83d861f1e/volumes"
Sep 11 19:22:15.443656 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:15.443624    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8280eb-5c759788c8-kgjp2"]
Sep 11 19:22:15.444143 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:15.443839    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8280eb-5c759788c8-kgjp2" podUID="4b3672a7-8dfe-47e5-b418-48bb90d73622" containerName="payload-pre-processing" containerID="cri-o://ea97ec975c4959012702854ffc6555679a9552896dc36e8f2d804e46fb79bce1" gracePeriod=30
Sep 11 19:22:15.686020 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:15.685991    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8280eb-5c759788c8-kgjp2"
Sep 11 19:22:15.732933 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:15.732892    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8280eb-mfqgz"]
Sep 11 19:22:15.742209 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:15.742170    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8280eb-mfqgz"]
Sep 11 19:22:15.751100 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:15.751069    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b3672a7-8dfe-47e5-b418-48bb90d73622-plugins-config-volume\") pod \"4b3672a7-8dfe-47e5-b418-48bb90d73622\" (UID: \"4b3672a7-8dfe-47e5-b418-48bb90d73622\") "
Sep 11 19:22:15.751264 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:15.751176    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vdxrk\" (UniqueName: \"kubernetes.io/projected/4b3672a7-8dfe-47e5-b418-48bb90d73622-kube-api-access-vdxrk\") pod \"4b3672a7-8dfe-47e5-b418-48bb90d73622\" (UID: \"4b3672a7-8dfe-47e5-b418-48bb90d73622\") "
Sep 11 19:22:15.751448 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:15.751424    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b3672a7-8dfe-47e5-b418-48bb90d73622-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4b3672a7-8dfe-47e5-b418-48bb90d73622" (UID: "4b3672a7-8dfe-47e5-b418-48bb90d73622"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 19:22:15.753337 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:15.753317    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b3672a7-8dfe-47e5-b418-48bb90d73622-kube-api-access-vdxrk" (OuterVolumeSpecName: "kube-api-access-vdxrk") pod "4b3672a7-8dfe-47e5-b418-48bb90d73622" (UID: "4b3672a7-8dfe-47e5-b418-48bb90d73622"). InnerVolumeSpecName "kube-api-access-vdxrk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:22:15.851790 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:15.851750    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdxrk\" (UniqueName: \"kubernetes.io/projected/4b3672a7-8dfe-47e5-b418-48bb90d73622-kube-api-access-vdxrk\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:22:15.851790 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:15.851785    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b3672a7-8dfe-47e5-b418-48bb90d73622-plugins-config-volume\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:22:16.360559 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:16.360518    2578 generic.go:356] "Generic (PLEG): container finished" podID="4b3672a7-8dfe-47e5-b418-48bb90d73622" containerID="ea97ec975c4959012702854ffc6555679a9552896dc36e8f2d804e46fb79bce1" exitCode=0
Sep 11 19:22:16.360791 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:16.360567    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8280eb-5c759788c8-kgjp2"
Sep 11 19:22:16.360791 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:16.360628    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8280eb-5c759788c8-kgjp2" event={"ID":"4b3672a7-8dfe-47e5-b418-48bb90d73622","Type":"ContainerDied","Data":"ea97ec975c4959012702854ffc6555679a9552896dc36e8f2d804e46fb79bce1"}
Sep 11 19:22:16.360791 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:16.360690    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8280eb-5c759788c8-kgjp2" event={"ID":"4b3672a7-8dfe-47e5-b418-48bb90d73622","Type":"ContainerDied","Data":"234ede84ea3a62edf00973e2046d2c88afed502577d380da04651b41485b3da4"}
Sep 11 19:22:16.360791 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:16.360713    2578 scope.go:117] "RemoveContainer" containerID="ea97ec975c4959012702854ffc6555679a9552896dc36e8f2d804e46fb79bce1"
Sep 11 19:22:16.370176 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:16.370152    2578 scope.go:117] "RemoveContainer" containerID="ea97ec975c4959012702854ffc6555679a9552896dc36e8f2d804e46fb79bce1"
Sep 11 19:22:16.370467 ip-10-0-134-189 kubenswrapper[2578]: E0911 19:22:16.370448    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea97ec975c4959012702854ffc6555679a9552896dc36e8f2d804e46fb79bce1\": container with ID starting with ea97ec975c4959012702854ffc6555679a9552896dc36e8f2d804e46fb79bce1 not found: ID does not exist" containerID="ea97ec975c4959012702854ffc6555679a9552896dc36e8f2d804e46fb79bce1"
Sep 11 19:22:16.370519 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:16.370477    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea97ec975c4959012702854ffc6555679a9552896dc36e8f2d804e46fb79bce1"} err="failed to get container status \"ea97ec975c4959012702854ffc6555679a9552896dc36e8f2d804e46fb79bce1\": rpc error: code = NotFound desc = could not find container \"ea97ec975c4959012702854ffc6555679a9552896dc36e8f2d804e46fb79bce1\": container with ID starting with ea97ec975c4959012702854ffc6555679a9552896dc36e8f2d804e46fb79bce1 not found: ID does not exist"
Sep 11 19:22:16.381393 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:16.381363    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8280eb-5c759788c8-kgjp2"]
Sep 11 19:22:16.387216 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:16.387190    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8280eb-5c759788c8-kgjp2"]
Sep 11 19:22:16.395908 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:16.395884    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b812694-cf9d-4ae3-9b86-f48bd716c4c9" path="/var/lib/kubelet/pods/0b812694-cf9d-4ae3-9b86-f48bd716c4c9/volumes"
Sep 11 19:22:16.396245 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:16.396233    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b3672a7-8dfe-47e5-b418-48bb90d73622" path="/var/lib/kubelet/pods/4b3672a7-8dfe-47e5-b418-48bb90d73622/volumes"
Sep 11 19:22:34.222986 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:34.222954    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3b7391-29819235-4zg72"]
Sep 11 19:22:34.226386 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:34.226353    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3b7391-29819235-4zg72"]
Sep 11 19:22:34.401773 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:34.401737    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d11c2973-70ae-461a-bfc4-08320c25c06d" path="/var/lib/kubelet/pods/d11c2973-70ae-461a-bfc4-08320c25c06d/volumes"
Sep 11 19:22:41.916560 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:41.916516    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-3b7391-openshift-default-7cd68cb6bd-kwtt7"]
Sep 11 19:22:41.917157 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:41.916913    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-3b7391-openshift-default-7cd68cb6bd-kwtt7" podUID="0eff5cf7-789d-4dbf-9894-568df3b2ba47" containerName="istio-proxy" containerID="cri-o://6100cc7be28085a781dc7153cc6e10dc7ddcb9676cb77e7a9b1cf4b720946445" gracePeriod=30
Sep 11 19:22:44.829672 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:44.829637    2578 scope.go:117] "RemoveContainer" containerID="fdd8a894bc89693e8cf2119e9e5c0e69fc915017243c48f7ab2f69658b36e170"
Sep 11 19:22:47.170575 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:47.170545    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-3b7391-openshift-default-7cd68cb6bd-kwtt7"
Sep 11 19:22:47.335007 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:47.334965    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0eff5cf7-789d-4dbf-9894-568df3b2ba47-credential-socket\") pod \"0eff5cf7-789d-4dbf-9894-568df3b2ba47\" (UID: \"0eff5cf7-789d-4dbf-9894-568df3b2ba47\") "
Sep 11 19:22:47.335007 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:47.335014    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0eff5cf7-789d-4dbf-9894-568df3b2ba47-workload-certs\") pod \"0eff5cf7-789d-4dbf-9894-568df3b2ba47\" (UID: \"0eff5cf7-789d-4dbf-9894-568df3b2ba47\") "
Sep 11 19:22:47.335337 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:47.335042    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-59wr7\" (UniqueName: \"kubernetes.io/projected/0eff5cf7-789d-4dbf-9894-568df3b2ba47-kube-api-access-59wr7\") pod \"0eff5cf7-789d-4dbf-9894-568df3b2ba47\" (UID: \"0eff5cf7-789d-4dbf-9894-568df3b2ba47\") "
Sep 11 19:22:47.335337 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:47.335086    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0eff5cf7-789d-4dbf-9894-568df3b2ba47-istio-envoy\") pod \"0eff5cf7-789d-4dbf-9894-568df3b2ba47\" (UID: \"0eff5cf7-789d-4dbf-9894-568df3b2ba47\") "
Sep 11 19:22:47.335337 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:47.335125    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0eff5cf7-789d-4dbf-9894-568df3b2ba47-workload-socket\") pod \"0eff5cf7-789d-4dbf-9894-568df3b2ba47\" (UID: \"0eff5cf7-789d-4dbf-9894-568df3b2ba47\") "
Sep 11 19:22:47.335337 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:47.335159    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0eff5cf7-789d-4dbf-9894-568df3b2ba47-istio-token\") pod \"0eff5cf7-789d-4dbf-9894-568df3b2ba47\" (UID: \"0eff5cf7-789d-4dbf-9894-568df3b2ba47\") "
Sep 11 19:22:47.335337 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:47.335199    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0eff5cf7-789d-4dbf-9894-568df3b2ba47-istiod-ca-cert\") pod \"0eff5cf7-789d-4dbf-9894-568df3b2ba47\" (UID: \"0eff5cf7-789d-4dbf-9894-568df3b2ba47\") "
Sep 11 19:22:47.335337 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:47.335220    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0eff5cf7-789d-4dbf-9894-568df3b2ba47-istio-data\") pod \"0eff5cf7-789d-4dbf-9894-568df3b2ba47\" (UID: \"0eff5cf7-789d-4dbf-9894-568df3b2ba47\") "
Sep 11 19:22:47.335337 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:47.335246    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0eff5cf7-789d-4dbf-9894-568df3b2ba47-istio-podinfo\") pod \"0eff5cf7-789d-4dbf-9894-568df3b2ba47\" (UID: \"0eff5cf7-789d-4dbf-9894-568df3b2ba47\") "
Sep 11 19:22:47.335337 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:47.335290    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0eff5cf7-789d-4dbf-9894-568df3b2ba47-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0eff5cf7-789d-4dbf-9894-568df3b2ba47" (UID: "0eff5cf7-789d-4dbf-9894-568df3b2ba47"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:22:47.335337 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:47.335312    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0eff5cf7-789d-4dbf-9894-568df3b2ba47-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0eff5cf7-789d-4dbf-9894-568df3b2ba47" (UID: "0eff5cf7-789d-4dbf-9894-568df3b2ba47"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:22:47.336076 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:47.335441    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0eff5cf7-789d-4dbf-9894-568df3b2ba47-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0eff5cf7-789d-4dbf-9894-568df3b2ba47" (UID: "0eff5cf7-789d-4dbf-9894-568df3b2ba47"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:22:47.336076 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:47.335524    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0eff5cf7-789d-4dbf-9894-568df3b2ba47-istio-data" (OuterVolumeSpecName: "istio-data") pod "0eff5cf7-789d-4dbf-9894-568df3b2ba47" (UID: "0eff5cf7-789d-4dbf-9894-568df3b2ba47"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:22:47.336076 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:47.335627    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0eff5cf7-789d-4dbf-9894-568df3b2ba47-workload-certs\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:22:47.336076 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:47.335649    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0eff5cf7-789d-4dbf-9894-568df3b2ba47-workload-socket\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:22:47.336076 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:47.335663    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0eff5cf7-789d-4dbf-9894-568df3b2ba47-istio-data\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:22:47.336076 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:47.335666    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0eff5cf7-789d-4dbf-9894-568df3b2ba47-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0eff5cf7-789d-4dbf-9894-568df3b2ba47" (UID: "0eff5cf7-789d-4dbf-9894-568df3b2ba47"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 19:22:47.336076 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:47.335677    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0eff5cf7-789d-4dbf-9894-568df3b2ba47-credential-socket\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:22:47.337583 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:47.337559    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0eff5cf7-789d-4dbf-9894-568df3b2ba47-istio-token" (OuterVolumeSpecName: "istio-token") pod "0eff5cf7-789d-4dbf-9894-568df3b2ba47" (UID: "0eff5cf7-789d-4dbf-9894-568df3b2ba47"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:22:47.337708 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:47.337613    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0eff5cf7-789d-4dbf-9894-568df3b2ba47-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0eff5cf7-789d-4dbf-9894-568df3b2ba47" (UID: "0eff5cf7-789d-4dbf-9894-568df3b2ba47"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 19:22:47.337708 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:47.337634    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0eff5cf7-789d-4dbf-9894-568df3b2ba47-kube-api-access-59wr7" (OuterVolumeSpecName: "kube-api-access-59wr7") pod "0eff5cf7-789d-4dbf-9894-568df3b2ba47" (UID: "0eff5cf7-789d-4dbf-9894-568df3b2ba47"). InnerVolumeSpecName "kube-api-access-59wr7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:22:47.337708 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:47.337652    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0eff5cf7-789d-4dbf-9894-568df3b2ba47-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0eff5cf7-789d-4dbf-9894-568df3b2ba47" (UID: "0eff5cf7-789d-4dbf-9894-568df3b2ba47"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:22:47.436621 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:47.436562    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59wr7\" (UniqueName: \"kubernetes.io/projected/0eff5cf7-789d-4dbf-9894-568df3b2ba47-kube-api-access-59wr7\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:22:47.436813 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:47.436643    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0eff5cf7-789d-4dbf-9894-568df3b2ba47-istio-envoy\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:22:47.436813 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:47.436655    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0eff5cf7-789d-4dbf-9894-568df3b2ba47-istio-token\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:22:47.436813 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:47.436664    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0eff5cf7-789d-4dbf-9894-568df3b2ba47-istiod-ca-cert\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:22:47.436813 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:47.436671    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0eff5cf7-789d-4dbf-9894-568df3b2ba47-istio-podinfo\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:22:47.507264 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:47.507227    2578 generic.go:356] "Generic (PLEG): container finished" podID="0eff5cf7-789d-4dbf-9894-568df3b2ba47" containerID="6100cc7be28085a781dc7153cc6e10dc7ddcb9676cb77e7a9b1cf4b720946445" exitCode=0
Sep 11 19:22:47.507468 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:47.507302    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-3b7391-openshift-default-7cd68cb6bd-kwtt7"
Sep 11 19:22:47.507468 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:47.507314    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-3b7391-openshift-default-7cd68cb6bd-kwtt7" event={"ID":"0eff5cf7-789d-4dbf-9894-568df3b2ba47","Type":"ContainerDied","Data":"6100cc7be28085a781dc7153cc6e10dc7ddcb9676cb77e7a9b1cf4b720946445"}
Sep 11 19:22:47.507468 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:47.507380    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-3b7391-openshift-default-7cd68cb6bd-kwtt7" event={"ID":"0eff5cf7-789d-4dbf-9894-568df3b2ba47","Type":"ContainerDied","Data":"09babab9da5ac7c41f9126300f1a457567f43559d14485784f2912318e77eb3a"}
Sep 11 19:22:47.507468 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:47.507404    2578 scope.go:117] "RemoveContainer" containerID="6100cc7be28085a781dc7153cc6e10dc7ddcb9676cb77e7a9b1cf4b720946445"
Sep 11 19:22:47.516650 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:47.516619    2578 scope.go:117] "RemoveContainer" containerID="f0218df372cea8a1db10419b3e77115741f0fdbb3c28f548de95847dc679d8f7"
Sep 11 19:22:47.524797 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:47.524776    2578 scope.go:117] "RemoveContainer" containerID="6100cc7be28085a781dc7153cc6e10dc7ddcb9676cb77e7a9b1cf4b720946445"
Sep 11 19:22:47.525076 ip-10-0-134-189 kubenswrapper[2578]: E0911 19:22:47.525058    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6100cc7be28085a781dc7153cc6e10dc7ddcb9676cb77e7a9b1cf4b720946445\": container with ID starting with 6100cc7be28085a781dc7153cc6e10dc7ddcb9676cb77e7a9b1cf4b720946445 not found: ID does not exist" containerID="6100cc7be28085a781dc7153cc6e10dc7ddcb9676cb77e7a9b1cf4b720946445"
Sep 11 19:22:47.525150 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:47.525084    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6100cc7be28085a781dc7153cc6e10dc7ddcb9676cb77e7a9b1cf4b720946445"} err="failed to get container status \"6100cc7be28085a781dc7153cc6e10dc7ddcb9676cb77e7a9b1cf4b720946445\": rpc error: code = NotFound desc = could not find container \"6100cc7be28085a781dc7153cc6e10dc7ddcb9676cb77e7a9b1cf4b720946445\": container with ID starting with 6100cc7be28085a781dc7153cc6e10dc7ddcb9676cb77e7a9b1cf4b720946445 not found: ID does not exist"
Sep 11 19:22:47.525150 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:47.525101    2578 scope.go:117] "RemoveContainer" containerID="f0218df372cea8a1db10419b3e77115741f0fdbb3c28f548de95847dc679d8f7"
Sep 11 19:22:47.525339 ip-10-0-134-189 kubenswrapper[2578]: E0911 19:22:47.525321    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0218df372cea8a1db10419b3e77115741f0fdbb3c28f548de95847dc679d8f7\": container with ID starting with f0218df372cea8a1db10419b3e77115741f0fdbb3c28f548de95847dc679d8f7 not found: ID does not exist" containerID="f0218df372cea8a1db10419b3e77115741f0fdbb3c28f548de95847dc679d8f7"
Sep 11 19:22:47.525398 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:47.525346    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0218df372cea8a1db10419b3e77115741f0fdbb3c28f548de95847dc679d8f7"} err="failed to get container status \"f0218df372cea8a1db10419b3e77115741f0fdbb3c28f548de95847dc679d8f7\": rpc error: code = NotFound desc = could not find container \"f0218df372cea8a1db10419b3e77115741f0fdbb3c28f548de95847dc679d8f7\": container with ID starting with f0218df372cea8a1db10419b3e77115741f0fdbb3c28f548de95847dc679d8f7 not found: ID does not exist"
Sep 11 19:22:47.527551 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:47.527529    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-3b7391-openshift-default-7cd68cb6bd-kwtt7"]
Sep 11 19:22:47.530879 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:47.530858    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-3b7391-openshift-default-7cd68cb6bd-kwtt7"]
Sep 11 19:22:48.396352 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:22:48.396319    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0eff5cf7-789d-4dbf-9894-568df3b2ba47" path="/var/lib/kubelet/pods/0eff5cf7-789d-4dbf-9894-568df3b2ba47/volumes"
Sep 11 19:23:44.461543 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:23:44.461511    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-hsbsf_d0880abe-ec36-48e4-a511-ec663873fffa/console-operator/1.log"
Sep 11 19:23:44.468972 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:23:44.468945    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-hsbsf_d0880abe-ec36-48e4-a511-ec663873fffa/console-operator/1.log"
Sep 11 19:26:44.889488 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:26:44.889451    2578 scope.go:117] "RemoveContainer" containerID="b2722492336c270deadf4fa9dc5c1a1c6368e87019d4e900ebbbc135a72973aa"
Sep 11 19:26:44.897961 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:26:44.897939    2578 scope.go:117] "RemoveContainer" containerID="88882d904f2de2effb2af1e6cc53146357d1b29005d84c3fd25673b76e8125e8"
Sep 11 19:27:19.930991 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:27:19.930956    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-cf8f6f1d-kserve-76d676cf7f-774jx"]
Sep 11 19:27:19.931452 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:27:19.931252    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b812694-cf9d-4ae3-9b86-f48bd716c4c9" containerName="revoke-keys"
Sep 11 19:27:19.931452 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:27:19.931262    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b812694-cf9d-4ae3-9b86-f48bd716c4c9" containerName="revoke-keys"
Sep 11 19:27:19.931452 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:27:19.931277    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2bfe0225-fb2e-4616-ba97-26f83d861f1e" containerName="istio-proxy"
Sep 11 19:27:19.931452 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:27:19.931282    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2bfe0225-fb2e-4616-ba97-26f83d861f1e" containerName="istio-proxy"
Sep 11 19:27:19.931452 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:27:19.931291    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b3672a7-8dfe-47e5-b418-48bb90d73622" containerName="payload-pre-processing"
Sep 11 19:27:19.931452 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:27:19.931296    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b3672a7-8dfe-47e5-b418-48bb90d73622" containerName="payload-pre-processing"
Sep 11 19:27:19.931452 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:27:19.931303    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0eff5cf7-789d-4dbf-9894-568df3b2ba47" containerName="istio-proxy"
Sep 11 19:27:19.931452 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:27:19.931308    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0eff5cf7-789d-4dbf-9894-568df3b2ba47" containerName="istio-proxy"
Sep 11 19:27:19.931452 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:27:19.931360    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b3672a7-8dfe-47e5-b418-48bb90d73622" containerName="payload-pre-processing"
Sep 11 19:27:19.931452 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:27:19.931368    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="0eff5cf7-789d-4dbf-9894-568df3b2ba47" containerName="istio-proxy"
Sep 11 19:27:19.931452 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:27:19.931375    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b812694-cf9d-4ae3-9b86-f48bd716c4c9" containerName="revoke-keys"
Sep 11 19:27:19.931452 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:27:19.931380    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2bfe0225-fb2e-4616-ba97-26f83d861f1e" containerName="istio-proxy"
Sep 11 19:27:19.931452 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:27:19.931387    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2bfe0225-fb2e-4616-ba97-26f83d861f1e" containerName="istio-proxy"
Sep 11 19:27:19.931452 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:27:19.931393    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="0eff5cf7-789d-4dbf-9894-568df3b2ba47" containerName="istio-proxy"
Sep 11 19:27:19.934092 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:27:19.934076    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-cf8f6f1d-kserve-76d676cf7f-774jx"
Sep 11 19:27:19.936019 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:27:19.936001    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-cf8f6f1d-kserve-self-signed-certs\""
Sep 11 19:27:19.942791 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:27:19.942770    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-cf8f6f1d-kserve-76d676cf7f-774jx"]
Sep 11 19:27:20.040771 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:27:20.040734    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22679cb8-d0b4-463c-8bba-84b34958b3f6-home\") pod \"e2e-same-id-a-cf8f6f1d-kserve-76d676cf7f-774jx\" (UID: \"22679cb8-d0b4-463c-8bba-84b34958b3f6\") " pod="llm/e2e-same-id-a-cf8f6f1d-kserve-76d676cf7f-774jx"
Sep 11 19:27:20.040963 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:27:20.040794    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22679cb8-d0b4-463c-8bba-84b34958b3f6-tmp-dir\") pod \"e2e-same-id-a-cf8f6f1d-kserve-76d676cf7f-774jx\" (UID: \"22679cb8-d0b4-463c-8bba-84b34958b3f6\") " pod="llm/e2e-same-id-a-cf8f6f1d-kserve-76d676cf7f-774jx"
Sep 11 19:27:20.040963 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:27:20.040846    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22679cb8-d0b4-463c-8bba-84b34958b3f6-model-cache\") pod \"e2e-same-id-a-cf8f6f1d-kserve-76d676cf7f-774jx\" (UID: \"22679cb8-d0b4-463c-8bba-84b34958b3f6\") " pod="llm/e2e-same-id-a-cf8f6f1d-kserve-76d676cf7f-774jx"
Sep 11 19:27:20.040963 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:27:20.040860    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22679cb8-d0b4-463c-8bba-84b34958b3f6-tls-certs\") pod \"e2e-same-id-a-cf8f6f1d-kserve-76d676cf7f-774jx\" (UID: \"22679cb8-d0b4-463c-8bba-84b34958b3f6\") " pod="llm/e2e-same-id-a-cf8f6f1d-kserve-76d676cf7f-774jx"
Sep 11 19:27:20.040963 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:27:20.040887    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22679cb8-d0b4-463c-8bba-84b34958b3f6-dshm\") pod \"e2e-same-id-a-cf8f6f1d-kserve-76d676cf7f-774jx\" (UID: \"22679cb8-d0b4-463c-8bba-84b34958b3f6\") " pod="llm/e2e-same-id-a-cf8f6f1d-kserve-76d676cf7f-774jx"
Sep 11 19:27:20.040963 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:27:20.040945    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwr7k\" (UniqueName: \"kubernetes.io/projected/22679cb8-d0b4-463c-8bba-84b34958b3f6-kube-api-access-bwr7k\") pod \"e2e-same-id-a-cf8f6f1d-kserve-76d676cf7f-774jx\" (UID: \"22679cb8-d0b4-463c-8bba-84b34958b3f6\") " pod="llm/e2e-same-id-a-cf8f6f1d-kserve-76d676cf7f-774jx"
Sep 11 19:27:20.141702 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:27:20.141665    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22679cb8-d0b4-463c-8bba-84b34958b3f6-home\") pod \"e2e-same-id-a-cf8f6f1d-kserve-76d676cf7f-774jx\" (UID: \"22679cb8-d0b4-463c-8bba-84b34958b3f6\") " pod="llm/e2e-same-id-a-cf8f6f1d-kserve-76d676cf7f-774jx"
Sep 11 19:27:20.141881 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:27:20.141719    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22679cb8-d0b4-463c-8bba-84b34958b3f6-tmp-dir\") pod \"e2e-same-id-a-cf8f6f1d-kserve-76d676cf7f-774jx\" (UID: \"22679cb8-d0b4-463c-8bba-84b34958b3f6\") " pod="llm/e2e-same-id-a-cf8f6f1d-kserve-76d676cf7f-774jx"
Sep 11 19:27:20.141881 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:27:20.141752    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22679cb8-d0b4-463c-8bba-84b34958b3f6-model-cache\") pod \"e2e-same-id-a-cf8f6f1d-kserve-76d676cf7f-774jx\" (UID: \"22679cb8-d0b4-463c-8bba-84b34958b3f6\") " pod="llm/e2e-same-id-a-cf8f6f1d-kserve-76d676cf7f-774jx"
Sep 11 19:27:20.141881 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:27:20.141766    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22679cb8-d0b4-463c-8bba-84b34958b3f6-tls-certs\") pod \"e2e-same-id-a-cf8f6f1d-kserve-76d676cf7f-774jx\" (UID: \"22679cb8-d0b4-463c-8bba-84b34958b3f6\") " pod="llm/e2e-same-id-a-cf8f6f1d-kserve-76d676cf7f-774jx"
Sep 11 19:27:20.141881 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:27:20.141790    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22679cb8-d0b4-463c-8bba-84b34958b3f6-dshm\") pod \"e2e-same-id-a-cf8f6f1d-kserve-76d676cf7f-774jx\" (UID: \"22679cb8-d0b4-463c-8bba-84b34958b3f6\") " pod="llm/e2e-same-id-a-cf8f6f1d-kserve-76d676cf7f-774jx"
Sep 11 19:27:20.141881 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:27:20.141811    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwr7k\" (UniqueName: \"kubernetes.io/projected/22679cb8-d0b4-463c-8bba-84b34958b3f6-kube-api-access-bwr7k\") pod \"e2e-same-id-a-cf8f6f1d-kserve-76d676cf7f-774jx\" (UID: \"22679cb8-d0b4-463c-8bba-84b34958b3f6\") " pod="llm/e2e-same-id-a-cf8f6f1d-kserve-76d676cf7f-774jx"
Sep 11 19:27:20.142143 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:27:20.142118    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22679cb8-d0b4-463c-8bba-84b34958b3f6-home\") pod \"e2e-same-id-a-cf8f6f1d-kserve-76d676cf7f-774jx\" (UID: \"22679cb8-d0b4-463c-8bba-84b34958b3f6\") " pod="llm/e2e-same-id-a-cf8f6f1d-kserve-76d676cf7f-774jx"
Sep 11 19:27:20.142194 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:27:20.142143    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22679cb8-d0b4-463c-8bba-84b34958b3f6-model-cache\") pod \"e2e-same-id-a-cf8f6f1d-kserve-76d676cf7f-774jx\" (UID: \"22679cb8-d0b4-463c-8bba-84b34958b3f6\") " pod="llm/e2e-same-id-a-cf8f6f1d-kserve-76d676cf7f-774jx"
Sep 11 19:27:20.142194 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:27:20.142186    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22679cb8-d0b4-463c-8bba-84b34958b3f6-tmp-dir\") pod \"e2e-same-id-a-cf8f6f1d-kserve-76d676cf7f-774jx\" (UID: \"22679cb8-d0b4-463c-8bba-84b34958b3f6\") " pod="llm/e2e-same-id-a-cf8f6f1d-kserve-76d676cf7f-774jx"
Sep 11 19:27:20.144212 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:27:20.144189    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22679cb8-d0b4-463c-8bba-84b34958b3f6-dshm\") pod \"e2e-same-id-a-cf8f6f1d-kserve-76d676cf7f-774jx\" (UID: \"22679cb8-d0b4-463c-8bba-84b34958b3f6\") " pod="llm/e2e-same-id-a-cf8f6f1d-kserve-76d676cf7f-774jx"
Sep 11 19:27:20.144374 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:27:20.144354    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22679cb8-d0b4-463c-8bba-84b34958b3f6-tls-certs\") pod \"e2e-same-id-a-cf8f6f1d-kserve-76d676cf7f-774jx\" (UID: \"22679cb8-d0b4-463c-8bba-84b34958b3f6\") " pod="llm/e2e-same-id-a-cf8f6f1d-kserve-76d676cf7f-774jx"
Sep 11 19:27:20.150184 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:27:20.150163    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwr7k\" (UniqueName: \"kubernetes.io/projected/22679cb8-d0b4-463c-8bba-84b34958b3f6-kube-api-access-bwr7k\") pod \"e2e-same-id-a-cf8f6f1d-kserve-76d676cf7f-774jx\" (UID: \"22679cb8-d0b4-463c-8bba-84b34958b3f6\") " pod="llm/e2e-same-id-a-cf8f6f1d-kserve-76d676cf7f-774jx"
Sep 11 19:27:20.244581 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:27:20.244531    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-cf8f6f1d-kserve-76d676cf7f-774jx"
Sep 11 19:27:20.521276 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:27:20.521247    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-cf8f6f1d-kserve-76d676cf7f-774jx"]
Sep 11 19:27:20.523186 ip-10-0-134-189 kubenswrapper[2578]: W0911 19:27:20.523154    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod22679cb8_d0b4_463c_8bba_84b34958b3f6.slice/crio-e0d9f0c8ac952387b2a42ba4f1c4915ffa3a4c5e6209f919f6a86133abbf7837 WatchSource:0}: Error finding container e0d9f0c8ac952387b2a42ba4f1c4915ffa3a4c5e6209f919f6a86133abbf7837: Status 404 returned error can't find the container with id e0d9f0c8ac952387b2a42ba4f1c4915ffa3a4c5e6209f919f6a86133abbf7837
Sep 11 19:27:20.749142 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:27:20.749106    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-cf8f6f1d-kserve-76d676cf7f-774jx" event={"ID":"22679cb8-d0b4-463c-8bba-84b34958b3f6","Type":"ContainerStarted","Data":"eaf401b1a2c78913976b696b09410b0a4a6a29043180209aebb343e227317603"}
Sep 11 19:27:20.749308 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:27:20.749151    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-cf8f6f1d-kserve-76d676cf7f-774jx" event={"ID":"22679cb8-d0b4-463c-8bba-84b34958b3f6","Type":"ContainerStarted","Data":"e0d9f0c8ac952387b2a42ba4f1c4915ffa3a4c5e6209f919f6a86133abbf7837"}
Sep 11 19:27:20.766057 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:27:20.766009    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-cf8f6f1d-kserve-76d676cf7f-774jx" podStartSLOduration=1.765995483 podStartE2EDuration="1.765995483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:27:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:27:20.764445081 +0000 UTC m=+2316.842035649" watchObservedRunningTime="2026-09-11 19:27:20.765995483 +0000 UTC m=+2316.843586038"
Sep 11 19:27:23.025801 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:27:23.025773    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-cf8f6f1d-kserve-855bc75c6c-vbwdw"]
Sep 11 19:27:23.026181 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:27:23.026088    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2bfe0225-fb2e-4616-ba97-26f83d861f1e" containerName="istio-proxy"
Sep 11 19:27:23.026181 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:27:23.026103    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2bfe0225-fb2e-4616-ba97-26f83d861f1e" containerName="istio-proxy"
Sep 11 19:27:23.026181 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:27:23.026116    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0eff5cf7-789d-4dbf-9894-568df3b2ba47" containerName="istio-proxy"
Sep 11 19:27:23.026181 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:27:23.026122    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0eff5cf7-789d-4dbf-9894-568df3b2ba47" containerName="istio-proxy"
Sep 11 19:27:23.029157 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:27:23.029139    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-cf8f6f1d-kserve-855bc75c6c-vbwdw"
Sep 11 19:27:23.030998 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:27:23.030978    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-cf8f6f1d-kserve-self-signed-certs\""
Sep 11 19:27:23.036883 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:27:23.036861    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-cf8f6f1d-kserve-855bc75c6c-vbwdw"]
Sep 11 19:27:23.070881 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:27:23.070856    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e2ba1ea-a352-491f-898e-507a6f40b2cf-home\") pod \"e2e-same-id-b-cf8f6f1d-kserve-855bc75c6c-vbwdw\" (UID: \"7e2ba1ea-a352-491f-898e-507a6f40b2cf\") " pod="llm/e2e-same-id-b-cf8f6f1d-kserve-855bc75c6c-vbwdw"
Sep 11 19:27:23.071028 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:27:23.070890    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thzvt\" (UniqueName: \"kubernetes.io/projected/7e2ba1ea-a352-491f-898e-507a6f40b2cf-kube-api-access-thzvt\") pod \"e2e-same-id-b-cf8f6f1d-kserve-855bc75c6c-vbwdw\" (UID: \"7e2ba1ea-a352-491f-898e-507a6f40b2cf\") " pod="llm/e2e-same-id-b-cf8f6f1d-kserve-855bc75c6c-vbwdw"
Sep 11 19:27:23.071028 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:27:23.070911    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e2ba1ea-a352-491f-898e-507a6f40b2cf-model-cache\") pod \"e2e-same-id-b-cf8f6f1d-kserve-855bc75c6c-vbwdw\" (UID: \"7e2ba1ea-a352-491f-898e-507a6f40b2cf\") " pod="llm/e2e-same-id-b-cf8f6f1d-kserve-855bc75c6c-vbwdw"
Sep 11 19:27:23.071028 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:27:23.070991    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e2ba1ea-a352-491f-898e-507a6f40b2cf-tmp-dir\") pod \"e2e-same-id-b-cf8f6f1d-kserve-855bc75c6c-vbwdw\" (UID: \"7e2ba1ea-a352-491f-898e-507a6f40b2cf\") " pod="llm/e2e-same-id-b-cf8f6f1d-kserve-855bc75c6c-vbwdw"
Sep 11 19:27:23.071028 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:27:23.071022    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e2ba1ea-a352-491f-898e-507a6f40b2cf-tls-certs\") pod \"e2e-same-id-b-cf8f6f1d-kserve-855bc75c6c-vbwdw\" (UID: \"7e2ba1ea-a352-491f-898e-507a6f40b2cf\") " pod="llm/e2e-same-id-b-cf8f6f1d-kserve-855bc75c6c-vbwdw"
Sep 11 19:27:23.071189 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:27:23.071077    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e2ba1ea-a352-491f-898e-507a6f40b2cf-dshm\") pod \"e2e-same-id-b-cf8f6f1d-kserve-855bc75c6c-vbwdw\" (UID: \"7e2ba1ea-a352-491f-898e-507a6f40b2cf\") " pod="llm/e2e-same-id-b-cf8f6f1d-kserve-855bc75c6c-vbwdw"
Sep 11 19:27:23.172181 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:27:23.172149    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thzvt\" (UniqueName: \"kubernetes.io/projected/7e2ba1ea-a352-491f-898e-507a6f40b2cf-kube-api-access-thzvt\") pod \"e2e-same-id-b-cf8f6f1d-kserve-855bc75c6c-vbwdw\" (UID: \"7e2ba1ea-a352-491f-898e-507a6f40b2cf\") " pod="llm/e2e-same-id-b-cf8f6f1d-kserve-855bc75c6c-vbwdw"
Sep 11 19:27:23.172181 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:27:23.172188    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e2ba1ea-a352-491f-898e-507a6f40b2cf-model-cache\") pod \"e2e-same-id-b-cf8f6f1d-kserve-855bc75c6c-vbwdw\" (UID: \"7e2ba1ea-a352-491f-898e-507a6f40b2cf\") " pod="llm/e2e-same-id-b-cf8f6f1d-kserve-855bc75c6c-vbwdw"
Sep 11 19:27:23.172429 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:27:23.172226    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e2ba1ea-a352-491f-898e-507a6f40b2cf-tmp-dir\") pod \"e2e-same-id-b-cf8f6f1d-kserve-855bc75c6c-vbwdw\" (UID: \"7e2ba1ea-a352-491f-898e-507a6f40b2cf\") " pod="llm/e2e-same-id-b-cf8f6f1d-kserve-855bc75c6c-vbwdw"
Sep 11 19:27:23.172429 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:27:23.172245    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e2ba1ea-a352-491f-898e-507a6f40b2cf-tls-certs\") pod \"e2e-same-id-b-cf8f6f1d-kserve-855bc75c6c-vbwdw\" (UID: \"7e2ba1ea-a352-491f-898e-507a6f40b2cf\") " pod="llm/e2e-same-id-b-cf8f6f1d-kserve-855bc75c6c-vbwdw"
Sep 11 19:27:23.172429 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:27:23.172275    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e2ba1ea-a352-491f-898e-507a6f40b2cf-dshm\") pod \"e2e-same-id-b-cf8f6f1d-kserve-855bc75c6c-vbwdw\" (UID: \"7e2ba1ea-a352-491f-898e-507a6f40b2cf\") " pod="llm/e2e-same-id-b-cf8f6f1d-kserve-855bc75c6c-vbwdw"
Sep 11 19:27:23.172429 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:27:23.172320    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e2ba1ea-a352-491f-898e-507a6f40b2cf-home\") pod \"e2e-same-id-b-cf8f6f1d-kserve-855bc75c6c-vbwdw\" (UID: \"7e2ba1ea-a352-491f-898e-507a6f40b2cf\") " pod="llm/e2e-same-id-b-cf8f6f1d-kserve-855bc75c6c-vbwdw"
Sep 11 19:27:23.172694 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:27:23.172669    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e2ba1ea-a352-491f-898e-507a6f40b2cf-tmp-dir\") pod \"e2e-same-id-b-cf8f6f1d-kserve-855bc75c6c-vbwdw\" (UID: \"7e2ba1ea-a352-491f-898e-507a6f40b2cf\") " pod="llm/e2e-same-id-b-cf8f6f1d-kserve-855bc75c6c-vbwdw"
Sep 11 19:27:23.172786 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:27:23.172715    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e2ba1ea-a352-491f-898e-507a6f40b2cf-model-cache\") pod \"e2e-same-id-b-cf8f6f1d-kserve-855bc75c6c-vbwdw\" (UID: \"7e2ba1ea-a352-491f-898e-507a6f40b2cf\") " pod="llm/e2e-same-id-b-cf8f6f1d-kserve-855bc75c6c-vbwdw"
Sep 11 19:27:23.172786 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:27:23.172729    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e2ba1ea-a352-491f-898e-507a6f40b2cf-home\") pod \"e2e-same-id-b-cf8f6f1d-kserve-855bc75c6c-vbwdw\" (UID: \"7e2ba1ea-a352-491f-898e-507a6f40b2cf\") " pod="llm/e2e-same-id-b-cf8f6f1d-kserve-855bc75c6c-vbwdw"
Sep 11 19:27:23.174748 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:27:23.174726    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e2ba1ea-a352-491f-898e-507a6f40b2cf-dshm\") pod \"e2e-same-id-b-cf8f6f1d-kserve-855bc75c6c-vbwdw\" (UID: \"7e2ba1ea-a352-491f-898e-507a6f40b2cf\") " pod="llm/e2e-same-id-b-cf8f6f1d-kserve-855bc75c6c-vbwdw"
Sep 11 19:27:23.174877 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:27:23.174855    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e2ba1ea-a352-491f-898e-507a6f40b2cf-tls-certs\") pod \"e2e-same-id-b-cf8f6f1d-kserve-855bc75c6c-vbwdw\" (UID: \"7e2ba1ea-a352-491f-898e-507a6f40b2cf\") " pod="llm/e2e-same-id-b-cf8f6f1d-kserve-855bc75c6c-vbwdw"
Sep 11 19:27:23.179345 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:27:23.179327    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-thzvt\" (UniqueName: \"kubernetes.io/projected/7e2ba1ea-a352-491f-898e-507a6f40b2cf-kube-api-access-thzvt\") pod \"e2e-same-id-b-cf8f6f1d-kserve-855bc75c6c-vbwdw\" (UID: \"7e2ba1ea-a352-491f-898e-507a6f40b2cf\") " pod="llm/e2e-same-id-b-cf8f6f1d-kserve-855bc75c6c-vbwdw"
Sep 11 19:27:23.340663 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:27:23.340548    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-cf8f6f1d-kserve-855bc75c6c-vbwdw"
Sep 11 19:27:23.613034 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:27:23.612953    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-cf8f6f1d-kserve-855bc75c6c-vbwdw"]
Sep 11 19:27:23.616433 ip-10-0-134-189 kubenswrapper[2578]: W0911 19:27:23.616401    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7e2ba1ea_a352_491f_898e_507a6f40b2cf.slice/crio-2b2ab40df1d69306cddf02b52b7b7d9c0edae8c3b71d7526c1654f01cb13835d WatchSource:0}: Error finding container 2b2ab40df1d69306cddf02b52b7b7d9c0edae8c3b71d7526c1654f01cb13835d: Status 404 returned error can't find the container with id 2b2ab40df1d69306cddf02b52b7b7d9c0edae8c3b71d7526c1654f01cb13835d
Sep 11 19:27:23.763676 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:27:23.763635    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-cf8f6f1d-kserve-855bc75c6c-vbwdw" event={"ID":"7e2ba1ea-a352-491f-898e-507a6f40b2cf","Type":"ContainerStarted","Data":"6ed024d945d2fa5545a6d2882dd42aa55d1a6dcee5d25e110b929d0bff5d9a4e"}
Sep 11 19:27:23.763845 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:27:23.763693    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-cf8f6f1d-kserve-855bc75c6c-vbwdw" event={"ID":"7e2ba1ea-a352-491f-898e-507a6f40b2cf","Type":"ContainerStarted","Data":"2b2ab40df1d69306cddf02b52b7b7d9c0edae8c3b71d7526c1654f01cb13835d"}
Sep 11 19:27:23.780657 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:27:23.780587    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-cf8f6f1d-kserve-855bc75c6c-vbwdw" podStartSLOduration=0.7805727 podStartE2EDuration="780.5727ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:27:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:27:23.778816555 +0000 UTC m=+2319.856407129" watchObservedRunningTime="2026-09-11 19:27:23.7805727 +0000 UTC m=+2319.858163255"
Sep 11 19:27:30.245116 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:27:30.245076    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-cf8f6f1d-kserve-76d676cf7f-774jx"
Sep 11 19:27:30.245639 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:27:30.245130    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-cf8f6f1d-kserve-76d676cf7f-774jx"
Sep 11 19:27:30.257484 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:27:30.257460    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-cf8f6f1d-kserve-76d676cf7f-774jx"
Sep 11 19:27:30.806675 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:27:30.806645    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-cf8f6f1d-kserve-76d676cf7f-774jx"
Sep 11 19:27:33.341373 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:27:33.341336    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-cf8f6f1d-kserve-855bc75c6c-vbwdw"
Sep 11 19:27:33.341804 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:27:33.341389    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-cf8f6f1d-kserve-855bc75c6c-vbwdw"
Sep 11 19:27:33.353800 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:27:33.353776    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-cf8f6f1d-kserve-855bc75c6c-vbwdw"
Sep 11 19:27:33.819150 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:27:33.819120    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-cf8f6f1d-kserve-855bc75c6c-vbwdw"
Sep 11 19:27:51.346213 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:27:51.346135    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-cf8f6f1d-kserve-76d676cf7f-774jx"]
Sep 11 19:27:53.012471 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:27:53.012438    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-cf8f6f1d-kserve-855bc75c6c-vbwdw"]
Sep 11 19:28:06.365719 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:28:06.365676    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-cf8f6f1d-kserve-76d676cf7f-774jx" podUID="22679cb8-d0b4-463c-8bba-84b34958b3f6" containerName="main" containerID="cri-o://eaf401b1a2c78913976b696b09410b0a4a6a29043180209aebb343e227317603" gracePeriod=45
Sep 11 19:28:06.610204 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:28:06.610180    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-cf8f6f1d-kserve-76d676cf7f-774jx"
Sep 11 19:28:06.642351 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:28:06.642272    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bwr7k\" (UniqueName: \"kubernetes.io/projected/22679cb8-d0b4-463c-8bba-84b34958b3f6-kube-api-access-bwr7k\") pod \"22679cb8-d0b4-463c-8bba-84b34958b3f6\" (UID: \"22679cb8-d0b4-463c-8bba-84b34958b3f6\") "
Sep 11 19:28:06.642351 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:28:06.642309    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22679cb8-d0b4-463c-8bba-84b34958b3f6-home\") pod \"22679cb8-d0b4-463c-8bba-84b34958b3f6\" (UID: \"22679cb8-d0b4-463c-8bba-84b34958b3f6\") "
Sep 11 19:28:06.642351 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:28:06.642331    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22679cb8-d0b4-463c-8bba-84b34958b3f6-dshm\") pod \"22679cb8-d0b4-463c-8bba-84b34958b3f6\" (UID: \"22679cb8-d0b4-463c-8bba-84b34958b3f6\") "
Sep 11 19:28:06.642639 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:28:06.642349    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22679cb8-d0b4-463c-8bba-84b34958b3f6-tmp-dir\") pod \"22679cb8-d0b4-463c-8bba-84b34958b3f6\" (UID: \"22679cb8-d0b4-463c-8bba-84b34958b3f6\") "
Sep 11 19:28:06.642639 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:28:06.642393    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22679cb8-d0b4-463c-8bba-84b34958b3f6-model-cache\") pod \"22679cb8-d0b4-463c-8bba-84b34958b3f6\" (UID: \"22679cb8-d0b4-463c-8bba-84b34958b3f6\") "
Sep 11 19:28:06.642639 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:28:06.642458    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22679cb8-d0b4-463c-8bba-84b34958b3f6-tls-certs\") pod \"22679cb8-d0b4-463c-8bba-84b34958b3f6\" (UID: \"22679cb8-d0b4-463c-8bba-84b34958b3f6\") "
Sep 11 19:28:06.642639 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:28:06.642578    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22679cb8-d0b4-463c-8bba-84b34958b3f6-home" (OuterVolumeSpecName: "home") pod "22679cb8-d0b4-463c-8bba-84b34958b3f6" (UID: "22679cb8-d0b4-463c-8bba-84b34958b3f6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:28:06.642878 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:28:06.642631    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22679cb8-d0b4-463c-8bba-84b34958b3f6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "22679cb8-d0b4-463c-8bba-84b34958b3f6" (UID: "22679cb8-d0b4-463c-8bba-84b34958b3f6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:28:06.642878 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:28:06.642695    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22679cb8-d0b4-463c-8bba-84b34958b3f6-home\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:28:06.642878 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:28:06.642804    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22679cb8-d0b4-463c-8bba-84b34958b3f6-model-cache" (OuterVolumeSpecName: "model-cache") pod "22679cb8-d0b4-463c-8bba-84b34958b3f6" (UID: "22679cb8-d0b4-463c-8bba-84b34958b3f6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:28:06.644686 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:28:06.644660    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22679cb8-d0b4-463c-8bba-84b34958b3f6-kube-api-access-bwr7k" (OuterVolumeSpecName: "kube-api-access-bwr7k") pod "22679cb8-d0b4-463c-8bba-84b34958b3f6" (UID: "22679cb8-d0b4-463c-8bba-84b34958b3f6"). InnerVolumeSpecName "kube-api-access-bwr7k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:28:06.644790 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:28:06.644692    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22679cb8-d0b4-463c-8bba-84b34958b3f6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "22679cb8-d0b4-463c-8bba-84b34958b3f6" (UID: "22679cb8-d0b4-463c-8bba-84b34958b3f6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 19:28:06.644790 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:28:06.644767    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22679cb8-d0b4-463c-8bba-84b34958b3f6-dshm" (OuterVolumeSpecName: "dshm") pod "22679cb8-d0b4-463c-8bba-84b34958b3f6" (UID: "22679cb8-d0b4-463c-8bba-84b34958b3f6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:28:06.743340 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:28:06.743315    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22679cb8-d0b4-463c-8bba-84b34958b3f6-tls-certs\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:28:06.743340 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:28:06.743340    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwr7k\" (UniqueName: \"kubernetes.io/projected/22679cb8-d0b4-463c-8bba-84b34958b3f6-kube-api-access-bwr7k\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:28:06.743513 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:28:06.743353    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22679cb8-d0b4-463c-8bba-84b34958b3f6-dshm\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:28:06.743513 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:28:06.743366    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22679cb8-d0b4-463c-8bba-84b34958b3f6-tmp-dir\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:28:06.743513 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:28:06.743376    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22679cb8-d0b4-463c-8bba-84b34958b3f6-model-cache\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:28:06.956382 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:28:06.956349    2578 generic.go:356] "Generic (PLEG): container finished" podID="22679cb8-d0b4-463c-8bba-84b34958b3f6" containerID="eaf401b1a2c78913976b696b09410b0a4a6a29043180209aebb343e227317603" exitCode=0
Sep 11 19:28:06.956566 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:28:06.956393    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-cf8f6f1d-kserve-76d676cf7f-774jx"
Sep 11 19:28:06.956566 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:28:06.956396    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-cf8f6f1d-kserve-76d676cf7f-774jx" event={"ID":"22679cb8-d0b4-463c-8bba-84b34958b3f6","Type":"ContainerDied","Data":"eaf401b1a2c78913976b696b09410b0a4a6a29043180209aebb343e227317603"}
Sep 11 19:28:06.956566 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:28:06.956494    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-cf8f6f1d-kserve-76d676cf7f-774jx" event={"ID":"22679cb8-d0b4-463c-8bba-84b34958b3f6","Type":"ContainerDied","Data":"e0d9f0c8ac952387b2a42ba4f1c4915ffa3a4c5e6209f919f6a86133abbf7837"}
Sep 11 19:28:06.956566 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:28:06.956518    2578 scope.go:117] "RemoveContainer" containerID="eaf401b1a2c78913976b696b09410b0a4a6a29043180209aebb343e227317603"
Sep 11 19:28:06.968708 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:28:06.968685    2578 scope.go:117] "RemoveContainer" containerID="eaf401b1a2c78913976b696b09410b0a4a6a29043180209aebb343e227317603"
Sep 11 19:28:06.969001 ip-10-0-134-189 kubenswrapper[2578]: E0911 19:28:06.968983    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eaf401b1a2c78913976b696b09410b0a4a6a29043180209aebb343e227317603\": container with ID starting with eaf401b1a2c78913976b696b09410b0a4a6a29043180209aebb343e227317603 not found: ID does not exist" containerID="eaf401b1a2c78913976b696b09410b0a4a6a29043180209aebb343e227317603"
Sep 11 19:28:06.969059 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:28:06.969010    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eaf401b1a2c78913976b696b09410b0a4a6a29043180209aebb343e227317603"} err="failed to get container status \"eaf401b1a2c78913976b696b09410b0a4a6a29043180209aebb343e227317603\": rpc error: code = NotFound desc = could not find container \"eaf401b1a2c78913976b696b09410b0a4a6a29043180209aebb343e227317603\": container with ID starting with eaf401b1a2c78913976b696b09410b0a4a6a29043180209aebb343e227317603 not found: ID does not exist"
Sep 11 19:28:06.977272 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:28:06.977250    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-cf8f6f1d-kserve-76d676cf7f-774jx"]
Sep 11 19:28:06.984284 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:28:06.984265    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-cf8f6f1d-kserve-76d676cf7f-774jx"]
Sep 11 19:28:08.032174 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:28:08.032134    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-cf8f6f1d-kserve-855bc75c6c-vbwdw" podUID="7e2ba1ea-a352-491f-898e-507a6f40b2cf" containerName="main" containerID="cri-o://6ed024d945d2fa5545a6d2882dd42aa55d1a6dcee5d25e110b929d0bff5d9a4e" gracePeriod=45
Sep 11 19:28:08.278853 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:28:08.278827    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-cf8f6f1d-kserve-855bc75c6c-vbwdw"
Sep 11 19:28:08.358027 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:28:08.357939    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-thzvt\" (UniqueName: \"kubernetes.io/projected/7e2ba1ea-a352-491f-898e-507a6f40b2cf-kube-api-access-thzvt\") pod \"7e2ba1ea-a352-491f-898e-507a6f40b2cf\" (UID: \"7e2ba1ea-a352-491f-898e-507a6f40b2cf\") "
Sep 11 19:28:08.358027 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:28:08.358005    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e2ba1ea-a352-491f-898e-507a6f40b2cf-home\") pod \"7e2ba1ea-a352-491f-898e-507a6f40b2cf\" (UID: \"7e2ba1ea-a352-491f-898e-507a6f40b2cf\") "
Sep 11 19:28:08.358224 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:28:08.358036    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e2ba1ea-a352-491f-898e-507a6f40b2cf-model-cache\") pod \"7e2ba1ea-a352-491f-898e-507a6f40b2cf\" (UID: \"7e2ba1ea-a352-491f-898e-507a6f40b2cf\") "
Sep 11 19:28:08.358270 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:28:08.358237    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e2ba1ea-a352-491f-898e-507a6f40b2cf-home" (OuterVolumeSpecName: "home") pod "7e2ba1ea-a352-491f-898e-507a6f40b2cf" (UID: "7e2ba1ea-a352-491f-898e-507a6f40b2cf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:28:08.358270 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:28:08.358249    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e2ba1ea-a352-491f-898e-507a6f40b2cf-model-cache" (OuterVolumeSpecName: "model-cache") pod "7e2ba1ea-a352-491f-898e-507a6f40b2cf" (UID: "7e2ba1ea-a352-491f-898e-507a6f40b2cf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:28:08.358270 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:28:08.358245    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e2ba1ea-a352-491f-898e-507a6f40b2cf-dshm\") pod \"7e2ba1ea-a352-491f-898e-507a6f40b2cf\" (UID: \"7e2ba1ea-a352-491f-898e-507a6f40b2cf\") "
Sep 11 19:28:08.358381 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:28:08.358303    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e2ba1ea-a352-491f-898e-507a6f40b2cf-tls-certs\") pod \"7e2ba1ea-a352-491f-898e-507a6f40b2cf\" (UID: \"7e2ba1ea-a352-491f-898e-507a6f40b2cf\") "
Sep 11 19:28:08.358381 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:28:08.358331    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e2ba1ea-a352-491f-898e-507a6f40b2cf-tmp-dir\") pod \"7e2ba1ea-a352-491f-898e-507a6f40b2cf\" (UID: \"7e2ba1ea-a352-491f-898e-507a6f40b2cf\") "
Sep 11 19:28:08.358700 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:28:08.358655    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e2ba1ea-a352-491f-898e-507a6f40b2cf-home\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:28:08.358700 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:28:08.358694    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e2ba1ea-a352-491f-898e-507a6f40b2cf-model-cache\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:28:08.358837 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:28:08.358666    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e2ba1ea-a352-491f-898e-507a6f40b2cf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7e2ba1ea-a352-491f-898e-507a6f40b2cf" (UID: "7e2ba1ea-a352-491f-898e-507a6f40b2cf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:28:08.360388 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:28:08.360360    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e2ba1ea-a352-491f-898e-507a6f40b2cf-kube-api-access-thzvt" (OuterVolumeSpecName: "kube-api-access-thzvt") pod "7e2ba1ea-a352-491f-898e-507a6f40b2cf" (UID: "7e2ba1ea-a352-491f-898e-507a6f40b2cf"). InnerVolumeSpecName "kube-api-access-thzvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:28:08.360388 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:28:08.360361    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e2ba1ea-a352-491f-898e-507a6f40b2cf-dshm" (OuterVolumeSpecName: "dshm") pod "7e2ba1ea-a352-491f-898e-507a6f40b2cf" (UID: "7e2ba1ea-a352-491f-898e-507a6f40b2cf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:28:08.360623 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:28:08.360579    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e2ba1ea-a352-491f-898e-507a6f40b2cf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7e2ba1ea-a352-491f-898e-507a6f40b2cf" (UID: "7e2ba1ea-a352-491f-898e-507a6f40b2cf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 19:28:08.396284 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:28:08.396255    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22679cb8-d0b4-463c-8bba-84b34958b3f6" path="/var/lib/kubelet/pods/22679cb8-d0b4-463c-8bba-84b34958b3f6/volumes"
Sep 11 19:28:08.459734 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:28:08.459707    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e2ba1ea-a352-491f-898e-507a6f40b2cf-dshm\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:28:08.459734 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:28:08.459731    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e2ba1ea-a352-491f-898e-507a6f40b2cf-tls-certs\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:28:08.459884 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:28:08.459741    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e2ba1ea-a352-491f-898e-507a6f40b2cf-tmp-dir\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:28:08.459884 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:28:08.459750    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thzvt\" (UniqueName: \"kubernetes.io/projected/7e2ba1ea-a352-491f-898e-507a6f40b2cf-kube-api-access-thzvt\") on node \"ip-10-0-134-189.ec2.internal\" DevicePath \"\""
Sep 11 19:28:08.968408 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:28:08.968369    2578 generic.go:356] "Generic (PLEG): container finished" podID="7e2ba1ea-a352-491f-898e-507a6f40b2cf" containerID="6ed024d945d2fa5545a6d2882dd42aa55d1a6dcee5d25e110b929d0bff5d9a4e" exitCode=0
Sep 11 19:28:08.968583 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:28:08.968419    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-cf8f6f1d-kserve-855bc75c6c-vbwdw"
Sep 11 19:28:08.968583 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:28:08.968423    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-cf8f6f1d-kserve-855bc75c6c-vbwdw" event={"ID":"7e2ba1ea-a352-491f-898e-507a6f40b2cf","Type":"ContainerDied","Data":"6ed024d945d2fa5545a6d2882dd42aa55d1a6dcee5d25e110b929d0bff5d9a4e"}
Sep 11 19:28:08.968583 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:28:08.968462    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-cf8f6f1d-kserve-855bc75c6c-vbwdw" event={"ID":"7e2ba1ea-a352-491f-898e-507a6f40b2cf","Type":"ContainerDied","Data":"2b2ab40df1d69306cddf02b52b7b7d9c0edae8c3b71d7526c1654f01cb13835d"}
Sep 11 19:28:08.968583 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:28:08.968485    2578 scope.go:117] "RemoveContainer" containerID="6ed024d945d2fa5545a6d2882dd42aa55d1a6dcee5d25e110b929d0bff5d9a4e"
Sep 11 19:28:08.976892 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:28:08.976858    2578 scope.go:117] "RemoveContainer" containerID="6ed024d945d2fa5545a6d2882dd42aa55d1a6dcee5d25e110b929d0bff5d9a4e"
Sep 11 19:28:08.977170 ip-10-0-134-189 kubenswrapper[2578]: E0911 19:28:08.977150    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ed024d945d2fa5545a6d2882dd42aa55d1a6dcee5d25e110b929d0bff5d9a4e\": container with ID starting with 6ed024d945d2fa5545a6d2882dd42aa55d1a6dcee5d25e110b929d0bff5d9a4e not found: ID does not exist" containerID="6ed024d945d2fa5545a6d2882dd42aa55d1a6dcee5d25e110b929d0bff5d9a4e"
Sep 11 19:28:08.977236 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:28:08.977179    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ed024d945d2fa5545a6d2882dd42aa55d1a6dcee5d25e110b929d0bff5d9a4e"} err="failed to get container status \"6ed024d945d2fa5545a6d2882dd42aa55d1a6dcee5d25e110b929d0bff5d9a4e\": rpc error: code = NotFound desc = could not find container \"6ed024d945d2fa5545a6d2882dd42aa55d1a6dcee5d25e110b929d0bff5d9a4e\": container with ID starting with 6ed024d945d2fa5545a6d2882dd42aa55d1a6dcee5d25e110b929d0bff5d9a4e not found: ID does not exist"
Sep 11 19:28:08.984016 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:28:08.983973    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-cf8f6f1d-kserve-855bc75c6c-vbwdw"]
Sep 11 19:28:08.985763 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:28:08.985743    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-cf8f6f1d-kserve-855bc75c6c-vbwdw"]
Sep 11 19:28:10.396609 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:28:10.396550    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e2ba1ea-a352-491f-898e-507a6f40b2cf" path="/var/lib/kubelet/pods/7e2ba1ea-a352-491f-898e-507a6f40b2cf/volumes"
Sep 11 19:28:44.487066 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:28:44.487035    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-hsbsf_d0880abe-ec36-48e4-a511-ec663873fffa/console-operator/1.log"
Sep 11 19:28:44.494095 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:28:44.494073    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-hsbsf_d0880abe-ec36-48e4-a511-ec663873fffa/console-operator/1.log"
Sep 11 19:28:44.926411 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:28:44.926327    2578 scope.go:117] "RemoveContainer" containerID="c6d16d226c3f3a7d389112e467b97c43c14f14e49abf7563134691e523e9b4a1"
Sep 11 19:28:44.934796 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:28:44.934762    2578 scope.go:117] "RemoveContainer" containerID="5bca4c15f362c3454655b27f8a6a5e3e03af078343a6d1954184d5c63f1401c8"
Sep 11 19:28:51.322558 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:28:51.322532    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-8b2g4_c7f755f2-ae41-42eb-80ee-5024178bf0b4/server/0.log"
Sep 11 19:28:51.436656 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:28:51.436580    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-7qmhn_f15ed915-2f6f-4809-bf86-8cf905cca8f2/manager/0.log"
Sep 11 19:28:51.660643 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:28:51.660546    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-c9988978f-lm6lb_a8434d8c-e2eb-4f72-94cd-d34c6a10b5b6/copy-manifests/0.log"
Sep 11 19:28:51.667428 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:28:51.667411    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-c9988978f-lm6lb_a8434d8c-e2eb-4f72-94cd-d34c6a10b5b6/manager/0.log"
Sep 11 19:28:52.442301 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:28:52.442258    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29819235-mdb4s_6e42c052-f443-4c3e-a254-aa6e376aa3aa/cleanup/0.log"
Sep 11 19:28:54.310536 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:28:54.310506    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-kjl6k_db1ed3da-67dc-4a03-a4e3-5976780ac68d/istio-proxy/0.log"
Sep 11 19:28:54.642796 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:28:54.642684    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5868776b49-nkdlk_70de1f38-8114-4f41-8d3f-76200ab1ff91/kube-auth-proxy/0.log"
Sep 11 19:28:54.982320 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:28:54.982288    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-pm7kk_1edf23d3-5888-4b32-9809-066efe3155c6/payload-processing/0.log"
Sep 11 19:28:55.540965 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:28:55.540914    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-t2k77_45e92205-f8aa-46e3-98f2-f7d2b1e400f6/main/0.log"
Sep 11 19:28:55.968412 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:28:55.968388    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-9v9gf_4449df87-c83f-4043-b2d4-706f8c46ddb0/main/0.log"
Sep 11 19:28:56.072327 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:28:56.072298    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-fpn8c_697b3bd3-5cca-43d1-9b05-ca5ba81a7e99/main/0.log"
Sep 11 19:29:01.067273 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:29:01.067235    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-6fmxz/must-gather-zbr4x"]
Sep 11 19:29:01.067679 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:29:01.067536    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22679cb8-d0b4-463c-8bba-84b34958b3f6" containerName="main"
Sep 11 19:29:01.067679 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:29:01.067547    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="22679cb8-d0b4-463c-8bba-84b34958b3f6" containerName="main"
Sep 11 19:29:01.067679 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:29:01.067571    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e2ba1ea-a352-491f-898e-507a6f40b2cf" containerName="main"
Sep 11 19:29:01.067679 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:29:01.067576    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e2ba1ea-a352-491f-898e-507a6f40b2cf" containerName="main"
Sep 11 19:29:01.067679 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:29:01.067643    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e2ba1ea-a352-491f-898e-507a6f40b2cf" containerName="main"
Sep 11 19:29:01.067679 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:29:01.067653    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="22679cb8-d0b4-463c-8bba-84b34958b3f6" containerName="main"
Sep 11 19:29:01.070426 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:29:01.070409    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6fmxz/must-gather-zbr4x"
Sep 11 19:29:01.072666 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:29:01.072644    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6fmxz\"/\"openshift-service-ca.crt\""
Sep 11 19:29:01.073130 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:29:01.073102    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-6fmxz\"/\"default-dockercfg-bb7j4\""
Sep 11 19:29:01.073244 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:29:01.073136    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6fmxz\"/\"kube-root-ca.crt\""
Sep 11 19:29:01.078418 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:29:01.078398    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6fmxz/must-gather-zbr4x"]
Sep 11 19:29:01.098253 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:29:01.098229    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/04b3a53e-4bdc-4987-9090-9d330fbc069b-must-gather-output\") pod \"must-gather-zbr4x\" (UID: \"04b3a53e-4bdc-4987-9090-9d330fbc069b\") " pod="openshift-must-gather-6fmxz/must-gather-zbr4x"
Sep 11 19:29:01.098359 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:29:01.098293    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sk2q\" (UniqueName: \"kubernetes.io/projected/04b3a53e-4bdc-4987-9090-9d330fbc069b-kube-api-access-4sk2q\") pod \"must-gather-zbr4x\" (UID: \"04b3a53e-4bdc-4987-9090-9d330fbc069b\") " pod="openshift-must-gather-6fmxz/must-gather-zbr4x"
Sep 11 19:29:01.199702 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:29:01.199673    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/04b3a53e-4bdc-4987-9090-9d330fbc069b-must-gather-output\") pod \"must-gather-zbr4x\" (UID: \"04b3a53e-4bdc-4987-9090-9d330fbc069b\") " pod="openshift-must-gather-6fmxz/must-gather-zbr4x"
Sep 11 19:29:01.199879 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:29:01.199738    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sk2q\" (UniqueName: \"kubernetes.io/projected/04b3a53e-4bdc-4987-9090-9d330fbc069b-kube-api-access-4sk2q\") pod \"must-gather-zbr4x\" (UID: \"04b3a53e-4bdc-4987-9090-9d330fbc069b\") " pod="openshift-must-gather-6fmxz/must-gather-zbr4x"
Sep 11 19:29:01.200026 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:29:01.200004    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/04b3a53e-4bdc-4987-9090-9d330fbc069b-must-gather-output\") pod \"must-gather-zbr4x\" (UID: \"04b3a53e-4bdc-4987-9090-9d330fbc069b\") " pod="openshift-must-gather-6fmxz/must-gather-zbr4x"
Sep 11 19:29:01.209661 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:29:01.209639    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sk2q\" (UniqueName: \"kubernetes.io/projected/04b3a53e-4bdc-4987-9090-9d330fbc069b-kube-api-access-4sk2q\") pod \"must-gather-zbr4x\" (UID: \"04b3a53e-4bdc-4987-9090-9d330fbc069b\") " pod="openshift-must-gather-6fmxz/must-gather-zbr4x"
Sep 11 19:29:01.379834 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:29:01.379757    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6fmxz/must-gather-zbr4x"
Sep 11 19:29:01.649450 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:29:01.649356    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6fmxz/must-gather-zbr4x"]
Sep 11 19:29:01.652312 ip-10-0-134-189 kubenswrapper[2578]: W0911 19:29:01.652285    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod04b3a53e_4bdc_4987_9090_9d330fbc069b.slice/crio-dba98a0db105e1022965cc6bf957f432ec3963b9d46d47496fb06483b684a259 WatchSource:0}: Error finding container dba98a0db105e1022965cc6bf957f432ec3963b9d46d47496fb06483b684a259: Status 404 returned error can't find the container with id dba98a0db105e1022965cc6bf957f432ec3963b9d46d47496fb06483b684a259
Sep 11 19:29:01.654617 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:29:01.654582    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 19:29:02.200799 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:29:02.200764    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6fmxz/must-gather-zbr4x" event={"ID":"04b3a53e-4bdc-4987-9090-9d330fbc069b","Type":"ContainerStarted","Data":"dba98a0db105e1022965cc6bf957f432ec3963b9d46d47496fb06483b684a259"}
Sep 11 19:29:03.208457 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:29:03.208410    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6fmxz/must-gather-zbr4x" event={"ID":"04b3a53e-4bdc-4987-9090-9d330fbc069b","Type":"ContainerStarted","Data":"e3f023517d7fe1e72f0016fce9d35906229bbbb26b5f3fffd58833f2a96985cc"}
Sep 11 19:29:03.208980 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:29:03.208474    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6fmxz/must-gather-zbr4x" event={"ID":"04b3a53e-4bdc-4987-9090-9d330fbc069b","Type":"ContainerStarted","Data":"1f0b716a09bc45c4cba04654234b113e4e7705f3466a914b2dcd95c9797d0da8"}
Sep 11 19:29:03.226321 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:29:03.226261    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-6fmxz/must-gather-zbr4x" podStartSLOduration=1.5198674269999999 podStartE2EDuration="2.226236733s" totalImagesPullingTime="706.369306ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:29:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:29:03.223734982 +0000 UTC m=+2419.301325537" watchObservedRunningTime="2026-09-11 19:29:03.226236733 +0000 UTC m=+2419.303827302"
Sep 11 19:29:03.954157 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:29:03.954126    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-vnkmt_90acf5d2-d7df-46bb-a380-73f3ae079b75/global-pull-secret-syncer/0.log"
Sep 11 19:29:04.041380 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:29:04.041352    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-8lq2v_679f1bfe-400c-46c5-837e-39d1dccccf5f/konnectivity-agent/0.log"
Sep 11 19:29:04.091625 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:29:04.091583    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-189.ec2.internal_4bf36dcd150da66bbf96efc4d354ac7b/haproxy/0.log"
Sep 11 19:29:10.806526 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:29:10.806330    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cd954b897-pvd8h_2edba74f-8029-4255-980e-1132dd147c0a/cluster-monitoring-operator/0.log"
Sep 11 19:29:11.190573 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:29:11.190538    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wm4kw_eddf1446-f626-4056-815d-32d95e637ae8/node-exporter/0.log"
Sep 11 19:29:11.220663 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:29:11.220634    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wm4kw_eddf1446-f626-4056-815d-32d95e637ae8/kube-rbac-proxy/0.log"
Sep 11 19:29:11.248445 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:29:11.248415    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wm4kw_eddf1446-f626-4056-815d-32d95e637ae8/init-textfile/0.log"
Sep 11 19:29:13.148570 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:29:13.148543    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-nqtr7_23b61d5d-8bb3-43c1-9ae8-38634ec8b885/networking-console-plugin/0.log"
Sep 11 19:29:13.738233 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:29:13.738197    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-hsbsf_d0880abe-ec36-48e4-a511-ec663873fffa/console-operator/1.log"
Sep 11 19:29:13.744891 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:29:13.744858    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-hsbsf_d0880abe-ec36-48e4-a511-ec663873fffa/console-operator/2.log"
Sep 11 19:29:13.817300 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:29:13.817251    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-6fmxz/perf-node-gather-daemonset-pcmrt"]
Sep 11 19:29:13.825123 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:29:13.825091    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6fmxz/perf-node-gather-daemonset-pcmrt"
Sep 11 19:29:13.827743 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:29:13.827712    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6fmxz/perf-node-gather-daemonset-pcmrt"]
Sep 11 19:29:13.929172 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:29:13.929116    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c3e35e81-afc7-4b89-a9e6-108234aae296-podres\") pod \"perf-node-gather-daemonset-pcmrt\" (UID: \"c3e35e81-afc7-4b89-a9e6-108234aae296\") " pod="openshift-must-gather-6fmxz/perf-node-gather-daemonset-pcmrt"
Sep 11 19:29:13.929376 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:29:13.929228    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c3e35e81-afc7-4b89-a9e6-108234aae296-sys\") pod \"perf-node-gather-daemonset-pcmrt\" (UID: \"c3e35e81-afc7-4b89-a9e6-108234aae296\") " pod="openshift-must-gather-6fmxz/perf-node-gather-daemonset-pcmrt"
Sep 11 19:29:13.929376 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:29:13.929345    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c3e35e81-afc7-4b89-a9e6-108234aae296-proc\") pod \"perf-node-gather-daemonset-pcmrt\" (UID: \"c3e35e81-afc7-4b89-a9e6-108234aae296\") " pod="openshift-must-gather-6fmxz/perf-node-gather-daemonset-pcmrt"
Sep 11 19:29:13.929501 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:29:13.929375    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpxpd\" (UniqueName: \"kubernetes.io/projected/c3e35e81-afc7-4b89-a9e6-108234aae296-kube-api-access-fpxpd\") pod \"perf-node-gather-daemonset-pcmrt\" (UID: \"c3e35e81-afc7-4b89-a9e6-108234aae296\") " pod="openshift-must-gather-6fmxz/perf-node-gather-daemonset-pcmrt"
Sep 11 19:29:13.929501 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:29:13.929419    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c3e35e81-afc7-4b89-a9e6-108234aae296-lib-modules\") pod \"perf-node-gather-daemonset-pcmrt\" (UID: \"c3e35e81-afc7-4b89-a9e6-108234aae296\") " pod="openshift-must-gather-6fmxz/perf-node-gather-daemonset-pcmrt"
Sep 11 19:29:14.030415 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:29:14.030318    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c3e35e81-afc7-4b89-a9e6-108234aae296-proc\") pod \"perf-node-gather-daemonset-pcmrt\" (UID: \"c3e35e81-afc7-4b89-a9e6-108234aae296\") " pod="openshift-must-gather-6fmxz/perf-node-gather-daemonset-pcmrt"
Sep 11 19:29:14.030415 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:29:14.030363    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpxpd\" (UniqueName: \"kubernetes.io/projected/c3e35e81-afc7-4b89-a9e6-108234aae296-kube-api-access-fpxpd\") pod \"perf-node-gather-daemonset-pcmrt\" (UID: \"c3e35e81-afc7-4b89-a9e6-108234aae296\") " pod="openshift-must-gather-6fmxz/perf-node-gather-daemonset-pcmrt"
Sep 11 19:29:14.030669 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:29:14.030453    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c3e35e81-afc7-4b89-a9e6-108234aae296-proc\") pod \"perf-node-gather-daemonset-pcmrt\" (UID: \"c3e35e81-afc7-4b89-a9e6-108234aae296\") " pod="openshift-must-gather-6fmxz/perf-node-gather-daemonset-pcmrt"
Sep 11 19:29:14.030669 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:29:14.030506    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c3e35e81-afc7-4b89-a9e6-108234aae296-lib-modules\") pod \"perf-node-gather-daemonset-pcmrt\" (UID: \"c3e35e81-afc7-4b89-a9e6-108234aae296\") " pod="openshift-must-gather-6fmxz/perf-node-gather-daemonset-pcmrt"
Sep 11 19:29:14.030669 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:29:14.030584    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c3e35e81-afc7-4b89-a9e6-108234aae296-podres\") pod \"perf-node-gather-daemonset-pcmrt\" (UID: \"c3e35e81-afc7-4b89-a9e6-108234aae296\") " pod="openshift-must-gather-6fmxz/perf-node-gather-daemonset-pcmrt"
Sep 11 19:29:14.030860 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:29:14.030668    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c3e35e81-afc7-4b89-a9e6-108234aae296-sys\") pod \"perf-node-gather-daemonset-pcmrt\" (UID: \"c3e35e81-afc7-4b89-a9e6-108234aae296\") " pod="openshift-must-gather-6fmxz/perf-node-gather-daemonset-pcmrt"
Sep 11 19:29:14.030860 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:29:14.030676    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c3e35e81-afc7-4b89-a9e6-108234aae296-lib-modules\") pod \"perf-node-gather-daemonset-pcmrt\" (UID: \"c3e35e81-afc7-4b89-a9e6-108234aae296\") " pod="openshift-must-gather-6fmxz/perf-node-gather-daemonset-pcmrt"
Sep 11 19:29:14.030860 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:29:14.030757    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c3e35e81-afc7-4b89-a9e6-108234aae296-sys\") pod \"perf-node-gather-daemonset-pcmrt\" (UID: \"c3e35e81-afc7-4b89-a9e6-108234aae296\") " pod="openshift-must-gather-6fmxz/perf-node-gather-daemonset-pcmrt"
Sep 11 19:29:14.030860 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:29:14.030794    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c3e35e81-afc7-4b89-a9e6-108234aae296-podres\") pod \"perf-node-gather-daemonset-pcmrt\" (UID: \"c3e35e81-afc7-4b89-a9e6-108234aae296\") " pod="openshift-must-gather-6fmxz/perf-node-gather-daemonset-pcmrt"
Sep 11 19:29:14.038190 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:29:14.038166    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpxpd\" (UniqueName: \"kubernetes.io/projected/c3e35e81-afc7-4b89-a9e6-108234aae296-kube-api-access-fpxpd\") pod \"perf-node-gather-daemonset-pcmrt\" (UID: \"c3e35e81-afc7-4b89-a9e6-108234aae296\") " pod="openshift-must-gather-6fmxz/perf-node-gather-daemonset-pcmrt"
Sep 11 19:29:14.145691 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:29:14.145656    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6fmxz/perf-node-gather-daemonset-pcmrt"
Sep 11 19:29:14.442447 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:29:14.442362    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6fmxz/perf-node-gather-daemonset-pcmrt"]
Sep 11 19:29:14.451923 ip-10-0-134-189 kubenswrapper[2578]: W0911 19:29:14.451875    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc3e35e81_afc7_4b89_a9e6_108234aae296.slice/crio-a61a55068c95b776a17c84879fb1074e1c99f26e9b5299d294426a829395be1a WatchSource:0}: Error finding container a61a55068c95b776a17c84879fb1074e1c99f26e9b5299d294426a829395be1a: Status 404 returned error can't find the container with id a61a55068c95b776a17c84879fb1074e1c99f26e9b5299d294426a829395be1a
Sep 11 19:29:15.043081 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:29:15.043032    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-86695b7cc7-zdvjh_c00f816e-59b2-4fa3-a666-ac15e94fd7d3/volume-data-source-validator/0.log"
Sep 11 19:29:15.302926 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:29:15.302829    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6fmxz/perf-node-gather-daemonset-pcmrt" event={"ID":"c3e35e81-afc7-4b89-a9e6-108234aae296","Type":"ContainerStarted","Data":"b64d2d6ee0e88da41dc79d0db2999937814361f4b4a8a7ff8c74a61fa98dcd6c"}
Sep 11 19:29:15.303105 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:29:15.302935    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6fmxz/perf-node-gather-daemonset-pcmrt" event={"ID":"c3e35e81-afc7-4b89-a9e6-108234aae296","Type":"ContainerStarted","Data":"a61a55068c95b776a17c84879fb1074e1c99f26e9b5299d294426a829395be1a"}
Sep 11 19:29:15.303174 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:29:15.303115    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-6fmxz/perf-node-gather-daemonset-pcmrt"
Sep 11 19:29:15.319606 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:29:15.319537    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-6fmxz/perf-node-gather-daemonset-pcmrt" podStartSLOduration=2.319518508 podStartE2EDuration="2.319518508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 19:29:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:29:15.31906237 +0000 UTC m=+2431.396652946" watchObservedRunningTime="2026-09-11 19:29:15.319518508 +0000 UTC m=+2431.397109066"
Sep 11 19:29:16.121092 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:29:16.121056    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9tzdn_9a3b9624-a629-437f-a700-3f24e40fc0ae/dns/0.log"
Sep 11 19:29:16.144001 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:29:16.143977    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9tzdn_9a3b9624-a629-437f-a700-3f24e40fc0ae/kube-rbac-proxy/0.log"
Sep 11 19:29:16.267042 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:29:16.266972    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dnzn9_43683762-3da6-4f69-a5ad-d685d63bbe4f/dns-node-resolver/0.log"
Sep 11 19:29:16.983000 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:29:16.982962    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-wmk8r_107db790-53d5-4969-8bf9-2ee3b9f9df09/node-ca/0.log"
Sep 11 19:29:18.033337 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:29:18.033301    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-kjl6k_db1ed3da-67dc-4a03-a4e3-5976780ac68d/istio-proxy/0.log"
Sep 11 19:29:18.476080 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:29:18.476033    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5868776b49-nkdlk_70de1f38-8114-4f41-8d3f-76200ab1ff91/kube-auth-proxy/0.log"
Sep 11 19:29:18.575550 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:29:18.575518    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-pm7kk_1edf23d3-5888-4b32-9809-066efe3155c6/payload-processing/0.log"
Sep 11 19:29:19.476722 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:29:19.476693    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bw8xb_0e108d53-e1ef-4459-9eee-2e4be2ccf6e3/serve-healthcheck-canary/0.log"
Sep 11 19:29:20.132289 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:29:20.132251    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mtksz_1e6ec1bf-e524-430b-950b-6ef0039b353b/kube-rbac-proxy/0.log"
Sep 11 19:29:20.157164 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:29:20.157140    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mtksz_1e6ec1bf-e524-430b-950b-6ef0039b353b/exporter/0.log"
Sep 11 19:29:20.182312 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:29:20.182270    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mtksz_1e6ec1bf-e524-430b-950b-6ef0039b353b/extractor/0.log"
Sep 11 19:29:21.316696 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:29:21.316661    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-6fmxz/perf-node-gather-daemonset-pcmrt"
Sep 11 19:29:22.781875 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:29:22.781847    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-8b2g4_c7f755f2-ae41-42eb-80ee-5024178bf0b4/server/0.log"
Sep 11 19:29:22.892109 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:29:22.892078    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-7qmhn_f15ed915-2f6f-4809-bf86-8cf905cca8f2/manager/0.log"
Sep 11 19:29:22.975225 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:29:22.975170    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-c9988978f-lm6lb_a8434d8c-e2eb-4f72-94cd-d34c6a10b5b6/manager/0.log"
Sep 11 19:29:23.000455 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:29:23.000425    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-c9988978f-lm6lb_a8434d8c-e2eb-4f72-94cd-d34c6a10b5b6/copy-manifests/0.log"
Sep 11 19:29:24.369331 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:29:24.369301    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-kjl6k_db1ed3da-67dc-4a03-a4e3-5976780ac68d/istio-proxy/0.log"
Sep 11 19:29:24.436365 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:29:24.436313    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-744bc9d7dd-wnrvb_432e191f-ddf8-4565-a879-99accb3723bd/manager/0.log"
Sep 11 19:29:24.468298 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:29:24.468260    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-jqk42_28145fdf-93bd-4892-b678-ad0670e859dd/openshift-lws-operator/0.log"
Sep 11 19:29:24.810894 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:29:24.810865    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5868776b49-nkdlk_70de1f38-8114-4f41-8d3f-76200ab1ff91/kube-auth-proxy/0.log"
Sep 11 19:29:24.894550 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:29:24.894507    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-pm7kk_1edf23d3-5888-4b32-9809-066efe3155c6/payload-processing/0.log"
Sep 11 19:29:26.487945 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:29:26.487915    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-kjl6k_db1ed3da-67dc-4a03-a4e3-5976780ac68d/istio-proxy/0.log"
Sep 11 19:29:26.948938 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:29:26.948862    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5868776b49-nkdlk_70de1f38-8114-4f41-8d3f-76200ab1ff91/kube-auth-proxy/0.log"
Sep 11 19:29:27.023525 ip-10-0-134-189 kubenswrapper[2578]: I0911 19:29:27.023475    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-pm7kk_1edf23d3-5888-4b32-9809-066efe3155c6/payload-processing/0.log"